Training methods and apparatus for image recognition models; image recognition methods and apparatus

Abstract

This application provides a training method and apparatus for an image recognition model, and an image recognition method and apparatus. The training method for the image recognition model includes: acquiring a pre-trained image recognition model and training data, wherein the training data includes sample images and standard descriptive information corresponding to the sample images; inputting the sample images into the image recognition model, and acquiring the descriptive information of the sample images according to the model's attention adjustment mechanism; calculating a loss value based on the descriptive information and the standard descriptive information corresponding to the sample images, and adjusting the model parameters and attention adjustment mechanism parameters of the image recognition model. Through the attention adjustment mechanism, the image recognition model can effectively focus on the local features of the sample images during the recognition process, avoiding repeated focus on the same local features, thereby improving the recognition accuracy and training efficiency of the image recognition model.

Description

Technical Field

[0001] This application relates to the field of machine learning technology, and in particular to training methods and apparatus for image recognition models, image recognition methods and apparatus, computing devices, computer-readable storage media and chips. Background Technology

[0002] In practical applications, it is often necessary to obtain descriptive information about images through image recognition models, such as recognizing the content in an image to generate tables or formulas, or recognizing the content in an image to generate descriptive information.

[0003] When training and using image recognition models, an encoder-decoder framework is often employed, incorporating an attention mechanism during the decoding process to identify content within the image. However, during the decoding process using the attention mechanism, the same location is frequently and repeatedly focused on, resulting in significant time consumption and inaccurate content generation. This negatively impacts the training efficiency and accuracy of the image recognition model.

[0004] Therefore, how to solve the above problems, improve the training efficiency of image recognition models, and improve the accuracy of image recognition has become an urgent issue to be addressed. Summary of the Invention

[0005] In view of this, embodiments of this application provide a training method and apparatus for an image recognition model, an image recognition method and apparatus, a computing device, a computer-readable storage medium, and a chip to address the technical deficiencies existing in the prior art.

[0006] According to a first aspect of the embodiments of this application, a method for training an image recognition model is provided, comprising:

[0007] Obtain a pre-trained image recognition model and training data, wherein the training data includes sample images and standard descriptive information corresponding to the sample images;

[0008] The sample image is input into the image recognition model, and the descriptive information of the sample image is obtained according to the model's attention adjustment mechanism;

[0009] The loss value is calculated based on the description information and the standard description information corresponding to the sample image, and the model parameters and attention adjustment mechanism parameters of the image recognition model are adjusted accordingly.

[0010] Optionally, the sample image is input into the image recognition model, and descriptive information of the sample image is obtained according to the model's attention adjustment mechanism, including:

[0011] The sample image is input into the encoding end of the image recognition model for encoding to obtain the local feature vector set and global encoding vector of the sample image;

[0012] The attention distribution is obtained based on the set of local feature vectors of the sample images;

[0013] The global encoding vector is input to the decoding end of the image recognition model, and the attention distribution is adjusted according to the model's attention adjustment mechanism to obtain the descriptive information of the sample image.

[0014] Optionally, the global encoding vector is input to the decoding end of the image recognition model, and the attention distribution is adjusted according to the model's attention adjustment mechanism to obtain the descriptive information of the sample image, including:

[0015] S31. Decode the global encoding vector according to the attention distribution to generate a decoded character set;

[0016] S32. Determine whether the decoding end of the image recognition model has completed decoding. If not, execute S33; if yes, execute S34.

[0017] S33. Adjust the attention distribution according to the model's attention adjustment mechanism, and execute S31;

[0018] S34. Generate description information of the image based on the decoded character set.

[0019] Optionally, the attention distribution is adjusted according to the model's attention adjustment mechanism, including:

[0020] Obtain the hyperparameters, the previous hidden state, and the attention count vector of the image recognition model;

[0021] Based on the hyperparameters of the image recognition model, the previous hidden state, and the global encoding vector, obtain the attention parameter vector;

[0022] Based on the attention parameter vector and the attention count vector, obtain the decay rate of the previous attention distribution;

[0023] The current attention distribution is obtained based on the previous attention distribution and its decay rate.

[0024] Optionally, the global encoding vector is decoded according to the attention distribution to generate a decoded character set, including:

[0025] The previous output is used as a reference encoding vector and input to the decoding end of the image recognition model. The decoding end generates the current decoded character set based on the reference encoding vector, the global encoding vector, and the attention distribution.

[0026] Optionally, the training method for the image recognition model further includes:

[0027] If the loss value is less than a preset threshold, stop adjusting the model parameters and attention adjustment mechanism parameters of the image recognition model.

[0028] According to a second aspect of the embodiments of this application, an image recognition method is provided, comprising:

[0029] Acquire the image to be recognized;

[0030] The image to be recognized is input into a pre-trained image recognition model, which is an image recognition model obtained by any of the image recognition model training methods described above.

[0031] The image recognition model responds to the image to be recognized as input and obtains descriptive information of the image to be recognized according to the model's attention adjustment mechanism.

[0032] Optionally, the image recognition model includes an encoding end and a decoding end;

[0033] The image recognition model, in response to the image to be recognized as input, obtains descriptive information of the image to be recognized according to the model's attention adjustment mechanism, including:

[0034] The image to be recognized is input into the encoding end of the image recognition model for encoding, thereby obtaining the set of local feature vectors and the global encoding vector of the image to be recognized;

[0035] The attention distribution is obtained based on the set of local feature vectors of the image to be identified;

[0036] The global encoding vector is input to the decoding end of the image recognition model, and the attention distribution is adjusted according to the model's attention adjustment mechanism to obtain the descriptive information of the image to be recognized.

[0037] According to a third aspect of the embodiments of this application, a training apparatus for an image recognition model is provided, comprising:

[0038] The first acquisition module is configured to acquire a pre-trained image recognition model and training data, wherein the training data includes sample images and standard description information corresponding to the sample images;

[0039] The input recognition module is configured to input the sample image into the image recognition model and obtain descriptive information of the sample image according to the model's attention adjustment mechanism;

[0040] The adjustment module is configured to calculate the loss value based on the description information and the standard description information corresponding to the sample image, and adjust the model parameters and attention adjustment mechanism parameters of the image recognition model.

[0041] Optionally, the input recognition module includes:

[0042] The encoding unit is configured to input the sample image into the encoding end of the image recognition model for encoding, thereby obtaining a set of local feature vectors and a global encoding vector of the sample image;

[0043] The acquisition unit is configured to acquire the attention distribution based on the set of local feature vectors of the sample image;

[0044] The decoding unit is configured to input the global encoding vector to the decoding end of the image recognition model, and adjust the attention distribution according to the model's attention adjustment mechanism to obtain the descriptive information of the sample image.

[0045] Optionally, the decoding unit includes:

[0046] The decoding subunit is configured to decode the global encoding vector according to the attention distribution to generate a decoded character set;

[0047] The judgment subunit is configured to determine whether the decoding end of the image recognition model has completed decoding;

[0048] The adjustment subunit is configured to adjust the attention distribution according to the model's attention adjustment mechanism;

[0049] The generation subunit is configured to generate descriptive information for the image based on the decoded character set.

[0050] Optionally, the adjustment subunit is further configured to: obtain the hyperparameters of the image recognition model, the previous hidden state, and the attention count vector; obtain the attention parameter vector based on the hyperparameters of the image recognition model, the previous hidden state, and the global encoding vector; obtain the decay rate of the previous attention distribution based on the attention parameter vector and the attention count vector; and obtain the current attention distribution based on the previous attention distribution and the decay rate of the previous attention distribution.

[0051] Optionally, the decoding subunit is further configured to input the previous output as a reference encoding vector into the decoding end of the image recognition model, and the decoding end generates the current decoding character set based on the reference encoding vector, the global encoding vector, and the attention distribution.

[0052] Optionally, the adjustment module is further configured to stop adjusting the model parameters and attention adjustment mechanism parameters of the image recognition model when the loss value is less than a preset threshold.

[0053] According to a fourth aspect of the embodiments of this application, an image recognition device is provided, comprising:

[0054] The second acquisition module is configured to acquire the image to be recognized;

[0055] The input module is configured to input the image to be recognized into a pre-trained image recognition model, the image recognition model being obtained by the image recognition model training method described above.

[0056] The response module is configured such that, in response to the image to be recognized as input, the image recognition model obtains descriptive information of the image to be recognized according to the model's attention adjustment mechanism.

[0057] Optionally, the image recognition model includes an encoding end and a decoding end;

[0058] The response module is further configured to input the image to be recognized into the encoding end of the image recognition model for encoding to obtain a set of local feature vectors and a global encoding vector of the image to be recognized; obtain an attention distribution based on the set of local feature vectors of the image to be recognized; input the global encoding vector into the decoding end of the image recognition model, adjust the attention distribution according to the model's attention adjustment mechanism, and obtain descriptive information of the image to be recognized.

[0059] According to a fifth aspect of the present application, a computing device is provided, including a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein the processor executes the instructions to implement a training method for the image recognition model or the steps of the image recognition method.

[0060] According to a sixth aspect of the embodiments of this application, a computer-readable storage medium is provided that stores computer instructions, which, when executed by a processor, implement the steps of the training method for the image recognition model or the image recognition method.

[0061] According to a seventh aspect of the present application, a chip is provided that stores computer instructions, which, when executed by the chip, implement the training method of the image recognition model or the steps of the image recognition method.

[0062] The image recognition model training method provided in this application embodiment involves acquiring a pre-trained image recognition model and training data, wherein the training data includes sample images and corresponding standard descriptive information; inputting the sample images into the image recognition model, and obtaining the descriptive information of the sample images according to the model's attention adjustment mechanism; calculating a loss value based on the descriptive information and the corresponding standard descriptive information of the sample images, and adjusting the model parameters and attention adjustment mechanism parameters of the image recognition model. During model training, the attention adjustment mechanism effectively focuses on local feature regions of the sample images during the recognition process, avoiding repeated focus on the same local feature regions, thereby improving the recognition accuracy and training efficiency of the image recognition model. Attached Figure Description

[0063] Figure 1 This is a structural block diagram of the computing device provided in the embodiments of this application;

[0064] Figure 2 This is a flowchart of the training method for the image recognition model provided in the embodiments of this application;

[0065] Figure 3 This is a flowchart of a training method for an image recognition model provided in another embodiment of this application;

[0066] Figure 4 This is a flowchart of the image recognition method provided in the embodiments of this application;

[0067] Figure 5 This is a schematic diagram of the structure of the image recognition model provided in the embodiments of this application;

[0068] Figure 6 This is a schematic diagram of the image recognition model for recognizing formula images provided in the embodiments of this application;

[0069] Figure 7 This is a schematic diagram of the structure of the training device for the image recognition model provided in the embodiments of this application;

[0070] Figure 8 This is a schematic diagram of the structure of the image recognition device provided in the embodiments of this application. Detailed Implementation

[0071] Many specific details are set forth in the following description to provide a full understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this application; therefore, this application is not limited to the specific embodiments disclosed below.

[0072] The terminology used in one or more embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this application. The singular forms “a,” “the,” and “the” used in one or more embodiments of this application and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” used in one or more embodiments of this application refers to and includes any or all possible combinations of one or more associated listed items.

[0073] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this application, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0074] First, the terminology used in one or more embodiments of the present invention will be explained.

[0075] Attention mechanism: Depending on the needs, focus attention on important points while ignoring other unimportant factors.

[0076] Hyperparameters: In the context of machine learning, hyperparameters are parameters whose values ​​are set before the learning process begins, rather than parameters obtained through training. Typically, hyperparameters need to be optimized to select an optimal set for the learning machine, thereby improving learning performance and effectiveness.

[0077] Hidden state: The hidden state is used for output and to control the choice of dependence on the preceding text at this moment. It is the result of various gating in the Long Short-Term Memory (LSTM) network.

[0078] This application provides a training method and apparatus for an image recognition model, an image recognition method and apparatus, a computing device, a computer-readable storage medium, and a chip, which will be described in detail in the following embodiments.

[0079] Figure 1 A structural block diagram of a computing device 100 according to an embodiment of this application is shown. The components of the computing device 100 include, but are not limited to, a memory 110 and a processor 120. The processor 120 is connected to the memory 110 via a bus 130, and a database 150 is used to store data.

[0080] The computing device 100 also includes an access device 140, which enables the computing device 100 to communicate via one or more networks 160. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 140 may include one or more of any type of wired or wireless network interface (e.g., a Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) interface, a Wi-MAX interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0081] In one embodiment of this application, the aforementioned components of the computing device 100 and Figure 1 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 1 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this application. Those skilled in the art can add or replace other components as needed.

[0082] The computing device 100 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs. The computing device 100 can also be a mobile or stationary server.

[0083] Among them, processor 120 can execute Figure 2 The steps in the training method of the image recognition model shown are as follows. Figure 2 A flowchart of a training method for an image recognition model according to an embodiment of this application is shown, including steps 202 to 206.

[0084] Step 202: Obtain a pre-trained image recognition model and training data, wherein the training data includes sample images and standard descriptive information corresponding to the sample images.

[0085] Pre-trained image recognition models are deep learning architectures that have already been trained to perform image recognition tasks on large datasets. Using pre-trained models eliminates the need to train a new model from scratch when solving image recognition problems; instead, deep training can be performed on top of an already trained model, improving the efficiency of model training.

[0086] Training data includes sample images and corresponding standard descriptive information. Sample images can be formula images, landscape images, portrait images, etc. The standard descriptive information corresponding to the sample images is the output information after the sample images are recognized by the image recognition model. For example, taking a formula image as an example, the formula in the image is... The standard description information corresponding to the formula image is: "y=\sum_(i=1)^{3}2x^i". Taking a landscape image as an example, the image is an island in the sea, and the standard description information of the landscape image is "an island in the sea". Taking a human image as an example, the image is a group of children playing football, and the standard description information of the human image is "a group of children playing football".

[0087] In one embodiment provided in this application, taking a formula image as an example, the formula in the image is: The standard description information corresponding to the formula image is: "y=\sum_(i=1)^{3}2x^i".

[0088] Step 204: Input the sample image into the image recognition model and obtain the descriptive information of the sample image according to the model's attention adjustment mechanism.

[0089] The sample image is input into the image recognition model. During the recognition of the sample image, the image recognition model adjusts the attention distribution of the image recognition model according to the model's attention adjustment mechanism, so as to fully and effectively focus on the local feature regions of the sample image, thereby obtaining the descriptive information of the sample image. The model's attention adjustment mechanism is used to adjust the attention distribution to determine the local feature regions that need to be focused on each time the sample image is recognized.

[0090] The sample image recognition model includes an encoding end and a decoding end. Optionally, step 204 can be implemented by the following steps S2041 to S2043.

[0091] S2041. Input the sample image into the encoding end of the image recognition model for encoding to obtain the local feature vector set and global encoding vector of the sample image.

[0092] The sample image is input to the encoding end of the image recognition model. The encoding end of the image recognition model receives the sample image and encodes it to obtain a set of local feature vectors of the sample image. The set of local feature vectors is then passed through a pooling layer to obtain the global encoding vector of the sample image.

[0093] In the embodiments of this application, taking a formula image as an example, the formula in the formula image is... The formula image is input into the image recognition model. The encoding end of the image recognition model receives the formula image, encodes the formula image, and obtains a set of local feature vectors of the formula image. The local feature vector set is pooled to obtain the global encoding vector [X1, ..., X]. n ].

[0094] S2042. Obtain the attention distribution based on the set of local feature vectors of the sample image.

[0095] The attention distribution [α1, ..., α2] corresponding to the sample image is obtained based on the set of local feature vectors of the sample image. m ], where α1 is the local feature vector [A1, ..., A n The attention weights α corresponding to the local features. m For local feature vectors [M1, ..., M] n Attention weights for the corresponding local features.

[0096] S2043. Input the global encoding vector into the decoding end of the image recognition model, and adjust the attention distribution according to the model's attention adjustment mechanism to obtain the descriptive information of the sample image.

[0097] The global encoding vector is input to the decoding end of the image recognition model. Each time the global encoding vector is decoded, the decoding end of the image recognition model adjusts the attention distribution by reducing the weight of regions that have already been focused on, according to the model's attention adjustment mechanism, thereby obtaining the descriptive information of the sample image.

[0098] In the embodiments provided in this application, taking the formula image as an example, the attention distribution [α1, ..., α] is adjusted. m The method to reduce the attention weight of already focused regions is applied to the global encoding vector [X1, ..., X]. n Decode the image to obtain the descriptive information of the formula image.

[0099] Step 206: Calculate the loss value based on the description information and the standard description information corresponding to the sample image, and adjust the model parameters and attention adjustment mechanism parameters of the image recognition model.

[0100] Based on the difference between the descriptive information and the standard descriptive information corresponding to the sample image, a loss value is calculated. The model parameters and attention adjustment mechanism parameters in the image recognition model are then adjusted by backpropagating the loss value, thereby achieving the purpose of training the image recognition model.

[0101] Optionally, if the loss value is less than a preset threshold, the adjustment of the model parameters and attention adjustment mechanism parameters of the image recognition model shall be stopped.

[0102] When the training times of all training data in the training dataset reach the preset number of times, the training of the image recognition model is paused. It is then determined whether the loss value is less than the preset threshold. If the loss value is greater than the preset threshold, it means that the training of the image recognition model has not yet achieved the desired effect, and the model parameters and attention adjustment mechanism parameters of the image recognition model need to be adjusted further. If the loss value is less than or equal to the preset threshold, the adjustment of the model parameters and attention adjustment mechanism parameters of the image recognition model is stopped.

[0103] The image recognition model training method provided in this application adjusts the attention distribution of the image recognition model through the model's attention adjustment mechanism, ensuring that the local feature regions of the sample image can be fully and effectively focused on during the image recognition process, avoiding repeated focus on the same local feature regions of the sample image, thereby improving the accuracy of the image recognition model in recognizing sample images. The loss value is calculated through the descriptive information and standard descriptive information of the sample image, and the model parameters and attention adjustment mechanism parameters of the image recognition model are adjusted using the loss value, further accelerating the training efficiency of the image recognition model.

[0104] Figure 3 A training method for an image recognition model according to another embodiment of this application is shown, including steps 302 to 316.

[0105] Step 302: Obtain a pre-trained image recognition model and training data, wherein the training data includes sample images and standard descriptive information corresponding to the sample images.

[0106] Step 302 is the same as the method of step 202 above. For a detailed explanation of step 302, please refer to the detailed content of step 202 in the foregoing embodiments, which will not be repeated here.

[0107] In the embodiments provided in this application, the sample image is a group of children playing football, and the standard description information corresponding to the sample image is "a group of children playing football".

[0108] Step 304: Input the sample image into the encoding end of the image recognition model for encoding to obtain the set of local feature vectors and the global encoding vector of the sample image.

[0109] In the embodiments provided in this application, a set of local feature vectors and a global encoding vector of a sample image are obtained.

[0110] Step 306: Obtain the attention distribution based on the set of local feature vectors of the sample image.

[0111] In the embodiments provided in this application, the attention distribution for acquiring sample images is [0.1, 0.1, 0.2, 0.2, 0.3, 0.1].

[0112] Steps 304 to 306 are the same as steps S2041 to S2042 above. For a detailed explanation of steps 304 to 306, please refer to the detailed content of steps S2041 to S2042 in the foregoing embodiments, which will not be repeated here.

[0113] Step 308: Decode the global encoding vector according to the attention distribution to generate a decoded character set.

[0114] Optionally, the previous output is used as a reference encoding vector and input to the decoding end of the image recognition model. The decoding end generates the current decoding character set based on the reference encoding vector, the global encoding vector, and the attention distribution.

[0115] During the decoding process, the decoding end of the image recognition model decodes the global encoding vector. During the first decoding, an initial reference encoding vector is set, and the global encoding vector is decoded according to the initial reference encoding vector and the current attention distribution to obtain the current decoded character set. During decoding other than the first decoding, the previously output decoded character set is used as the reference encoding vector, and the global encoding vector is decoded according to the reference encoding vector and the current attention distribution to obtain the current decoded character set.

[0116] Step 310: Determine whether the decoding end of the image recognition model has completed decoding. If not, proceed to step 312; if yes, proceed to step 314.

[0117] Determine whether the decoding end of the image recognition model has completed the decoding of the global encoding vector of the image. If the decoding has not been completed, proceed to step 312; if the decoding has been completed, proceed to step 314.

[0118] Step 312: Adjust the attention distribution according to the model's attention adjustment mechanism, and proceed to step 308.

[0119] Optionally, adjusting the attention distribution according to the model's attention adjustment mechanism can be achieved through the following steps S3121 to S3124.

[0120] S3121. Obtain the hyperparameters, the previous hidden state, and the attention count vector of the image recognition model.

[0121] The attention count vector is used to record the number of times different local features have been noticed. The dimension of the attention count vector is the same as the dimension of the attention distribution.

[0122] Obtain the hyperparameters n of the pre-set image recognition model and the hidden state h of the previous image recognition model. t-1 And the count vector of the number of times it receives attention.

[0123] S3122. Obtain the attention parameter vector based on the hyperparameters of the image recognition model, the previous hidden state, and the global encoding vector.

[0124] Attention parameter vector [e1, ... e2] n w1, ..., w n The formula (1) is used to obtain the result, where [e1, ..., e] n The elements in [w1, ..., w] represent different attenuation strategies, each a decimal between 0 and 1. Multiple attenuation strategies are combined to obtain the final attenuation rate. The number of attenuation strategies is the same as the hyperparameters of the image recognition model. n [ represents the proportions corresponding to different attenuation strategies, where w1 + w2 + ... + w] n =1. v is the global encoding vector, h t-1 This represents the hidden state of the previous image recognition model.

[0125] (1)

[0126] S3123. Based on the attention parameter vector and the attention count vector, obtain the decay rate of the previous attention distribution.

[0127] The decay rate of the previous attention distribution is obtained by formula (2), where e is the decay strategy, w is the proportion corresponding to the decay strategy, count is the attention count vector, and n is the hyperparameter of the image recognition model.

[0128] (2)

[0129] In the embodiments provided in this application, the calculated attenuation rate rate of the attention distribution is [0.8, 0.6, 0.7, 0.8, 0.5, 0.8].

[0130] S3124. Obtain the current attention distribution based on the previous attention distribution and the decay rate of the previous attention distribution.

[0131] Get the previous attention distribution The attention distribution for the current iteration can be obtained using formula (3). .

[0132] (3)

[0133] Last attention distribution Given [0.1, 0.1, 0.2, 0.2, 0.3, 0.1], and the decay rate of the previous attention distribution was [0.8, 0.6, 0.7, 0.8, 0.5, 0.8], the current attention distribution... [0.8] 0.1, 0.6 0.1, 0.7 0.2, 0.8 0.1, 0.5 0.3, 0.8 [0.1], i.e., [0.08, 0.06, 0.14, 0.08, 0.15, 0.08].

[0134] Step 314: Generate description information of the image based on the decoded character set.

[0135] The decoded character sets generated each time are combined in the output order to form the descriptive information of the image.

[0136] In the embodiments provided in this application, the decoded character sets "a group", "children", "at", and "playing football" generated each time are combined to form the image description information "a group of children playing football".

[0137] Step 316: Calculate the loss value based on the description information and the standard description information corresponding to the sample image, and adjust the model parameters and attention adjustment mechanism parameters of the image recognition model.

[0138] Step 316 is the same as the method of step 206 above. For a detailed explanation of step 316, please refer to the detailed content of step 206 in the foregoing embodiments, which will not be repeated here.

[0139] The image recognition model training method provided in this application, during each encoding process, calculates the decay rate of the attention distribution in the previous encoding process by statistically analyzing the number of times each local feature region is focused, and combining this with the attention parameter vector. Based on the previous attention distribution and its decay rate, the current attention distribution is obtained. This ensures that the local feature regions of the sample image are fully and effectively focused on during image recognition, avoiding repeated focus on the same local feature regions, thus improving the accuracy of the image recognition model in recognizing sample images. The method also calculates the loss value using the descriptive information and standard descriptive information of the sample image, and uses this loss value to adjust the model parameters and attention adjustment mechanism parameters of the image recognition model, further accelerating the training efficiency of the image recognition model.

[0140] Figure 4 An image recognition method according to an embodiment of this application is shown, including steps 402 to 406.

[0141] Step 402: Obtain the image to be recognized.

[0142] An image to be identified is an image waiting to be identified. It can be obtained from the Internet or pre-stored in the device.

[0143] Step 404: Input the image to be recognized into a pre-trained image recognition model, wherein the image recognition model is obtained by any of the image recognition model training methods described above.

[0144] In the embodiments provided in this application, such as Figure 5 As shown, taking a landscape image as an example, Image is the landscape image, which is an island in the sea. The landscape image is input into a pre-trained image recognition model, which is trained to output descriptive information of the landscape image based on the input landscape image.

[0145] Step 406: The image recognition model responds to the image to be recognized as input and obtains the descriptive information of the image to be recognized according to the model's attention adjustment mechanism.

[0146] The image recognition model includes an encoding end and a decoding end. Figure 5 A schematic diagram of the image recognition model is shown, such as... Figure 5 As shown, CNN is the encoding end of the image recognition model, LSTM is the decoding end of the image recognition model, MLP is a multilayer perceptron used to output the recognition result, and Attention is the attention mechanism in the decoding end of the image recognition model.

[0147] The decoding end of the image recognition model receives the input image to be recognized, encodes the image to be recognized to obtain the encoding vector of the image to be recognized, and inputs the encoding vector to the decoding end of the image recognition model.

[0148] The decoding end of the image recognition model receives the encoding vector of the image to be recognized. In each decoding process, the previous output is used as the reference encoding vector for the current decoding. The encoding vector is decoded by adjusting the attention distribution of the image recognition model to generate the character set for the current decoding.

[0149] Optionally, step 406 can be implemented by the following steps S4061 to S4063.

[0150] S4061. Input the image to be recognized into the encoding end of the image recognition model for encoding to obtain the set of local feature vectors and the global encoding vector of the image to be recognized.

[0151] In the embodiments provided in this application, a landscape image Image is input to the encoding end of an image recognition model for encoding to obtain a set of local feature vectors V of the landscape image. The set of local feature vectors is then passed through a pooling layer to obtain a global encoding vector V of the landscape image. mean .

[0152] S4062. Obtain the attention distribution based on the set of local feature vectors of the image to be identified.

[0153] In the embodiments provided in this application, the attention distribution α of the landscape image is obtained based on the set of local feature vectors corresponding to the landscape image. old .

[0154] S4063. Input the global encoding vector into the decoding end of the image recognition model, and adjust the attention distribution according to the model's attention adjustment mechanism to obtain the descriptive information of the image to be recognized.

[0155] In the embodiments provided in this application, such as Figure 5 As shown, the global encoding vector V of the landscape image Image is... mean The input is sent to the decoding end of the image recognition model. The decoding end uses the previous hidden state h... t-1 and global encoding vector V mean Obtain the attention parameter vector and, combined with the attention count vector (count), calculate the decay rate (rate) of the previous attention distribution. Then, use the previous attention distribution α... old Calculate the current attention distribution α using the decay rate (rate) of the previous attention distribution. new By adjusting the attention distribution, the weights of previously attended local feature regions are reduced, while the LSTM receives the decoding result y from the previous input. t-1 Using the reference encoding vector, the global encoding vector of the landscape image is decoded, and the decoding result y is output through the MLP. t After the decoding end completes the decoding of the encoded vector, the final description information of the landscape image is obtained as: "There is an island in the sea".

[0156] In the embodiments provided in this application, such as Figure 6 As shown, Figure 6Taking the recognition of a formula image as an example, a schematic diagram of adjusting the attention distribution in the formula image is shown. The formula image is divided into 9 regions, 601-609. During the recognition of the formula by the pre-trained image recognition model, after region 601 has been paid attention to, the attention distribution of the formula image is adjusted according to the pre-trained attention adjustment mechanism, reducing the weight of the attention distribution in region 601. Similarly, the attention distribution weight of regions 602 to 609 is reduced according to the pre-trained attention adjustment mechanism, so that the image recognition model can quickly and accurately recognize the formula image. Finally, the description information of the formula image is recognized as "y=\sum_(i=1)^{3}2x^i".

[0157] The image recognition method provided in this application encodes the input image to be recognized at the encoding end to obtain an encoding vector and an attention distribution. Then, the encoding vector is input to the decoding end of the image to be recognized. When the character is generated by decoding at the decoding end, the attention distribution is adjusted according to the attention adjustment mechanism of the trained model to reduce the weight of the feature regions that have already been paid attention to. This can effectively focus on the complete local feature regions of the image and improve the recognition efficiency and accuracy of the image recognition model.

[0158] Corresponding to the above-described embodiments of the image recognition model training method, this application also provides embodiments of the image recognition model training apparatus. Figure 7 A schematic diagram of the structure of a training apparatus for an image recognition model according to an embodiment of this application is shown. Figure 7 As shown, the device includes:

[0159] The first acquisition module 702 is configured to acquire a pre-trained image recognition model and training data, wherein the training data includes sample images and standard description information corresponding to the sample images;

[0160] The input recognition module 704 is configured to input the sample image into the image recognition model and obtain the descriptive information of the sample image according to the model's attention adjustment mechanism;

[0161] The adjustment module 706 is configured to calculate the loss value based on the description information and the standard description information corresponding to the sample image, and adjust the model parameters and attention adjustment mechanism parameters of the image recognition model.

[0162] Optionally, the input recognition module 704 includes:

[0163] The encoding unit is configured to input the sample image into the encoding end of the image recognition model for encoding, thereby obtaining a set of local feature vectors and a global encoding vector of the sample image;

[0164] The acquisition unit is configured to acquire the attention distribution based on the set of local feature vectors of the sample image;

[0165] The decoding unit is configured to input the global encoding vector to the decoding end of the image recognition model, and adjust the attention distribution according to the model's attention adjustment mechanism to obtain the descriptive information of the sample image.

[0166] Optionally, the decoding unit includes:

[0167] The decoding subunit is configured to decode the global encoding vector according to the attention distribution to generate a decoded character set;

[0168] The judgment subunit is configured to determine whether the decoding end of the image recognition model has completed decoding;

[0169] The adjustment subunit is configured to adjust the attention distribution according to the model's attention adjustment mechanism;

[0170] The generation subunit is configured to generate descriptive information for the image based on the decoded character set.

[0171] Optionally, the adjustment subunit is further configured to: obtain the hyperparameters of the image recognition model, the previous hidden state, and the attention count vector; obtain the attention parameter vector based on the hyperparameters of the image recognition model, the previous hidden state, and the global encoding vector; obtain the decay rate of the previous attention distribution based on the attention parameter vector and the attention count vector; and obtain the current attention distribution based on the previous attention distribution and the decay rate of the previous attention distribution.

[0172] Optionally, the decoding subunit is further configured to input the previous output as a reference encoding vector into the decoding end of the image recognition model, and the decoding end generates the current decoding character set based on the reference encoding vector, the global encoding vector, and the attention distribution.

[0173] Optionally, the adjustment module 706 is further configured to stop adjusting the model parameters and attention adjustment mechanism parameters of the image recognition model when the loss value is less than a preset threshold.

[0174] The image recognition model training device provided in this application, during each encoding process, calculates the decay rate of the attention distribution in the previous encoding process by counting the number of times each local feature region is focused, and combining this with the attention parameter vector. Based on the previous attention distribution and its decay rate, the current attention distribution is obtained. This ensures that the local feature regions of the sample image are fully and effectively focused on during image recognition, avoiding repeated focus on the same local feature regions, thus improving the accuracy of the image recognition model in recognizing sample images. The loss value is calculated using the descriptive information and standard descriptive information of the sample image, and the model parameters and attention adjustment mechanism parameters of the image recognition model are adjusted using the loss value, further accelerating the training efficiency of the image recognition model.

[0175] Corresponding to the above-described image recognition method embodiments, this application also provides image recognition device embodiments. Figure 8 A schematic diagram of the structure of an image recognition device according to an embodiment of this application is shown. Figure 8 As shown, the device includes:

[0176] The second acquisition module 802 is configured to acquire the image to be recognized;

[0177] The input module 804 is configured to input the image to be recognized into a pre-trained image recognition model, wherein the image recognition model is obtained by the image recognition model training method according to any one of claims 1-6;

[0178] The response module 806 is configured such that the image recognition model, in response to the image to be recognized as input, obtains descriptive information of the image to be recognized according to the model's attention adjustment mechanism.

[0179] Optionally, the image recognition model includes an encoding end and a decoding end;

[0180] The response module 806 is further configured to input the image to be recognized to the encoding end of the image recognition model for encoding to obtain a set of local feature vectors and a global encoding vector of the image to be recognized; obtain an attention distribution based on the set of local feature vectors of the image to be recognized; input the global encoding vector to the decoding end of the image recognition model, adjust the attention distribution according to the model's attention adjustment mechanism, and obtain descriptive information of the image to be recognized.

[0181] The image recognition device provided in this application encodes the input image to be recognized at the encoding end to obtain an encoding vector and an attention distribution. Then, the encoding vector is input to the decoding end of the image to be recognized. When the character is generated by decoding at the decoding end, the attention distribution is adjusted according to the attention adjustment mechanism of the trained model to reduce the weight of the feature regions that have already been paid attention to. This can effectively focus on the complete local feature regions of the image and improve the recognition efficiency and accuracy of the image recognition model.

[0182] In one embodiment of this application, a computing device is also provided, including a memory, a processor, and computer instructions stored in the memory and executable on the processor. When the processor executes the instructions, it implements the steps of the training method for the image recognition model or the image recognition method.

[0183] An embodiment of this application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the steps of the training method for the image recognition model or the image recognition method as described above.

[0184] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solution of the image recognition model training method or the image recognition method described above. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the image recognition model training method or the image recognition method described above.

[0185] This application discloses a chip that stores computer instructions, which, when executed by a processor, implement the steps of the image recognition model training method or image recognition method as described above.

[0186] The foregoing has described specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0187] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0188] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0189] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0190] The preferred embodiments disclosed above are merely illustrative of this application. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this application. These embodiments are selected and specifically described in this application to better explain the principles and practical applications of this application, thereby enabling those skilled in the art to better understand and utilize this application. This application is limited only by the claims and their full scope and equivalents.

Claims

1. A training method for an image recognition model, characterized in that, The method comprises: obtaining a pre-trained image recognition model and training data, wherein the training data comprises sample images and standard description information corresponding to the sample images; inputting the sample images into the image recognition model to obtain description information of the sample images according to an attention adjustment mechanism of the model, wherein the attention adjustment mechanism adjusts an attention distribution by reducing the weight of a local feature region that has been focused on, so as to determine a local feature region that needs to be focused on each time the sample image is recognized; calculating a loss value according to the description information and the standard description information corresponding to the sample images, and adjusting model parameters and attention adjustment mechanism parameters of the image recognition model; wherein the attention adjustment mechanism adjusts the attention distribution by reducing the weight of a local feature region that has been focused on, comprising: Obtaining a preset hyperparameter n of the image recognition model, a hidden state of a previous image recognition model and a number of attention times vector ; According to the following formula, the attention parameter vector is obtained according to the hyperparameter n of the image recognition model, the last hidden state and the global encoding vector of the sample image , , The elements in the attention parameter vector represent different attenuation strategies, each attenuation strategy being a decimal between 0 and 1, The proportion corresponding to each attenuation strategy is different. ； According to the following formula, the decay rate of the last attention distribution is obtained according to the attention parameter vector and the attention frequency vector : a = 1 - (1 - a) * (1 - f) / (1 + f) : ； According to the following formula, the attention distribution of the current time is obtained according to the last attention distribution and the decay rate of the last attention distribution :​ 。 2.The method of Claim 1, wherein, inputting the sample images into the image recognition model to obtain description information of the sample images according to an attention adjustment mechanism of the model, comprising: inputting the sample images into the encoding end of the image recognition model for encoding to obtain a local feature vector set and a global encoding vector of the sample images; obtaining an attention distribution according to the local feature vector set of the sample images; inputting the global encoding vector into the decoding end of the image recognition model to adjust the attention distribution according to the attention adjustment mechanism of the model to obtain the description information of the sample images. 3.The method of Claim 2, wherein, inputting the global encoding vector into the decoding end of the image recognition model to adjust the attention distribution according to the attention adjustment mechanism of the model to obtain the description information of the sample images, comprising: S31, decoding the global encoding vector according to the attention distribution to generate a decoding character set; S32, judging whether the decoding end of the image recognition model has completed decoding, if not, executing S33, and if yes, executing S34; S33, adjusting the attention distribution according to the attention adjustment mechanism of the model, and executing S31; S34, generating the description information of the image according to the decoding character set. 4.The method of Claim 3, wherein, decoding the global encoding vector according to the attention distribution to generate a decoding character set, comprising: inputting the output of the last time as a reference encoding vector into the decoding end of the image recognition model, and the decoding end generates a decoding character set of the current time according to the reference encoding vector, the global encoding vector and the attention distribution. 5.The method of Claim 1, wherein Further comprising: stopping adjusting the model parameters and the attention adjustment mechanism parameters of the image recognition model if the loss value is less than a preset threshold.

6. An image recognition method characterized by, The method comprises: obtaining a to-be-recognized image; inputting the to-be-recognized image into a pre-trained image recognition model, wherein the image recognition model is obtained by the image recognition model training method according to any one of claims 1-5; the image recognition model obtains description information of the to-be-recognized image according to an attention adjustment mechanism of the model in response to the to-be-recognized image as input.

7. The image recognition method of claim 6, wherein, the image recognition model comprises an encoding end and a decoding end; The image recognition model, in response to the image to be recognized as input, obtains description information of the image to be recognized according to an attention adjustment mechanism of the model, including: inputting the image to be recognized into an encoding end of the image recognition model for encoding to obtain a local feature vector set and a global encoding vector of the image to be recognized; obtaining an attention distribution according to the local feature vector set of the image to be recognized; inputting the global encoding vector into a decoding end of the image recognition model, and adjusting the attention distribution according to an attention adjustment mechanism of the model to obtain the description information of the image to be recognized.

8. A training device for an image recognition model, characterized in that, including: The first obtaining module is configured to obtain a pre-trained image recognition model and training data, wherein the training data includes a sample image and standard description information corresponding to the sample image; The input recognition module is configured to input the sample image into the image recognition model, and obtain description information of the sample image according to an attention adjustment mechanism of the model, wherein the attention adjustment mechanism adjusts an attention distribution by reducing the weight of a local feature region that has been focused on, so as to determine a local feature region that needs to be focused on each time the sample image is recognized; The adjustment module is configured to calculate a loss value according to the description information and the standard description information corresponding to the sample image, and adjust model parameters and attention adjustment mechanism parameters of the image recognition model; The attention adjustment mechanism adjusts the attention distribution by reducing the weight of a local feature region that has been focused on, including: acquiring a preset hyperparameter n of the image recognition model, a hidden state of a previous image recognition model and a number of attention times vector ; According to the following formula, the attention parameter vector is obtained according to the hyperparameter n of the image recognition model, the last hidden state and the global encoding vector of the sample image The elements in the attention parameter vector , represent different attenuation strategies, each of which is a decimal between 0 and 1, and the corresponding proportion of different attenuation strategies: ； According to the following formula, an attention parameter vector and the number of attention times vector , the decay rate of the last attention distribution is obtained : ； According to the following formula, the attention distribution of the current time is obtained according to the last attention distribution and the decay rate of the last attention distribution :​ 。 9. An image recognition apparatus characterized by comprising: including: The second obtaining module is configured to obtain an image to be recognized; The input module is configured to input the image to be recognized into a pre-trained image recognition model, and the image recognition model is obtained by the image recognition model training method according to any one of claims 1-5; The response module is configured to the image recognition model, in response to the image to be recognized as input, obtains description information of the image to be recognized according to an attention adjustment mechanism of the model.

10. A computing device comprising a memory, a processor, and computer instructions stored on the memory and executable on the processor, wherein, The processor executes the instructions to implement the steps of the method of any one of claims 1-5 or 6-7.

11. A computer-readable storage medium storing computer instructions, wherein, The instructions are executed by the processor to implement the steps of the method of any one of claims 1-5 or 6-7.

Images