Image recognition method, device and equipment and computer readable storage medium

Abstract

The invention discloses an image recognition method, apparatus and device, and a computer readable storage medium. The method comprises the steps of obtaining a to-be-recognized image and a preset target problem; transcoding the to-be-recognized image to obtain image transcoded data; transcoding the target problem to obtain data after problem transcoding; splicing the data after image transcoding and the data after problem transcoding to obtain a splicing vector; recognizing the splicing vector by using an image recognition model obtained through self-chess-playing fine-tuning training to obtain an image recognition result; wherein the iterative training process of the image recognition model comprises the steps of training a main model to distinguish an auxiliary model reply and a human reply text, corresponding to a target question, of an image sample, and training the auxiliary model to generate a reply similar to the human reply text; the main model is a model obtained through current iteration, and the auxiliary model is a model obtained through last iteration. According to the method, the image recognition capability of the model is remarkably improved under the condition that no extra human annotation data exists.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an image recognition method, device, equipment and computer-readable storage medium. Background Art

[0002] The training of image recognition models often relies on human-annotated data. The current main training methods for image recognition models include supervised fine-tuning and reinforcement learning from human feedback.

[0003] Supervised fine-tuning is to further train the pre-trained model using a labeled dataset and evaluate the performance of the model through a validation set, so that it can perform better image recognition. Reinforcement learning from human feedback is a technology that combines reinforcement learning and human feedback, aiming to optimize image recognition models through evaluations provided by humans. It is usually divided into the following steps: collect a large amount of human-labeled data, use the collected human feedback data image sample recognition model, and the model can predict human evaluation of a certain behavior. However, both methods have their own shortcomings. For supervised fine-tuning, high-quality labeled data is required, which increases the difficulty and cost of data collection and annotation. The high-quality human feedback data required for reinforcement learning from human feedback also requires a lot of time and resources.

[0004] In summary, how to effectively solve the problems of the current image recognition model training data collection and annotation being difficult, costly, and requiring a lot of time and resources is an issue that technical personnel in this field urgently need to solve. Summary of the invention

[0005] An object of the present invention is to provide an image recognition method, which significantly improves the image recognition capability of a model without additional human-annotated data; another object of the present invention is to provide an image recognition device, equipment and computer-readable storage medium.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] An image recognition method, comprising:

[0008] Obtain the image to be recognized and the preset target question;

[0009] Transcoding the image to be identified to obtain image transcoded data;

[0010] Transcoding the target question to obtain transcoded question data;

[0011] Splicing the image transcoded data with the problem transcoded data to obtain a splicing vector;

[0012] The splicing vector is identified by using the image recognition model obtained through self-playing fine-tuning training to obtain an image recognition result; wherein the iterative training process of the image recognition model includes training the main model to distinguish the sub-model reply and the human reply text corresponding to the image sample regarding the target question, and training the sub-model to generate a reply similar to the human reply text; the main model is the model obtained in this iteration, and the sub-model is the model obtained in the previous iteration.

[0013] In a specific implementation of the present invention, after acquiring the image to be identified and before transcoding the image to be identified, the method further includes:

[0014] Obtaining preset image recognition conditions;

[0015] When it is determined that the image to be recognized does not meet the image recognition condition, the image to be recognized is adjusted.

[0016] In a specific embodiment of the present invention, training the main model to distinguish between the secondary model response and the human response text corresponding to the image sample regarding the target question, and training the secondary model to generate a response similar to the human response text, includes:

[0017] Collect pre-training image datasets;

[0018] Using the pre-trained image data set to train the pre-built initial model to obtain a pre-trained model;

[0019] Collecting a fine-tuning data set; wherein each fine-tuning data in the fine-tuning data set includes the target question, an image sample related to the target question, and the human response text;

[0020] Inputting the fine-tuning data set into the pre-trained model to obtain a current model reply text;

[0021] Determining whether the pre-trained model can distinguish each human response text from the corresponding current model response text;

[0022] If not, determining the pre-trained model as the image recognition model;

[0023] If yes, adjusting the parameters of the pre-trained model according to the objective function set by the integral probability metric;

[0024] The adjusted iterative model is determined as the main model, and the model corresponding to the model parameters before this iteration is determined as the secondary model;

[0025] Input the fine-tuning data set into the secondary model, and output the current model reply text;

[0026] Determining whether the main model can distinguish each human response text from the corresponding current model response text;

[0027] If not, determining the main model as the image recognition model;

[0028] If so, the parameters of the main model are adjusted according to the objective function set by the integral probability metric, and the step of determining the adjusted iterative model as the main model is repeated.

[0029] In a specific embodiment of the present invention, adjusting the parameters of the pre-trained model according to the objective function set by the integral probability metric includes:

[0030] According to the principle of maximizing the expected value between each human response text and the corresponding current model response text, the parameters of the pre-trained model are adjusted according to the objective function set by the integral probability metric;

[0031] Accordingly, the parameters of the main model are adjusted according to the objective function set by the integral probability metric, including:

[0032] According to the principle of maximizing the expected value between each human response text and the corresponding current model response text, the parameters of the main model are adjusted according to the objective function set by the integral probability metric.

[0033] In a specific embodiment of the present invention, adjusting the parameters of the pre-trained model according to the objective function set by the integral probability metric includes:

[0034] Adjusting parameters of the pre-trained model according to a cross entropy loss function set by an integral probability metric;

[0035] Accordingly, the parameters of the main model are adjusted according to the objective function set by the integral probability metric, including:

[0036] The parameters of the main model are adjusted according to the cross entropy loss function set by the integral probability metric.

[0037] In a specific implementation of the present invention, determining the model corresponding to the model parameters before the current iteration as the secondary model includes:

[0038] Get the regularization term corresponding to the model parameters before this iteration;

[0039] The regularization term is used to correct the model parameters before this iteration to obtain the secondary model.

[0040] In a specific implementation of the present invention, adjusting the parameters of the main model according to the objective function set by the integral probability metric includes:

[0041] Get the current statistical iteration round;

[0042] Determining whether the statistical iteration round exceeds a preset value;

[0043] If yes, executing the step of determining the main model as the image recognition model;

[0044] If not, the step of adjusting the parameters of the main model according to the objective function set according to the integral probability metric is performed.

[0045] An image recognition device, comprising:

[0046] A target question acquisition module is used to acquire the image to be recognized and the preset target question;

[0047] The image transcoded data acquisition module is used to transcode the image to be identified to obtain the image transcoded data;

[0048] A module for obtaining transcoded question data, used for transcoding the target question to obtain transcoded question data;

[0049] A splicing vector obtaining module, used for splicing the image transcoded data with the problem transcoded data to obtain a splicing vector;

[0050] The image recognition result acquisition module is used to use the image recognition model obtained through self-game fine-tuning training to recognize the spliced ​​text to obtain an image recognition result; wherein the iterative training process of the image recognition model includes training the main model to distinguish the sub-model reply and human reply text corresponding to the image sample regarding the target question, and training the sub-model to generate a reply similar to the human reply text; the main model is the model obtained in this iteration, and the sub-model is the model obtained in the previous iteration.

[0051] An image recognition device, comprising:

[0052] Memory for storing computer programs;

[0053] A processor is used to implement the steps of the image recognition method as described above when executing the computer program.

[0054] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the image recognition method described above are implemented.

[0055] The image recognition method provided by the present invention obtains an image to be recognized and a preset target question; transcodes the image to be recognized to obtain image transcoded data; transcodes the target question to obtain question transcoded data; splices the image transcoded data with the question transcoded data to obtain a splicing vector; uses an image recognition model obtained through self-game fine-tuning training to recognize the splicing vector to obtain an image recognition result; wherein the iterative training process of the image recognition model includes training the main model to distinguish between the sub-model replies and human reply texts corresponding to the image samples regarding the target question, and training the sub-model to generate replies similar to the human reply texts; the main model is the model obtained in this iteration, and the sub-model is the model obtained in the previous iteration.

[0056] The beneficial effect of the present invention is that, by transcoding the image to be recognized to obtain the image transcoded data, transcoding the target question to obtain the question transcoded data, splicing the image transcoded data with the question transcoded data to obtain the splicing vector. The image recognition model obtained by self-game fine-tuning training in advance, the model obtained in this iteration is determined as the main model, the model obtained in the previous iteration is determined as the sub-model, the main model is trained to distinguish the sub-model reply and the human reply text corresponding to the image sample about the target question, and the sub-model is trained to generate a reply similar to the human reply text. The splicing vector is identified by the image recognition model to obtain the image recognition result. Self-game fine-tuning promotes the iterative self-evaluation and enhancement of the image recognition model through self-play. Compared with the traditional supervised fine-tuning and image recognition model training method of reinforced learning from human feedback, the present invention does not need to add different manually labeled data or self-improvement from stronger feedback in each iteration process, and realizes the significant improvement of the image recognition ability of the model without additional human labeled data.

[0057] Correspondingly, the present invention also provides an image recognition device, equipment and computer-readable storage medium corresponding to the above-mentioned image recognition method, which have the above-mentioned technical effects and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0059] Figure 1 is a flowchart of an implementation method of an image recognition method in an embodiment of the present invention;

[0060] Figure 2is a flowchart of another image recognition method in an embodiment of the present invention;

[0061] Figure 3 is a structural block diagram of an image recognition device in an embodiment of the present invention;

[0062] Figure 4 is a structural block diagram of an image recognition device in an embodiment of the present invention;

[0063] Figure 5 A schematic diagram of the specific structure of an image recognition device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0065] See also Figure 1 , Figure 1 The following is a flowchart of an image recognition method according to an embodiment of the present invention. The method may include the following steps:

[0066] S101: Obtain an image to be recognized and a preset target question.

[0067] The image recognition model is trained in advance for a preset target problem. When image recognition is required, the image to be recognized and the preset target problem are obtained.

[0068] S102: Transcode the image to be recognized to obtain image transcoded data.

[0069] After the image to be recognized is obtained, the image to be recognized is transcoded, such as using an image encoder to transcode the image to be recognized, to obtain image transcoded data. By transcoding the image to be recognized in an image format, image transcoded data that can be recognized by an image recognition model is obtained.

[0070] S103: Transcode the target question to obtain transcoded question data.

[0071] After obtaining the target question, the target question is transcoded to obtain the transcoded question data. By transcoding the target question in text format, the transcoded question data that can be recognized by the image recognition model is obtained.

[0072] S104: Splicing the image transcoded data with the problem transcoded data to obtain a splicing vector.

[0073] After transcoding the image to be recognized to obtain the image transcoded data, and transcoding the target question to obtain the question transcoded data, the image transcoded data and the question transcoded data are spliced ​​to obtain a spliced ​​vector, thereby realizing the fusion of the image content to be recognized and the target question, so that the model can fully learn the image features and fuse them with the text information during subsequent model training.

[0074] S105: Using the image recognition model obtained through self-game fine-tuning training to recognize the splicing vector, obtain an image recognition result.

[0075] Among them, the iterative training process of the image recognition model includes training the main model to distinguish the sub-model replies and human reply texts corresponding to the target questions of the image samples, and training the sub-model to generate replies similar to the human reply texts; the main model is the model obtained in this iteration, and the sub-model is the model obtained in the previous iteration.

[0076] The image recognition model is iteratively trained in advance through self-playing fine-tuning, and the model obtained in this iteration is determined as the main model, and the model obtained in the previous iteration is determined as the sub-model. The main model is trained to distinguish the sub-model replies and human reply texts corresponding to the image samples on the target question, and the sub-model is trained to generate replies similar to human reply texts. After obtaining the splicing vector, the image recognition model obtained through self-playing fine-tuning is used to recognize the splicing vector to obtain the image recognition result.

[0077] It can be seen from the above technical solution that by transcoding the image to be recognized, the image transcoded data is obtained, the target question is transcoded, the question transcoded data is obtained, and the image transcoded data and the question transcoded data are spliced ​​to obtain a splicing vector. The image recognition model obtained by self-game fine-tuning training in advance determines the model obtained in this iteration as the main model, and the model obtained in the previous iteration as the sub-model. The main model is trained to distinguish the sub-model reply and the human reply text corresponding to the image sample about the target question, and the sub-model is trained to generate a reply similar to the human reply text. The splicing vector is identified by the image recognition model to obtain an image recognition result. Self-game fine-tuning promotes the iterative self-evaluation and enhancement of the image recognition model through self-play. Compared with the traditional supervised fine-tuning and image recognition model training method of reinforced learning from human feedback, the present invention does not need to add different manually labeled data or self-improvement from stronger feedback in each iteration process, and achieves a significant improvement in the image recognition ability of the model without additional human labeled data.

[0078] It should be noted that, based on the above embodiment, the embodiment of the present invention also provides corresponding improved solutions. In the subsequent embodiments, the same steps or corresponding steps as those in the above embodiment can be referenced to each other, and the corresponding beneficial effects can also be referenced to each other, which will not be repeated one by one in the following improved embodiments.

[0079] See also Figure 2 , Figure 2 This is a flowchart of another image recognition method in an embodiment of the present invention. The method may include the following steps:

[0080] S201: Collect a pre-training image dataset.

[0081] When training an image recognition model, first collect a pre-training image dataset.

[0082] There are many ways to collect pre-trained image datasets, including public datasets, crawlers, social media APIs (Application Programming Interfaces), internal enterprise data, and data markets. You can choose the appropriate data source and perform appropriate data cleaning and preprocessing to ensure the quality and effect of model training.

[0083] S202: Using a pre-trained image dataset to train a pre-built initial model to obtain a pre-trained model.

[0084] The model architecture, number of layers, and hidden layer size are selected in advance according to the task requirements to build the initial model. After the pre-trained image dataset is acquired, the pre-built initial model is trained using the pre-trained image dataset to obtain the pre-trained model. For example, the pre-trained model can be obtained by loading the weights of the pre-trained model, selecting the loss function and optimizer, and executing the training cycle, including forward propagation, calculating the loss, back propagation, and updating the weights.

[0085] S203: Collect a fine-tuning data set.

[0086] Among them, each fine-tuning data in the fine-tuning dataset contains the target question, image samples related to the target question, and human response text.

[0087] After training the pre-trained model, the data collection scope is adjusted according to the specific image recognition task requirements, and a fine-tuning dataset is collected. Each fine-tuning data in the fine-tuning dataset contains the target question, image samples related to the target question, and human response text.

[0088] S204: Input the fine-tuning dataset into the pre-training model to obtain the current model reply text.

[0089] After collecting the fine-tuning dataset, the fine-tuning dataset is input into the pre-training model, and the pre-training model is used to respond to the target question to obtain the current model response text.

[0090] S205: Determine whether the pre-trained model can distinguish each human reply text and the corresponding current model reply text. If not, execute step S206; if so, execute step S207.

[0091] After obtaining the current model reply text, determine whether the pre-trained model can distinguish between each human reply text and the corresponding current model reply text. If not, it means that the current model can already output a reply text that is basically consistent with the human reply text for the target question, and execute step S206. If so, it means that the reply text that the current model can output for the target question still has a certain gap with the human reply text, and further training is needed, and execute step S207.

[0092] S206: Determine the pre-trained model as an image recognition model.

[0093] When it is determined that the pre-trained model cannot distinguish between each human response text and the corresponding current model response text, it means that the current model is able to output a response text that is basically consistent with the human response text for the target question, and the pre-trained model is determined to be an image recognition model.

[0094] S207: Adjust parameters of the pre-trained model according to the objective function set by the integral probability metric.

[0095] When it is determined that the pre-trained model cannot distinguish between each human reply text and the corresponding current model reply text, it means that the reply text that the current model can output for the target question is still somewhat different from the human reply text. The parameters of the pre-trained model are adjusted according to the objective function set by the integral probability measurement.

[0096] Integral Probability Metric (IPM) is a method for measuring the difference between probability distributions. In statistics and machine learning, IPM is often used to measure the difference between the true distribution and the model distribution, thereby evaluating the degree of fit of the model.

[0097] The objective function set according to the integral probability metric can be expressed by the following formula:

[0098] ;

[0099] Where x is the question and y is the human response. is the sub-model response, t is the iteration round, f is the objective function, F is the function family, there are different choices, you can use the Lipschitz continuous function family, that is, the function that satisfies |f(x)−f(y)|≤L≤| x−y |, L is the Lipschitz constant.

[0100] Solving the above function is equivalent to the following expression, where I is a monotonically decreasing and convex loss function. The embodiment of the present invention uses a cross entropy loss function:

[0101] ;

[0102] When updating parameters, the model can output a corresponding reward signal based on the quality of the current model response, that is, output a corresponding score. The closer the model response is to the corresponding human response text, the higher the corresponding score is. The lower the similarity between the model response and the corresponding human response text is, the lower the corresponding score is. By adjusting the reward parameters in the direction of the model parameters with high scores when updating according to the scoring results, the accuracy of the model parameter adjustment direction is improved, and the efficiency of model iterative training is improved.

[0103] S208: The adjusted iterative model is determined as the main model, and the model corresponding to the model parameters before this iteration is determined as the secondary model.

[0104] After adjusting the parameters of the pre-trained model, the adjusted iterative model is determined as the main model, and the model corresponding to the model parameters before this iteration is determined as the secondary model.

[0105] S209: Input the fine-tuning dataset into the secondary model, and output the current model reply text.

[0106] After the secondary model is determined, the fine-tuning dataset is input into the secondary model and the current model reply text is output.

[0107] S210: Determine whether the main model can distinguish each human reply text and the corresponding current model reply text, if not, execute step S211, if yes, execute step S212.

[0108] After outputting the current model reply text, determine whether the main model can distinguish between each human reply text and the corresponding current model reply text. If not, it means that the sub-model can already output a reply text that is basically consistent with the human reply text for the target question, and execute step S211. If so, it means that the reply text that the sub-model can output for the target question still has a certain gap with the human reply text, and execute step S212.

[0109] S211: Determine the main model as the image recognition model.

[0110] When it is determined that the main model cannot distinguish between each human response text and the corresponding current model response text, it means that the secondary model can output a response text that is basically consistent with the human response text for the target question, and the main model is determined to be an image recognition model.

[0111] S212: Adjust the parameters of the main model according to the objective function set by the integral probability metric, and return to execute step S208.

[0112] When it is determined that the main model can distinguish between each human reply text and the corresponding current model reply text, it means that the reply text output by the sub-model for the target question is still somewhat different from the human reply text. The adjusted iterative model is determined as the main model again, and the model corresponding to the model parameters before this iteration is determined as the sub-model. A new round of model training is performed based on the newly determined main model and sub-model. By letting the large language model play against itself and continuously optimizing its generation and discrimination capabilities, the image recognition ability of the model is significantly improved without additional human-annotated data, so that the trained image recognition model can generate high-quality text replies.

[0113] S213: Obtain an image to be recognized and a preset target question.

[0114] S214: Obtaining preset image recognition conditions.

[0115] Preset image recognition conditions and obtain preset image recognition conditions. The image recognition conditions may include image clarity, image format, and image arrangement direction.

[0116] S215: When it is determined that the image to be recognized does not meet the image recognition condition, the image to be recognized is adjusted.

[0117] After obtaining the image to be identified and the preset image recognition conditions, when it is determined that the image to be identified does not meet the image recognition conditions, the image to be identified is adjusted, such as cropping, rotating, changing the clarity of the image to be identified, etc. By adjusting the image to be identified, the image to be identified can be more fully utilized, thereby improving the accuracy of image recognition.

[0118] S216: Transcode the image to be recognized to obtain image transcoded data.

[0119] S217: Transcode the target question to obtain transcoded question data.

[0120] S218: Splicing the image transcoded data with the problem transcoded data to obtain a splicing vector.

[0121] S219: Using the image recognition model obtained through self-game fine-tuning training to identify the splicing vector, obtain an image recognition result.

[0122] Among them, the iterative training process of the image recognition model includes training the main model to distinguish the sub-model replies and human reply texts corresponding to the target questions of the image samples, and training the sub-model to generate replies similar to the human reply texts; the main model is the model obtained in this iteration, and the sub-model is the model obtained in the previous iteration.

[0123] The present invention introduces a game mechanism. Through self-playing, a weak model can be converted into a strong model without the need for additional labeled data. The old version of the large language model (i.e., the secondary model) of the previous iteration is used to generate responses to the target question. The subsequent goal is to use a new version of the large language model (i.e., the main model) to distinguish between the responses generated by the old version of the large language model and the responses generated by humans. The main model and the secondary model are the same large language model, but from different iteration rounds. This process can be regarded as a game: the new version of the large language model tries to distinguish the responses of the old version of the large language model. By letting the large language model play against itself and continuously optimizing its generation and discrimination capabilities, it is achieved that the image recognition ability of the model is significantly improved without additional human labeled data, so that the trained image recognition model can generate high-quality text responses.

[0124] In a specific implementation of the present invention, step S207 may include the following steps:

[0125] According to the principle of maximizing the expected value between each human response text and the corresponding current model response text, the parameters of the pre-trained model are adjusted according to the objective function set by the integral probability metric;

[0126] Accordingly, adjusting the parameters of the main model according to the objective function set by the integral probability metric may include the following steps:

[0127] According to the principle of maximizing the expected value between each human response text and the corresponding current model response text, the parameters of the main model are adjusted according to the objective function set by the integral probability metric.

[0128] For the convenience of description, the above steps can be combined for explanation.

[0129] When adjusting the parameters of the pre-trained model, the parameters of the pre-trained model are adjusted according to the principle of maximizing the expected value between each human reply text and the corresponding current model reply text, and according to the objective function set by the integral probability measurement. When adjusting the parameters of the main model, the parameters of the main model are adjusted according to the principle of maximizing the expected value between each human reply text and the corresponding current model reply text, and according to the objective function set by the integral probability measurement. By adjusting the model parameters according to the principle of maximizing the expected value between each human reply text and the corresponding current model reply text, and according to the objective function set by the integral probability measurement, the expected value of the secondary model reply calculated in the objective function is lower, and the expected value of the human reply text is higher, so that the difference between the expected value of the secondary model reply and the human reply text is maximized, thereby achieving effective updating of the model parameters.

[0130] In a specific implementation of the present invention, step S207 may include the following steps:

[0131] Adjust the parameters of the pre-trained model according to the cross entropy loss function set by the integral probability metric;

[0132] Accordingly, adjusting the parameters of the main model according to the objective function set by the integral probability metric may include the following steps:

[0133] The parameters of the main model are adjusted according to the cross entropy loss function set by the integral probability metric.

[0134] For the convenience of description, the above steps can be combined for explanation.

[0135] A linear loss function will produce an unbounded target value, which will cause the secondary model's response to appear negative infinite values ​​during continuous training. Therefore, the pre-trained model parameters are adjusted according to the cross-entropy loss function set by the integral probability metric, and the main model parameters are adjusted according to the cross-entropy loss function set by the integral probability metric. By adjusting the model parameters according to the cross-entropy loss function, the non-negativity, smoothness and exponential decay of the cross-entropy loss function are fully utilized to achieve the discrimination function, while avoiding infinite values.

[0136] The cross entropy loss function is a commonly used loss function that evaluates the performance of a model by measuring the difference between the model's predicted probability and the true label. It has the advantages of smoothness, penalty, and probabilistic interpretation. Smoothness: The function is continuous and differentiable, which allows optimization algorithms such as gradient descent to effectively find the optimal solution. Penalty: For wrong predictions, the loss function will give a larger penalty, which helps the model correct errors faster. Probabilistic interpretation: The function is directly based on probability and can provide confidence in the model's predictions.

[0137] In a specific implementation of the present invention, determining the model corresponding to the model parameters before the current iteration as the secondary model may include the following steps:

[0138] Step 1: Get the regularization term corresponding to the model parameters before this iteration;

[0139] Step 2: Use the regularization term to correct the model parameters before this iteration to obtain the secondary model.

[0140] For the convenience of description, the above two steps can be combined for explanation.

[0141] Obtain the regularization term corresponding to the model parameters before this iteration, and use the regularization term to correct the model parameters before this iteration to obtain the secondary model. By introducing the regularization term, excessive deviation is avoided, making the model stable in self-playing.

[0142] KL regularization is a commonly used regularization technique to prevent the model from deviating too much from the previous state during training, thereby maintaining the stability and generalization ability of the model. KL regularization minimizes the difference between the current distribution of the model and the reference distribution by introducing the Kullback-Leibler divergence (KL divergence) as a regularization term.

[0143] In a specific implementation of the present invention, adjusting the parameters of the main model according to the objective function set by the integral probability metric may include the following steps:

[0144] Step 1: Get the current statistical iteration round;

[0145] Step 2: Determine whether the statistical iteration round exceeds the preset value. If so, return to step S211. If not, adjust the parameters of the main model according to the objective function set by the integral probability metric.

[0146] For the convenience of description, the above two steps can be combined for explanation.

[0147] A preset value is set in advance as the upper limit of the model iteration number, the current statistical iteration round is obtained, and it is determined whether the statistical iteration round exceeds the preset value. If so, it means that the upper limit of the model iteration number has been reached, and the main model is determined as the image recognition model. If not, it means that the upper limit of the model iteration number has not been reached, and the parameters of the main model are adjusted according to the objective function set by the integral probability measurement. By setting the upper limit of the model iteration number, it is avoided to iterate the model infinitely and save resources.

[0148] Corresponding to the above method embodiment, the present invention further provides an image recognition device. The image recognition device described below and the image recognition method described above can be referenced to each other.

[0149] See also Figure 3 , Figure 3 This is a structural block diagram of an image recognition device in an embodiment of the present invention. The device may include:

[0150] A target question acquisition module 31 is used to acquire an image to be recognized and a preset target question;

[0151] The image transcoded data acquisition module 32 is used to transcode the image to be identified to obtain the image transcoded data;

[0152] The question transcoded data obtaining module 33 is used to transcode the target question and obtain the question transcoded data;

[0153] A splicing vector obtaining module 34 is used to splice the image transcoded data with the problem transcoded data to obtain a splicing vector;

[0154] The image recognition result acquisition module 35 is used to use the image recognition model obtained through self-game fine-tuning training to recognize the spliced ​​text and obtain the image recognition result; wherein, the iterative training process of the image recognition model includes training the main model to distinguish the sub-model replies and human reply texts corresponding to the target questions of the image samples, and training the sub-model to generate replies similar to the human reply texts; the main model is the model obtained in this iteration, and the sub-model is the model obtained in the previous iteration.

[0155] It can be seen from the above technical solution that by transcoding the image to be recognized, the image transcoded data is obtained, the target question is transcoded, the question transcoded data is obtained, and the image transcoded data and the question transcoded data are spliced ​​to obtain a splicing vector. The image recognition model obtained by self-game fine-tuning training in advance determines the model obtained in this iteration as the main model, and the model obtained in the previous iteration as the sub-model. The main model is trained to distinguish the sub-model reply and the human reply text corresponding to the image sample about the target question, and the sub-model is trained to generate a reply similar to the human reply text. The splicing vector is identified by the image recognition model to obtain an image recognition result. Self-game fine-tuning promotes the iterative self-evaluation and enhancement of the image recognition model through self-play. Compared with the traditional supervised fine-tuning and image recognition model training method of reinforced learning from human feedback, the present invention does not need to add different manually labeled data or self-improvement from stronger feedback in each iteration process, and achieves a significant improvement in the image recognition ability of the model without additional human labeled data.

[0156] In a specific embodiment of the present invention, the device may further include:

[0157] An image recognition condition acquisition module is used to acquire preset image recognition conditions after acquiring the image to be recognized and before transcoding the image to be recognized;

[0158] The image adjustment module is used to adjust the image to be identified when it is determined that the image to be identified does not meet the image recognition condition.

[0159] In a specific embodiment of the present invention, the device may further include a model training module, the model training module including:

[0160] A pre-training image data set acquisition submodule is used to acquire a pre-training image data set;

[0161] A pre-trained model acquisition submodule is used to train the pre-built initial model using a pre-trained image dataset to obtain a pre-trained model;

[0162] A fine-tuning dataset collection submodule is used to collect a fine-tuning dataset; wherein each fine-tuning data in the fine-tuning dataset includes a target question, image samples related to the target question, and human response text;

[0163] The current model reply text acquisition submodule is used to input the fine-tuning dataset into the pre-trained model to obtain the current model reply text;

[0164] A first judgment submodule is used to judge whether the pre-trained model can distinguish each human reply text from the corresponding current model reply text;

[0165] A first image recognition model determination submodule, configured to determine the pre-trained model as an image recognition model when it is determined that the pre-trained model cannot distinguish between each human response text and the corresponding current model response text;

[0166] A parameter adjustment submodule, for adjusting the parameters of the pre-trained model according to an objective function set by an integral probability metric when it is determined that the pre-trained model can distinguish between each human reply text and the corresponding current model reply text;

[0167] The main model and sub-model determination submodule is used to determine the adjusted iterative model as the main model, and determine the model corresponding to the model parameters before this iteration as the sub-model;

[0168] The current model reply text output submodule is used to input the fine-tuning dataset into the secondary model and output the current model reply text;

[0169] The second judgment submodule is used to judge whether the main model can distinguish each human reply text from the corresponding current model reply text;

[0170] A second image recognition model determination submodule, configured to determine the main model as an image recognition model when the main model cannot distinguish between each human response text and the corresponding current model response text;

[0171] The submodule is repeatedly executed, and is used to adjust the parameters of the main model according to the objective function set by the integral probability measurement when the main model can distinguish each human reply text and the corresponding current model reply text, and repeatedly execute the step of determining the adjusted iterative model as the main model.

[0172] In a specific embodiment of the present invention, the parameter adjustment submodule is specifically a module for adjusting the parameters of the pre-trained model according to the objective function set by the integral probability metric in accordance with the principle of maximizing the expected value between each human reply text and the corresponding current model reply text;

[0173] Correspondingly, the repeated execution submodule is specifically a module that adjusts the parameters of the main model according to the objective function set by the integral probability measurement in accordance with the principle of maximizing the expected value between each human response text and the corresponding current model response text.

[0174] In a specific embodiment of the present invention, the parameter adjustment submodule is specifically a module for adjusting the parameters of the pre-trained model according to the cross entropy loss function set by the integral probability metric;

[0175] Correspondingly, the repeated execution submodule is specifically a module for adjusting parameters of the main model according to the cross entropy loss function set by the integral probability metric.

[0176] In a specific embodiment of the present invention, the main model and the secondary model determination submodule include:

[0177] A regularization item acquisition unit, used to obtain the regularization item corresponding to the model parameter before this iteration;

[0178] The sub-model obtaining unit is used to modify the model parameters before this iteration by using the regularization term to obtain the sub-model.

[0179] In a specific embodiment of the present invention, the repeated execution submodule includes:

[0180] A statistical iteration round acquisition unit, used to acquire the current statistical iteration round;

[0181] A judging unit, used to judge whether the statistical iteration round exceeds a preset value;

[0182] An image recognition model determination unit, configured to determine the main model as the image recognition model when it is determined that the number of statistical iteration rounds exceeds a preset value;

[0183] The main model parameter adjustment unit is used to adjust the parameters of the main model according to the objective function set by the integral probability measurement when it is determined that the statistical iteration round does not exceed the preset value.

[0184] Corresponding to the above method embodiment, see Figure 4 , Figure 4 This is a schematic diagram of an image recognition device provided by the present invention, and the device may include:

[0185] A memory 332, for storing computer programs;

[0186] The processor 322 is used to implement the steps of the image recognition method of the above method embodiment when executing the computer program.

[0187] For details, please refer to Figure 5 , Figure 5 A specific structural diagram of an image recognition device provided for this embodiment, the image recognition device may have relatively large differences due to different configurations or performances, and may include a processor (central processing units, CPU) 322 (for example, one or more processors) and a memory 332, and the memory 332 stores one or more computer programs 342 or data 344. Among them, the memory 332 can be a temporary storage or a permanent storage. The program stored in the memory 332 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the data processing device. Furthermore, the processor 322 can be configured to communicate with the memory 332 to execute a series of instruction operations in the memory 332 on the image recognition device 301.

[0188] The image recognition device 301 may further include one or more power supplies 326 , one or more wired or wireless network interfaces 350 , one or more input and output interfaces 358 , and / or one or more operating systems 341 .

[0189] The steps in the image recognition method described above can be implemented by the structure of an image recognition device.

[0190] Corresponding to the above method embodiment, the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the following steps can be implemented:

[0191] Obtain an image to be recognized and a preset target question; transcode the image to be recognized to obtain image transcoded data; transcode the target question to obtain question transcoded data; splice the image transcoded data with the question transcoded data to obtain a splicing vector; use the image recognition model obtained through self-playing fine-tuning training to recognize the splicing vector to obtain an image recognition result; wherein, the iterative training process of the image recognition model includes training the main model to distinguish between the sub-model replies and human reply texts corresponding to the image samples regarding the target questions, and training the sub-model to generate replies similar to human reply texts; the main model is the model obtained in this iteration, and the sub-model is the model obtained in the previous iteration.

[0192] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0193] For an introduction to the computer-readable storage medium provided by the present invention, please refer to the above method embodiment, and the present invention will not be elaborated here.

[0194] Corresponding to the above method embodiments, the present invention further provides a computer program product, including a computer program, which implements the steps of the above image recognition method when executed by a processor.

[0195] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the devices, equipment and computer-readable storage media disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0196] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the technical solution and core ideas of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. An image recognition method, characterized in that: include: Obtain the image to be recognized and the preset target question; Transcoding the image to be identified to obtain image transcoded data; Transcoding the target question to obtain transcoded question data; Splicing the image transcoded data with the problem transcoded data to obtain a splicing vector; The splicing vector is identified by using the image recognition model obtained through self-playing fine-tuning training to obtain an image recognition result; wherein the iterative training process of the image recognition model includes training the main model to distinguish the sub-model reply and the human reply text corresponding to the image sample regarding the target question, and training the sub-model to generate a reply similar to the human reply text; the main model is the model obtained in this iteration, and the sub-model is the model obtained in the previous iteration.

2. The image recognition method according to claim 1, characterized in that: After acquiring the image to be identified and before transcoding the image to be identified, the method further includes: Obtaining preset image recognition conditions; When it is determined that the image to be recognized does not meet the image recognition condition, the image to be recognized is adjusted.

3. The image recognition method according to claim 1 or 2, characterized in that: The main model is trained to distinguish between the secondary model response and the human response text corresponding to the image sample regarding the target question, and the secondary model is trained to generate a response similar to the human response text, including: Collect pre-training image datasets; Using the pre-trained image data set to train the pre-built initial model to obtain a pre-trained model; Collecting a fine-tuning data set; wherein each fine-tuning data in the fine-tuning data set includes the target question, an image sample related to the target question, and the human response text; Inputting the fine-tuning data set into the pre-trained model to obtain a current model reply text; Determining whether the pre-trained model can distinguish each human response text from the corresponding current model response text; If not, determining the pre-trained model as the image recognition model; If yes, adjusting the parameters of the pre-trained model according to the objective function set by the integral probability metric; The adjusted iterative model is determined as the main model, and the model corresponding to the model parameters before this iteration is determined as the secondary model; Input the fine-tuning data set into the secondary model, and output the current model reply text; Determining whether the main model can distinguish each human response text from the corresponding current model response text; If not, determining the main model as the image recognition model; If so, the parameters of the main model are adjusted according to the objective function set by the integral probability metric, and the step of determining the adjusted iterative model as the main model is repeated.

4. The image recognition method according to claim 3, characterized in that: The pre-trained model is parameterized according to the objective function set by the integral probability metric, including: According to the principle of maximizing the expected value between each human response text and the corresponding current model response text, the parameters of the pre-trained model are adjusted according to the objective function set by the integral probability metric; Accordingly, the parameters of the main model are adjusted according to the objective function set by the integral probability metric, including: According to the principle of maximizing the expected value between each human response text and the corresponding current model response text, the parameters of the main model are adjusted according to the objective function set by the integral probability metric.

5. The image recognition method according to claim 3, characterized in that: The pre-trained model is parameterized according to the objective function set by the integral probability metric, including: Adjusting parameters of the pre-trained model according to a cross entropy loss function set by an integral probability metric; Accordingly, the parameters of the main model are adjusted according to the objective function set by the integral probability metric, including: The parameters of the main model are adjusted according to the cross entropy loss function set by the integral probability metric.

6. The image recognition method according to claim 3, characterized in that: The model corresponding to the model parameters before this iteration is determined as the secondary model, including: Get the regularization term corresponding to the model parameters before this iteration; The regularization term is used to correct the model parameters before this iteration to obtain the secondary model.

7. The image recognition method according to claim 3, characterized in that: The parameters of the main model are adjusted according to the objective function set by the integral probability metric, including: Get the current statistical iteration round; Determining whether the statistical iteration round exceeds a preset value; If yes, executing the step of determining the main model as the image recognition model; If not, the step of adjusting the parameters of the main model according to the objective function set according to the integral probability metric is performed.

8. An image recognition device, characterized in that: include: A target question acquisition module is used to acquire the image to be recognized and the preset target question; The image transcoded data acquisition module is used to transcode the image to be identified to obtain the image transcoded data; A module for obtaining transcoded question data, used for transcoding the target question to obtain transcoded question data; A splicing vector obtaining module, used for splicing the image transcoded data with the problem transcoded data to obtain a splicing vector; The image recognition result acquisition module is used to use the image recognition model obtained through self-game fine-tuning training to recognize the spliced ​​text to obtain an image recognition result; wherein the iterative training process of the image recognition model includes training the main model to distinguish the sub-model reply and human reply text corresponding to the image sample regarding the target question, and training the sub-model to generate a reply similar to the human reply text; the main model is the model obtained in this iteration, and the sub-model is the model obtained in the previous iteration.

9. An image recognition device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the image recognition method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the image recognition method according to any one of claims 1 to 7 are implemented.

Images