Model training method and device, image recognition method and device and storage medium

Abstract

The invention provides a pattern training method and device, an image recognition method and device and a storage medium, and the method comprises the steps: obtaining a plurality of sample images which comprise a plurality of sample to-be-recognized images; according to the plurality of sample images, determining a plurality of sample text data correspondingly related to the plurality of sample images; according to the plurality of sample images and the plurality of sample text data, performing fine tuning on a pre-trained generative model to obtain a target image recognition model; wherein the target image recognition model is used for determining the recognition result of the image, and the recognition accuracy of the image recognition model for the image can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence (AI) technology, and in particular to a model training method, an image recognition method, a device, and a storage medium. Background Art

[0002] At present, with the development of deep learning technology, neural network technology has been widely used in image recognition technology. For example, discriminative AI is often used to determine whether an image involves specific content. Discriminative AI refers to an intelligent system / model that directly classifies or predicts based on input data.

[0003] However, due to the weak semantic understanding ability of discriminative AI, and the fact that some images to be identified are relatively abstract and obscure, or even contain spoofs or tampering, there is a problem of low image recognition accuracy. Summary of the invention

[0004] The present application provides a model training method, an image recognition method, a device and a storage medium, which can improve the recognition accuracy of images.

[0005] In a first aspect, a model training method is provided, comprising: obtaining a plurality of sample images, the plurality of sample images including a plurality of sample images to be recognized; determining, based on the plurality of sample images, a plurality of sample text data respectively corresponding to the plurality of sample images; and fine-tuning a pre-trained generative model based on the plurality of sample images and the plurality of sample text data to obtain a target image recognition model; wherein the target image recognition model is used to determine a recognition result of an image.

[0006] In a second aspect, an image recognition method is provided, comprising: acquiring an image to be recognized; determining target prompt data, the target prompt data being used to indicate that a speech-to-target image recognition model trained by the first aspect or an optional method of the first aspect outputs a recognition result of the image to be recognized; and obtaining a recognition result of the image to be recognized through the target image recognition model based on the image to be recognized and the target prompt data.

[0007] In a third aspect, a model training device is provided, comprising: a first acquisition module, used to acquire multiple sample images, the multiple sample images including multiple sample images to be identified; a first determination module, used to determine, based on the multiple sample images, multiple sample text data corresponding to the multiple sample images respectively; a first training module, used to fine-tune a pre-trained generative model based on the multiple sample images and the multiple sample text data to obtain a target image recognition model; wherein the target image recognition model is used to determine the recognition result of the image.

[0008] In a fourth aspect, an image recognition device is provided, comprising: a first acquisition module, used to acquire an image to be recognized; a first determination module, used to determine target prompt data, the target prompt data being used to indicate that a target image recognition model trained according to the first aspect or an optional method of the first aspect outputs a recognition result of the image to be recognized; a first recognition module, used to obtain a recognition result of the image to be recognized through the target image recognition model according to the image to be recognized and the target prompt data.

[0009] In a fifth aspect, an electronic device is provided, comprising: a processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory to execute the method described in the first aspect or the second aspect.

[0010] In a sixth aspect, a computer-readable storage medium is provided for storing a computer program, wherein the computer program enables a computer to execute the method described in the first aspect or the second aspect.

[0011] In summary, the present application uses the image to be identified and the text data related to the image to be identified to fine-tune the pre-trained generative model (i.e., generative AI) to obtain a target image recognition model, so as to identify the image through the model and determine the recognition result of the image. The generative model has good semantic understanding / analysis capabilities and can accurately identify abstract, obscure images, and images that are mixed with spoofs or tampering, thereby improving the recognition accuracy of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for describing the embodiments are briefly introduced below.

[0013] Figure 1 A flowchart of a model training method provided in an embodiment of the present application;

[0014] Figure 2 A schematic diagram of a model training method provided in an embodiment of the present application;

[0015] Figure 3 A schematic diagram of another model training method provided in an embodiment of the present application;

[0016] Figure 4 A schematic diagram of an image recognition method provided in an embodiment of the present application;

[0017] Figure 5 A flowchart of an image recognition method provided in an embodiment of the present application;

[0018] Figure 6 A schematic diagram of a model training device 600 provided in an embodiment of the present application;

[0019] Figure 7 A schematic diagram of an image recognition device 700 provided in an embodiment of the present application;

[0020] Figure 8 It is a schematic diagram of an electronic device 800 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] The technical solution of the present application will be described below in conjunction with the accompanying drawings in the embodiments of the present application.

[0022] It should be noted that the information, data (including, but not limited to: data for analysis, data for storage, data for display, etc., such as various images, texts and prompt data) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions. For example, the images to be identified and the operations performed on the images to be identified involved in this application are all obtained with full authorization.

[0023] In one embodiment, the technical solution of the present application can be used in an image recognition scenario. Specifically, it can be applied to a scenario of identifying whether an image involves specific content, or to a scenario of identifying whether an image is a specific image, wherein a specific image may refer to an image containing specific content.

[0024] Exemplarily, the specific content may refer to sensitive, disharmonious or unhealthy content such as pornography or violence; the specific image may refer to a sensitive image, a disharmonious image or an unhealthy image, and this application does not impose any restrictions on this.

[0025] Exemplarily, the specific content may also be content such as trees, buildings, sky, people, etc., and this application does not impose any limitation on this.

[0026] In one embodiment, the solution provided by the present application can be executed by any electronic device with data processing capabilities. For example, the electronic device can be a server, specifically an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services. For another example, the electronic device can be a terminal device, specifically a tablet computer, a laptop computer, or a desktop computer. For another example, the electronic device can be a combination of a server and a terminal device, wherein the server and the terminal device in the combination can communicate wirelessly or wired, and the present application does not impose specific restrictions on the electronic device.

[0027] Figure 1 A flowchart of a model training method provided in an embodiment of the present application, which method can be executed by the electronic device in the above embodiment, such as Figure 1As shown, the method includes:

[0028] S110: Acquire a plurality of sample images, where the plurality of sample images include a plurality of sample images to be recognized;

[0029] S120: determining, according to the plurality of sample images, a plurality of sample text data respectively corresponding to the plurality of sample images;

[0030] S130: fine-tuning the pre-trained generative model according to the plurality of sample images and the plurality of sample text data to obtain a target image recognition model, where the target image recognition model is used to determine a recognition result of the image.

[0031] In one embodiment, the sample image to be identified may be an image involving specific content or a specific image in the above embodiment. In addition, in addition to the sample image to be identified, the plurality of sample images may also include images not involving specific content, so as to improve the richness of the samples used in the training model and prevent the model from overfitting.

[0032] In one embodiment, the sample text data related to the sample image may be text or dialogue describing or explaining the image content of the sample image, that is, text description or dialogue description, wherein the dialogue description may be at least one round of dialogue description.

[0033] Exemplarily, a preset large language model can be used to determine a plurality of sample text data corresponding to the plurality of sample images according to the plurality of sample images.

[0034] Specifically, firstly, the first prompt data of each of the plurality of sample images may be determined, the first prompt data indicating that the preset first large language model outputs the text description of the corresponding sample image; then, according to the plurality of first prompt data and the plurality of sample images, the text description of each of the plurality of sample images is obtained through the first large language model; then, the second prompt data of each of the plurality of text descriptions may be determined, the second prompt data indicating that the preset second large language model outputs the conversation description of the corresponding sample image, the conversation description including at least one round of conversation; then, according to the plurality of second prompt data and the plurality of text descriptions, the conversation description of each of the plurality of sample images is obtained through the second large language model; then, according to the plurality of second prompt data and the plurality of text descriptions, the plurality of sample images may be determined through the plurality of sample text data. That is, the sample text data may include at least one of the corresponding conversation description and the corresponding text description.

[0035] Among them, prompt data such as the first prompt data and the second prompt data refers to an instruction or prompt (Prompt), which can be specifically a phrase or a question, and can be used as the input of the large language model, and can be specifically used to indicate the output format or form of the model, etc., to trigger the large language model to generate the output indicated by the prompt data.

[0036] In addition, the first prompt statements or second prompt statements corresponding to the multiple sample images may be different or the same. For example, the first prompt data corresponding to the multiple sample images may all be: please output a text description of the input image, or please describe the input image in the form of text; the second prompt data may all be: please output a dialogue description of the input image, or please describe the input image in the form of dialogue.

[0037] Optionally, the conversation descriptions of the plurality of sample images can be obtained through the second largest language model according to the plurality of sample images, the plurality of second prompt data and the plurality of text descriptions. That is, in the process of determining the conversation description through the second largest language model, the sample image can be input to enrich the basis for determining the conversation description by the second largest language model, and further improve the accuracy of the output conversation description.

[0038] Optionally, the first language model may be a model for implementing image description, and the sample image and the first prompt sentence are input into the first language model, and a text description of the sample image may be output, and the format of the text description, such as the font and the number of words, may be determined based on the content indicated by the first prompt sentence. The second language model may be a model for implementing picture-text question-answering, and the text description (specifically, it may be obtained through the first language model) and the second prompt sentence are input into the second language model (or the sample image may also be input into the second language model), and a dialogue description of the sample image may be output, and the format of the dialogue description, such as the number of dialogue turns and the number of words, may be determined based on the content indicated by the second prompt sentence.

[0039] For example, the first language model or the second speech model may be an open source large language model. For example, the first language model may be a model based on LLaVA (Visual Instruction Tuning), and the second language model may be a model based on Qwen. This application does not limit the first language model or the second speech model.

[0040] The following is an introduction to the first and second largest language models from the model architecture level:

[0041] Optionally, the first language model may include an image feature extraction unit, a text generation unit, and a fusion unit. The image feature extraction unit is used to extract features from the sample image to obtain sample image features; the text generation unit is used to generate an initial text description based on the sample image features; and the fusion unit is used to adjust the initial text description based on the sample image features to obtain a text description corresponding to the sample image.

[0042] For example, the image feature extraction unit may include any of the following, but is not limited to: Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Generative Adversarial Networks (GAN), and Fully Connected Neural Networks (FCNN).

[0043] For example, the text generation unit may include: an embedding layer, a Transformer architecture, and a decoder. The embedding layer may process the sample image features to obtain a vector of a specific dimension; the Transformer architecture may process the vector of a specific dimension based on a self-attention mechanism to generate multiple text words corresponding to the sample image features; the decoder may connect and combine multiple text words according to the vector of a specific dimension to obtain an initial text description (specifically, the decoder may include at least one layer of a self-attention mechanism and an encoder-decoder attention mechanism, wherein the self-attention mechanism may obtain the first initial text description that has been output in the initial text description, and output the second initial text description to be output in the initial text description according to the first initial text description and multiple text words; the encoder-decoder attention mechanism may process multiple text words according to the vector of a specific dimension to obtain the first initial text description).

[0044] For example, the fusion unit may include: a feature extraction layer, a feature fusion layer, and a text conversion layer. The feature extraction layer may use a word embedding or transformer model to extract features from the initial text description to obtain initial text features; the feature fusion layer may fuse the initial text features and the sample image features according to the attention mechanism and the area to be paid attention to in the sample image indicated in the first prompt data or the vocabulary to be paid attention to in the initial text description to obtain fused features; the text conversion layer may be a long short-term memory network (LSTM for short), which may convert the fused features into text descriptions.

[0045] Optionally, the second language model may include a text initial processing unit, a question generation unit, an answer generation unit, and a dialogue generation unit. The text initial processing unit is used to extract features from the text description of the sample image to obtain sample text features; the question generation unit is used to generate an initial question description based on the sample text features; the answer generation unit is used to obtain an initial answer description corresponding to the initial question description based on the sample text features and the initial question description; and the dialogue generation module is used to correspondingly splice the initial question module and the initial answer module to generate a dialogue description.

[0046] For example, the text initial processing unit can divide the text description into multiple word units (one word unit corresponds to one word or phrase); and convert the multiple word units into word unit vectors respectively (for example, based on Word2Vec or BERT, etc., but not limited thereto).

[0047] For example, the question generation unit may include an attention mechanism and a sequence generation model. The attention mechanism may determine a key word meta-vector among multiple word meta-vectors based on multiple word meta-vectors. The sequence generation model may be an LSTM or a Transformer, etc., which may generate an initial question description based on the key word meta-vector and multiple word meta-vectors.

[0048] For example, the answer generation unit may include an information retrieval module and a combination module. The information retrieval module may determine the word meta-vector related to the initial question description from multiple word meta-vectors; the combination module may combine the word meta-vectors related to the initial question description to obtain an initial answer description.

[0049] Optionally, the above-mentioned conversation description may include a questioner description and an answerer description. The questioner description may be included in the second prompt data, or may be determined by the second largest language model based on the text description and / or sample image (specifically, the text description may be rewritten to obtain the questioner description. For example, the affirmative sentence in the text description may be modified into an interrogative sentence, etc.); thereafter, the second largest language model may determine the answerer description based on the questioner description in combination with the text description and / or sample image.

[0050] In one embodiment, Figure 2As shown, the sample image and the corresponding first prompt data can be input into the first language model, and the first language model can output a text description of the sample image. For example, the first prompt data can be "Please describe the content of this image in detail.", and the text description of the image content output by the first language model can be "There is a man in this image, he is standing next to the bathtub...". After that, the above-obtained text description and the second prompt sentence can be input into the second language model, and the second language model can output a dialogue description of the sample image. For example, the second prompt data can be "You are now an AI visual assistant, and now I will provide you with 5 text descriptions of an image, which all describe the same image. Please design a multi-round dialogue between you and humans based on these 5 text descriptions. Humans will ask any questions about this image. The following are these 5 text descriptions...", and the dialogue description can be "Q: How many people are there in this image? A: There are 2 people in the picture, a man and a woman. Q: What clothes is the woman wearing in the picture? A: The woman in the picture is wearing a red bikini. Q: In what scene was this picture taken? A: The picture should be taken at the seaside, with a beach and the sea...".

[0051] In the above embodiment, since the first large language model can capture the key information and details in the image, the diversity and richness of text generation can be improved by generating text descriptions of sample images through the first large language model; the second large language model can further generate dialogue descriptions based on these text descriptions, increasing the level and depth of the descriptions for the sample images, making the generated sample text data more vivid and rich. Moreover, using two different large language models for step-by-step processing, each model focuses on its specific task (text description generation and dialogue description generation), which helps the model better understand and process complex image information.

[0052] It is understandable that fine-tuning refers to the process of further training using a pre-trained model as a basis. Specifically, the generative model can be fine-tuned using LoRA (Low-Rank Adaptation) fine-tuning, which can improve the fine-tuning efficiency and ensure the fine-tuning effect. This application does not limit the fine-tuning method.

[0053] Among them, LoRA fine-tuning is a parameter-efficient fine-tuning method that only needs to update a small number of model parameters in the model. Specifically, at least one network layer can be added to the generative model. During fine-tuning training, only the newly added network layer parameters are updated, and the original parameters of the generative model are not updated. Therefore, only a small number of network layer parameters are updated during the fine-tuning training process, which can reduce the training time.

[0054] A generative model refers to an artificial intelligence system / model that can generate content such as text, images, audio, etc. A generative model can specifically be a text large model ChatGPT, a multimodal large model GPT4, etc., but is not limited to these.

[0055] For example, the generative model can be a Mixture of Experts (MoE) architecture. The core idea of ​​the MoE is to assign input data to different expert models for processing, and then use a mechanism called a "gating network" to determine the weight of each expert model's output in the final output of the entire model, which can achieve inference acceleration of large models.

[0056] In one embodiment, the generative model may include multiple sub-models and a weight model; for the corresponding target sample images and target sample text data in the multiple sample images and the multiple sample text data, the multiple sub-models are respectively used to process different contents in the target sample images and the target sample text data to obtain sample sub-recognition results of each of the multiple sub-models; the weight model is used to determine the recognition weights of each of the multiple sample sub-recognition results; and the sample recognition result of the target sample image is determined based on the multiple recognition weights and the multiple sample sub-recognition results.

[0057] Specifically, the generative model can be fine-tuned according to the difference between the sample recognition result and the actual recognition result corresponding to the sample recognition result (the difference can be determined according to the loss function) to obtain the target image recognition model.

[0058] The following is an introduction to the generative model and target image recognition model from the perspective of model architecture:

[0059] In one embodiment, the generative model may include an initial block unit, multiple sub-models and a weight model. The initial block unit is used to determine the different contents in the target sample image and the target sample text data, and obtain multiple image initial blocks corresponding to the target sample image and multiple text initial blocks corresponding to the target sample text data; and the multiple image initial blocks and the multiple text initial blocks are correspondingly combined to obtain multiple initial block groups, each of which includes a related image initial block and text initial block; the multiple sub-models are used to process the multiple initial block groups respectively, and any target sub-model in the multiple sub-models includes an image block processing unit, a text block processing unit and an image-text fusion unit, the image block processing unit is used to process the image initial blocks in the initial block group to obtain image block features; the text block processing unit is used to process the text initial blocks in the initial block group to obtain text block features; the image-text fusion unit is used to fuse the image block features and the text block features to obtain the initial block features, that is, the sample sub-recognition results. The weight model can determine the recognition weights of each of the multiple sample sub-recognition results, ie, the initial block features; and determine the sample recognition result of the target sample image according to the multiple recognition weights and the multiple initial block features.

[0060] Specifically, the initial block model can first block the target sample text data to obtain the initial block of the text. Specifically, the text can be divided into multiple initial blocks of text according to punctuation marks, line breaks or specific text lengths, or the target sample text data can be blocked according to semantics to obtain the initial block of text corresponding to each semantic (specifically, the text under each semantic in the target sample text data can be determined by semantic recognition models such as word embedding, so as to obtain multiple initial blocks of text). Then, the target sample image can be blocked to obtain the initial block of the image. Specifically, it can be blocked according to pixel values ​​to obtain the initial block of the image corresponding to different objects in the target sample image (different objects in the target sample image correspond to different pixel value ranges, for example, the pixel value ranges corresponding to the sky and trees are different). Alternatively, it can also be blocked according to the initial block of the text to obtain multiple initial blocks of the image (for example, the objects to be extracted (for example, person 1, person 2, article 1, plant 1, etc.) involved in multiple initial blocks of text can be determined first, and then the objects to be extracted in the target sample image are identified and extracted to obtain multiple initial blocks of the image). Afterwards, the key information involved in each of the multiple image initial blocks and the multiple text initial blocks can be determined; the image initial blocks and text initial blocks with the highest similarity in key information are determined as the same initial block group. The key information of the text initial blocks can be keywords in the text initial blocks, and the key information of the image initial blocks can be obtained by detecting attributes such as color and shape in the image initial blocks and performing object recognition.

[0061] Specifically, for the target sub-model, its image block processing unit can be CNN, which can extract the features of the initial image blocks in the initial block group (for example, color, texture, shape and other features) to obtain image block features; the text block processing unit can be RNN or LSTM, which performs feature extraction on the initial text blocks in the initial block group (for example, extracts at least one of vocabulary features, grammatical features and semantic features) to obtain text block features; the image-text fusion unit can fuse the image block features and the text block features by splicing, weighted summation, bilinear pooling and other methods to obtain initial block features.

[0062] Specifically, the weight model can determine the recognition weights of each of the multiple initial block features based on the attention mechanism or the gating mechanism, or can also consider the corresponding positions of the multiple initial block features in the target sample image to determine the recognition weights (for example, the recognition weight at the visual center or geometric center of the target sample image is higher); through weighted averaging, the corresponding block fusion features of the target sample image are determined according to the multiple recognition weights and the multiple initial block features, and then the sample recognition result can be determined according to the block fusion features (for example, the block fusion features can be identified and predicted by a classifier. The category or genus of the target sample image is determined to determine whether the target sample image is a specific image or whether it involves specific content, and obtain a sample recognition result).

[0063] In another embodiment, the generative model may include an image encoder, a connector, and a preset large language model based on the MoE architecture. Among them, the image encoder can identify the features of the sample image and obtain the sample image features; the connector can fuse the sample image features with the preprocessed and encoded sample text data (specifically, refer to the content in the above embodiment) to obtain the fused features; the large language model based on the MoE architecture can process the fused features (specifically, it can include multiple sub-models and a weight model; the multiple sub-models are used to process different contents in the fused features, respectively, to obtain the sample sub-recognition results of the multiple sub-models; the weight model is used to determine the recognition weights of the multiple sample sub-recognition results; according to the multiple recognition weights and the multiple sample sub-recognition results, the sample recognition result of the target sample image is determined) to obtain the sample recognition result.

[0064] It should be noted that the model architecture of the target image recognition model can refer to the model architecture of the generative model. For example, the model architecture of the target image recognition model can be based on the model architecture of the generative model, with at least one network layer added. The at least one network layer is used for fine-tuning training.

[0065] It is understandable that the generative model has a strong ability to understand general visual semantics, for example, the ability to understand the abstract meaning and implicit content of the image content. Therefore, by using the image to be recognized and the text data related to the image to be recognized, fine-tuning the pre-trained generative model can obtain a target image recognition model with more accurate recognition results.

[0066] In one embodiment, in combination with the above content, the recognition result for the image determined by the target image recognition model may be: the sample image is a specific image or involves specific content; or the sample image is not a specific image or does not involve specific content. This application does not limit the specific form of the recognition result.

[0067] In one embodiment, after the above S130, it also includes: acquiring multiple test images, the multiple test images include multiple test images to be recognized; recognizing the multiple test images through a target image recognition model to obtain test recognition results of each of the multiple test images; determining, from the multiple test images, a target test image corresponding to an abnormal test recognition result among the multiple test recognition results; determining, based on the target test image, a target text description and a target dialogue description of the target test image through a first language model and a second language model; determining target test text data related to the target test image based on the target text description and / or target dialogue description; training the target image recognition model based on the target test image and the target test text data to obtain an updated target image recognition model.

[0068] Exemplarily, the test image may be a randomly selected image, or an image that is misidentified among sample images, and the present application does not impose any limitation on this.

[0069] Exemplarily, the true recognition results of each of the multiple test images may be predetermined, and by comparing the multiple true recognition results with the test recognition results, the test recognition results that are inconsistent with the true recognition results may be determined as abnormal test recognition results.

[0070] Exemplarily, at least one of the target text description and the target dialogue description may be determined as the target test text data; or, the target text description may be used to supplement the corresponding content in the target dialogue description to obtain the target test text data.

[0071] Exemplarily, the target image recognition model is trained according to the target test image and the target test text data to obtain the updated target image recognition model, including: fine-tuning the target image recognition model according to multiple sample images, multiple sample text data, the target test image and the target test text data to obtain the updated target image recognition model. That is, the training samples used in this fine-tuning include not only abnormal data (i.e., the target test image and the target test text data), but also other data, so as to prevent the model from overfitting.

[0072] In one embodiment, Figure 3 As shown, the target image recognition model can be used to infer and recognize the test image to be recognized, and output the test recognition result; and filter out the image data with incorrect recognition results; then determine its text description and dialogue description through the first and second language models; then, it can be merged into the original training data set (i.e., multiple sample images and sample text data), and can also be combined with prompt statements (i.e., prompt data, used to instruct the target image recognition model to output the recognition result), through fine-tuning, optimize and update the target image recognition model, and repeat this process until the target image recognition model effect meets the target (for example, the number of fine-tuning times is greater than the preset number, or the number of image data with incorrect recognition results is less than the preset number).

[0073] It is understandable that retraining the model through abnormal recognition results can effectively alleviate the model's hallucination phenomenon and improve the model's recognition accuracy for any image. Hallucination refers to the situation where the model answers questions that are not asked, for example, the output result is very different from the input question. For example, if a picture of a car is input and a prompt sentence is given: "Does the image contain tree elements?", the model output is "It contains tree elements and the reason."

[0074] In one embodiment, in combination with the above, Figure 4 As shown in (a), the sample text data can be first determined based on the sample image and the first largest language model and the second largest language model, thereby determining a multimodal training set containing the sample text data and the sample image; then, the generative model can be fine-tuned using the multimodal training set to obtain a target image recognition model; thereafter, the target image recognition model can be tested in combination with target prompt data (which will be introduced in subsequent embodiments) and random images to determine images with abnormal recognition results; then, through the images with abnormal recognition results, a difficult example data set containing images with abnormal recognition results and their text descriptions and / or dialogue descriptions is determined based on the first largest language model and the second largest language model; the difficult example data set is supplemented back to the end of the multimodal training set, that is, the multimodal training set and the difficult example data set are merged, and the target image recognition model is fine-tuned based on the merged data set to obtain an updated target image recognition model.

[0075] The image recognition method involved in this application is introduced below.

[0076] Figure 5 This is a flowchart of an image recognition method provided in an embodiment of the present application. The method can be executed by the electronic device in the above embodiment, such as Figure 5 As shown, the method includes:

[0077] S510: Acquire an image to be recognized;

[0078] S520: Determine target prompt data, where the target prompt data is used to instruct the target image recognition model to output a recognition result of the image to be recognized;

[0079] S530: Obtaining a recognition result of the image to be recognized according to the image to be recognized and the target prompt data through a target image recognition model.

[0080] It should be understood that the target image recognition model can be trained by the above-mentioned model training method, and the contents and effects involved in the image recognition method can refer to the contents and effects involved in the above-mentioned model training method. To avoid repetition, this application does not elaborate on this. In addition, since the target image recognition model obtained by the above-mentioned training method has a higher recognition accuracy for the image, based on this, this image recognition method can realize image recognition more accurately.

[0081] In one embodiment, the target prompt data can be determined by the following steps: based on multiple actual recognition results, determine multiple specific elements based on which the image is judged to be a specific image; obtain a first recognition result that is an erroneous recognition result among the multiple actual recognition results; determine a first specific element corresponding to the first recognition result among the multiple specific elements; rewrite the first specific element to obtain a rewritten specific element; use the rewritten specific element to update multiple specific elements; and generate target prompt data based on the updated multiple specific elements.

[0082] Exemplarily, the actual recognition result may be a real and accurate recognition result corresponding to a sample image or a test image, but is not limited thereto.

[0083] Exemplarily, the actual recognition result may include a reason for determining the recognition result, and the reason for determining the recognition result may include multiple specific elements based on which the image is judged to be a specific image or involves specific content, and the multiple specific elements are obtained by extracting information from the actual recognition result. Alternatively, the specific elements may also be determined manually.

[0084] For example, the first specific element can be rewritten by adding a restrictive description to obtain a rewritten specific element, or a combination of other specific elements and the first specific element can be determined as the rewritten specific element. For example, when identifying a balloon, it is mistakenly identified as a ball based on the first specific element "circular shape", and the rewritten specific element can be determined to be "circular shape with a rope" or "rope + circular shape".

[0085] Exemplarily, a question-type sentence may be determined based on the updated multiple specific elements to obtain target prompt data to assist the target image recognition model in outputting the recognition result. For example, the question-type sentence may be: Does the image to be recognized involve / contain a specific element after the update?

[0086] In the above content, the target prompt sentence can be determined through more specific, finer-grained and more accurate specific elements, which can assist the target image recognition model to output effective and accurate recognition results.

[0087] Exemplarily, before the target prompt statement is input into the target image recognition model, the target prompt statement can also be simplified to reduce the amount of data and facilitate model understanding and processing. In addition, the output form of the target image recognition model can be set to a token output through the target prompt statement to improve the model running speed. For example, if the recognition result is an image involving specific content or a specific image, it is determined that the target prompt statement instructs the target image recognition model to output "yes", otherwise it outputs "no".

[0088] In one embodiment, in combination with the above, Figure 4 As shown in (b) in FIG. 1 , the recognition result of the image to be recognized can be determined by a question-answering method. The prompt data can be used as a question and input into the target image recognition model together with the image to be recognized to output the recognition result.

[0089] It should be noted that all the above technical solutions can be combined in any way to form optional embodiments of the present application, which will not be described one by one here.

[0090] Figure 6 A schematic diagram of a model training device 600 provided in an embodiment of the present application is shown as follows: Figure 6 As shown, the device 600 includes: a first acquisition module 601, a first determination module 602, a first training module 603, a second acquisition module 604, a first identification module 605, a second determination module 606, a third determination module 607, a fourth determination module 608, and a second training module 609.

[0091] In one embodiment, a first acquisition module 601 is used to acquire multiple sample images, where the multiple sample images include multiple sample images to be identified; a first determination module 602 is used to determine multiple sample text data corresponding to the multiple sample images respectively based on the multiple sample images; a first training module 603 is used to fine-tune a pre-trained generative model based on the multiple sample images and the multiple sample text data to obtain a target image recognition model, wherein the target image recognition model is used to determine the recognition result of the image.

[0092] Exemplarily, the first determination module 602 is specifically used to: determine first prompt data for each of a plurality of sample images, the first prompt data indicating that a preset first large language model outputs a text description of the corresponding sample image; obtain text descriptions for each of the plurality of sample images through the first large language model based on the plurality of first prompt data and the plurality of sample images; determine second prompt data for each of the plurality of text descriptions, the second prompt data indicating that a preset second large language model outputs a conversation description of the corresponding sample image, the conversation description including at least one round of conversation; obtain conversation descriptions for each of the plurality of sample images through the second large language model based on the plurality of second prompt data and the plurality of text descriptions; determine a plurality of sample text data based on the plurality of conversation descriptions and / or the plurality of text descriptions.

[0093] Exemplarily, the second acquisition module 604 is used to acquire multiple test images, wherein the multiple test images include multiple test images to be recognized; the first recognition module 605 is used to recognize the multiple test images through the target image recognition model to obtain the test recognition results of each of the multiple test images; the second determination module 606 is used to determine the target test image corresponding to the abnormal test recognition result among the multiple test recognition results from the multiple test images; the third determination module 607 is used to determine the target text description and target dialogue description of the target test image through the first language model and the second language model according to the target test image; the fourth determination module 608 is used to determine the target test text data related to the target test image according to the target text description and / or the target dialogue description; the second training module 609 is used to train the target image recognition model according to the target test image and the target test text data to obtain the updated target image recognition model. Exemplarily, the second training module 609 is specifically used to fine-tune the target image recognition model according to the multiple sample images, the multiple sample text data, the target test image and the target test text data to obtain the updated target image recognition model.

[0094] Exemplarily, the generative model includes multiple sub-models and a weight model; for the corresponding target sample images and target sample text data in the multiple sample images and the multiple sample text data, the multiple sub-models are respectively used to process different contents in the target sample images and the target sample text data to obtain sample sub-recognition results of each of the multiple sub-models; the weight model is used to determine the recognition weights of each of the multiple sample sub-recognition results; and the sample recognition result of the target sample image is determined based on the multiple recognition weights and the multiple sample sub-recognition results.

[0095] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here. Specifically, Figure 6 The device 600 shown can execute the above-mentioned model training method embodiment, and the aforementioned and other operations and / or functions of each module in the device 600 are respectively for realizing the corresponding processes in the above-mentioned model training method, which will not be repeated here for the sake of brevity.

[0096] Figure 7 A schematic diagram of an image recognition device 700 provided in an embodiment of the present application is shown as follows: Figure 7 As shown, the device 700 includes: a first acquisition module 701, a first determination module 702, and a first identification module 703.

[0097] In one embodiment, the first acquisition module 701 is used to acquire the image to be identified; the first determination module 702 is used to determine target prompt data, and the target prompt data is used to indicate that the target image recognition model trained by the above method outputs the recognition result of the image to be identified; the first recognition module 703 is used to obtain the recognition result of the image to be identified through the target image recognition model according to the image to be identified and the target prompt data.

[0098] Exemplarily, the first determination module 702 is specifically used to determine multiple specific elements based on which an image is judged to be a specific image based on multiple actual recognition results; obtain a first recognition result with an erroneous recognition result among the multiple actual recognition results; determine a first specific element corresponding to the first recognition result among the multiple specific elements; rewrite the first specific element to obtain a rewritten specific element; use the rewritten specific element to update multiple specific elements; and generate target prompt data based on the updated multiple specific elements.

[0099] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here. Specifically, Figure 7The device 700 shown can execute the above-mentioned speech-to-speech translation method embodiment, and the above-mentioned and other operations and / or functions of each module in the device 700 are respectively for realizing the corresponding processes in the above-mentioned speech-to-speech translation method, which will not be described in detail here for the sake of brevity.

[0100] The above describes the apparatus 600 and 700 of the embodiment of the present application from the perspective of the functional module in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or a combination of hardware and software modules in the decoding processor to perform. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above-mentioned model training method or image recognition method embodiment in conjunction with its hardware.

[0101] Figure 8 A schematic diagram of an electronic device 800 provided in an embodiment of the present application.

[0102] like Figure 8 As shown, the electronic device 800 may include:

[0103] The memory 810 and the processor 820, the memory 810 is used to store the computer program and transmit the program code to the processor 820. In other words, the processor 820 can call and run the computer program from the memory 810 to implement the method in the embodiment of the present application.

[0104] For example, the processor 820 may be configured to execute the above method embodiments according to instructions in the computer program.

[0105] In some embodiments of the present application, the processor 820 may include but is not limited to:

[0106] General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0107] In some embodiments of the present application, the memory 810 includes but is not limited to:

[0108] Volatile memory and / or non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAMbus RAM (DR RAM).

[0109] In some embodiments of the present application, the computer program may be divided into one or more modules, which are stored in the memory 810 and executed by the processor 820 to complete the method provided by the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.

[0110] like Figure 8 As shown, the electronic device may also include:

[0111] The transceiver 830 may be connected to the processor 820 or the memory 810 .

[0112] The processor 820 may control the transceiver 830 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 830 may include a transmitter and a receiver. The transceiver 830 may further include an antenna, and the number of antennas may be one or more.

[0113] It should be understood that the various components in the electronic device are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0114] The present application also provides a computer storage medium on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. In other words, the present application embodiment also provides a computer program product containing instructions, and when the instructions are executed by a computer, the computer can perform the method of the above method embodiment.

[0115] When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instruction is loaded and executed on a computer, the computer can be made to perform the corresponding flow in each method in the embodiment of the present application in whole or in part, and generate the functions that can be realized by each method in the embodiment of the present application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instruction can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instruction can be transmitted from a website site, computer, server or data center by wired (such as coaxial cable, optical fiber, digital subscriber line (Digital Subscriber Line, DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, a data center, etc. that contains one or more available media integrations. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital video disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0116] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0117] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the system, device or module can be electrical, mechanical or other forms.

[0118] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. For example, each functional module in each embodiment of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

Claims

1. A model training method, characterized in that: include: Acquire a plurality of sample images, wherein the plurality of sample images include a plurality of sample images to be identified; According to the plurality of sample images, determining a plurality of sample text data respectively corresponding to the plurality of sample images; Fine-tuning the pre-trained generative model according to the multiple sample images and the multiple sample text data to obtain a target image recognition model; The target image recognition model is used to determine the recognition result of the image.

2. The method according to claim 1, characterized in that The step of determining, based on the plurality of sample images, a plurality of sample text data respectively corresponding to the plurality of sample images comprises: Determining first prompt data for each of the plurality of sample images, wherein the first prompt data indicates that a preset first language model outputs a text description corresponding to the sample image; According to the plurality of first prompt data and the plurality of sample images, obtaining text descriptions of the plurality of sample images respectively through the first large language model; Determining second prompt data for each of the plurality of text descriptions, wherein the second prompt data indicates that a preset second largest language model outputs a conversation description corresponding to the sample image, wherein the conversation description includes at least one round of conversation; According to the plurality of the second prompt data and the plurality of the text descriptions, obtaining the dialogue descriptions of the plurality of sample images respectively through the second large language model; The plurality of sample text data are determined according to the plurality of conversation descriptions and / or the plurality of text descriptions.

3. The method according to claim 2, characterized in that After fine-tuning the pre-trained generative model according to the plurality of sample images and the plurality of sample text data to obtain a target image recognition model, the method further includes: Acquire a plurality of test images, wherein the plurality of test images include a plurality of test images to be recognized; Recognize the multiple test images through the target image recognition model to obtain test recognition results of the multiple test images respectively; Determine, from the plurality of test images, a target test image corresponding to an abnormal test recognition result among the plurality of test recognition results; According to the target test image, determining a target text description and a target dialogue description of the target test image by using the first large language model and the second large language model; Determining target test text data associated with the target test image according to the target text description and / or the target dialogue description; The target image recognition model is trained according to the target test image and the target test text data to obtain an updated target image recognition model.

4. An image recognition method, characterized in that: include: Obtain an image to be recognized; Determine target prompt data, wherein the target prompt data is used to instruct the target image recognition model trained by the method according to any one of claims 1 to 3 to output a recognition result of the image to be recognized; According to the image to be recognized and the target prompt data, a recognition result of the image to be recognized is obtained through the target image recognition model.

5. The method according to claim 4, characterized in that The target prompt data is determined, including: Determining, based on the plurality of actual recognition results, a plurality of specific elements on which the image is judged to be a specific image; Obtaining a first recognition result having an erroneous recognition result among the multiple actual recognition results; Determine a first specific element corresponding to the first recognition result among the multiple specific elements; rewriting the first specific element to obtain a rewritten specific element; updating the plurality of specific elements using the rewriting specific element; The target prompt data is generated according to the updated multiple specific elements.

6. A model training device, characterized in that: include: A first acquisition module, used to acquire a plurality of sample images, wherein the plurality of sample images include a plurality of sample images to be identified; A first determining module, configured to determine, based on the plurality of sample images, a plurality of sample text data respectively corresponding to the plurality of sample images; A first training module, used for fine-tuning the pre-trained generative model according to the plurality of sample images and the plurality of sample text data to obtain a target image recognition model; The target image recognition model is used to determine the recognition result of the image.

7. An image recognition device, characterized in that: include: A first acquisition module, used to acquire an image to be identified; A first determination module, used to determine target prompt data, wherein the target prompt data is used to instruct the target image recognition model trained by the method according to any one of claims 1 to 3 to output a recognition result of the image to be recognized; The first recognition module is used to obtain a recognition result of the image to be recognized according to the image to be recognized and the target prompt data through the target image recognition model.

8. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 5 by executing the executable instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

10. A computer program product comprising instructions, characterized in that When the computer program product runs on an electronic device, the electronic device is enabled to execute the method according to any one of claims 1 to 5.

Images