Training method, device, equipment and storage medium for large image and text recognition model

Abstract

The present disclosure provides a training method, device, equipment and storage medium for a large model of image and text recognition, which relates to the field of artificial intelligence technology, specifically to the technical fields of computer vision, deep learning, large models, etc., and can be applied to scenes such as image and text dialogue based on artificial intelligence. The method includes: obtaining a training sample set, wherein the training samples in the training sample set include sample images, prompts of the sample images, face recognition results of the sample images, and image description information of the sample images; inputting the sample images into an image processing module, and outputting image feature information; inputting the image feature information into an image-text semantic alignment module, and outputting semantic feature information; using the semantic feature information, face recognition results and prompt as inputs, and the image description information as output, training a pre-constructed large language model to obtain a large model of image and text recognition.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically to technical fields such as computer vision, deep learning, and large models, and in particular to a training method, device, equipment, and storage medium for a large model of image and text recognition, which can be applied to scenarios such as image and text dialogue based on artificial intelligence. Background Art

[0002] As large language models have shown extraordinary capabilities in various fields, multimodal large language models (MLLMs) have gradually shown greater potential. Using powerful large language models (LLMs) as the "brain", MLLMs can perform various tasks, such as text-to-picture dialogues. However, MLLMs are still in their early stages, and improving their perception capabilities is an important direction for improvement. Summary of the invention

[0003] The present invention provides a training method, device, equipment and storage medium for a large image and text recognition model.

[0004] According to a first aspect of the present disclosure, a training method for a large model for image and text recognition is provided, comprising: obtaining a training sample set, wherein the training samples in the training sample set include a sample image, a prompt word corresponding to the sample image, a face recognition result corresponding to the sample image, and image description information of the sample image; inputting the sample image into an image processing module, and outputting image feature information of the sample image; inputting the image feature information into an image-text semantic alignment module, and outputting semantic feature information of the sample image; using the semantic feature information, face recognition result, and prompt as input, and using the image description information as output, to train a pre-constructed large language model to obtain the large model for image and text recognition.

[0005] According to a second aspect of the present disclosure, a method for image and text recognition is provided, including: obtaining a picture to be recognized and a prompt corresponding to the picture to be recognized; determining a face recognition result of the picture to be recognized based on a face feature extraction model and a face feature library; inputting the picture to be recognized, the prompt and the face recognition result into a large image and text recognition model, and outputting image description information of the picture to be recognized, wherein the large image and text recognition model is trained using the method described in the first aspect.

[0006] According to a third aspect of the present disclosure, a training device for a large model of image and text recognition is provided, comprising: a first acquisition module, configured to acquire a training sample set, wherein the training samples in the training sample set include a sample image, a prompt word prompt corresponding to the sample image, a face recognition result corresponding to the sample image, and image description information of the sample image; a first output module, configured to input the sample image into an image processing module, and output image feature information of the sample image; a second output module, configured to input the image feature information into an image-text semantic alignment module, and output semantic feature information of the sample image; a first training module, configured to take the semantic feature information, face recognition result, and prompt as input, and take the image description information as output, to train a pre-constructed large language model to obtain the large model of image and text recognition.

[0007] According to a fourth aspect of the present disclosure, there is provided a graphic and text recognition device, comprising: a third acquisition module, configured to acquire a picture to be recognized and a prompt corresponding to the picture to be recognized; a determination module, configured to determine a face recognition result of the picture to be recognized based on a face feature extraction model and a face feature library; a third output module, configured to input the picture to be recognized, the prompt and the face recognition result into a graphic and text recognition large model, and output image description information of the picture to be recognized, wherein the graphic and text recognition large model is trained using the method described in the first aspect.

[0008] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in any implementation manner of the first aspect or the second aspect.

[0009] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to execute the method described in any implementation manner of the first aspect or the second aspect.

[0010] According to a seventh aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the computer program implements the method described in any implementation manner of the first aspect or the second aspect.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0013] Figure 1 is an exemplary system architecture diagram in which the present disclosure may be applied;

[0014] Figure 2 is a flowchart of an embodiment of a training method for a large model of image and text recognition according to the present disclosure;

[0015] Figure 3 is a flowchart of another embodiment of a method for training a large model for image and text recognition according to the present disclosure;

[0016] Figure 4 is a flowchart of another embodiment of a training method for a large model of image and text recognition according to the present disclosure;

[0017] Figure 5 is a flow chart of an embodiment of the image and text recognition method according to the present disclosure;

[0018] Figure 6 is a flow chart of another embodiment of the image and text recognition method according to the present disclosure;

[0019] Figure 7 is a schematic diagram of an application scenario of the image and text recognition method disclosed in the present invention;

[0020] Figure 8 It is a structural schematic diagram of an embodiment of a training device for a large model of image and text recognition according to the present disclosure;

[0021] Fig. 9 is a structural schematic diagram of an embodiment of an image and text recognition device according to the present disclosure;

[0022] Fig.10 It is a block diagram of an electronic device used to implement the training method of the large image and text recognition model or the image and text recognition method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0024] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0025] Figure 1 An exemplary system architecture 100 is shown to which an embodiment of a method for training a large image and text recognition model or a device for training a large image and text recognition model of the present disclosure can be applied.

[0026] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0027] The user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send information, etc. Various client applications can be installed on the terminal devices 101, 102, 103.

[0028] The terminal devices 101, 102, 103 may be hardware or software. When the terminal devices 101, 102, 103 are hardware, they may be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc. When the terminal devices 101, 102, 103 are software, they may be installed in the above electronic devices. They may be implemented as multiple software or software modules, or as a single software or software module. No specific limitation is made here.

[0029] The server 105 can provide various services. For example, the server 105 can analyze and process the training sample sets obtained from the terminal devices 101, 102, and 103, and generate processing results (such as a large model for image and text recognition).

[0030] It should be noted that the server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server 105 is software, it can be implemented as multiple software or software modules (for example, for providing distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0031] It should be noted that the training method of the large image and text recognition model provided in the embodiment of the present disclosure is generally executed by the server 105 , and accordingly, the training device of the large image and text recognition model is generally set in the server 105 .

[0032] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0033] Continue to refer Figure 2 , which shows a process 200 of an embodiment of a training method for a large image and text recognition model according to the present disclosure. The training method for a large image and text recognition model includes the following steps:

[0034] Step 201: Obtain a training sample set.

[0035] In this embodiment, the execution subject of the training method of the large image-text recognition model (for example Figure 1 The server 105 shown in the figure will obtain a training sample set, wherein the training samples in the training sample set include a sample image, a prompt word corresponding to the sample image, a face recognition result corresponding to the sample image, and image description information of the sample image. The training sample set includes multiple training samples, and each training sample includes a sample image, a prompt word corresponding to the sample image, a face recognition result corresponding to the sample image, and image description information of the sample image.

[0036] Here, the sample images can be images collected on the Internet that contain faces of public figures, historical figures, and various celebrities. The prompt corresponding to the sample image is the content entered by the user during the search. For example, for a certain image, the user enters the question "Who is in the image", then "Who is in the image" is the prompt corresponding to the image. The face recognition result corresponding to the sample image includes the name information of the person in the sample image and the facial feature information of the person in the sample image. The face recognition result here can be obtained through the face recognition model. The image description information of the sample image is the complete description information of the image, such as how many people are in the image, what each person is doing, etc.

[0037] Step 202: input the sample image into the image processing module, and output the image feature information of the sample image.

[0038] In this embodiment, the above-mentioned execution subject will input the sample image into the image processing module, and output the image feature information of the sample image. Here, the image processing module can extract the image features of the sample image, thereby obtaining the image feature information. The image processing module can be a ViT (vision transformer) model. ViT is a model that applies Transformer to image classification, which has good effect and strong scalability. The ViT model is a text image pre-training model pre-trained on open source data. In this embodiment, only the image encoder (image encoding) module is used as the image processing model. The resolution of its input image is 448*448, and the output is a 768-dimensional feature, that is, the output image feature information is 768 dimensions. Here, the image processing module can also be other models with image feature extraction functions, such as CLIP (Contrastive Language-Image Pre-Training) model, etc., which are not specifically limited in this embodiment.

[0039] Step 203: input the image feature information into the image-text semantic alignment module, and output the semantic feature information of the sample image.

[0040] In this embodiment, the above-mentioned execution subject will input the image feature information into the image text semantic alignment module, and output the semantic feature information of the sample image. Here, the image text semantic alignment module can align the image feature information of the sample image with the semantic feature information, that is, input the image feature information of the sample image, and output the corresponding semantic feature information. The image text semantic alignment module can be a Bert (Bidirectional Encoder Representations from Transformers) model. Bert is a pre-trained language model with strong applicability. It only needs to fine-tune the model to be applied to a variety of tasks. In this embodiment, the Bert model is used to perform semantic compression on the extracted image feature information. That is, the above-mentioned 768-dimensional image feature information is input into the Bert model, and 32 semantic feature information can be output through the combination calculation of self-attention and cross-attention, and the dimension of each semantic feature information is 768. Here, the image text semantic alignment module can also be other models with image text semantic alignment function, such as GPT (Generative Pre-Trained Transformer) model, etc., which is not specifically limited in this embodiment.

[0041] Step 204, taking the semantic feature information, face recognition results and prompt as input, taking the image description information as output, training the pre-built large language model, and obtaining a large image and text recognition model.

[0042] In this embodiment, the execution subject takes the semantic feature information, face recognition results and prompt as input, takes the image description information as output, trains the pre-built large language model, and obtains a trained large model for image and text recognition. The trained large model for image and text recognition can recognize the image and output the complete description information of the image.

[0043] It should be noted that a large language model (large model) is an artificial intelligence model designed to understand and generate human language. By training on large amounts of text data, it can perform a wide range of tasks, including text summarization, translation, sentiment analysis, and more. A large model refers to a deep neural network with more than 1 billion parameters that can process massive amounts of data and complete various complex tasks. With the continuous improvement of computer hardware performance and the continuous optimization of deep learning algorithms, the development of large models is also accelerating. The parameter scale of large models continues to expand, the training time is getting longer and longer, but the performance is also improving. Large models are usually based on deep learning architectures, such as transformers, which enable impressive capabilities in various natural language processing tasks. Common large models can include but are not limited to ChatGPT, GPT-4, ERNIE Bot (Wenxin Yiyan), etc.

[0044] The training method of the large model of image and text recognition provided in the embodiment of the present disclosure first obtains a training sample set; then, the sample image is input into the image processing module, and the image feature information of the sample image is output; thereafter, the image feature information is input into the image-text semantic alignment module, and the semantic feature information of the sample image is output; finally, the semantic feature information, face recognition results, and prompt are used as input, and the image description information is used as output to train the pre-built large language model to obtain the large model of image and text recognition. The training method of the large model of image and text recognition in this embodiment uses the face recognition result as the new input of the large language model, and simultaneously inputs the features of the face as soft features and image features into the large language model to obtain the trained large model of image and text recognition, thereby improving the image recognition accuracy of the large model of image and text recognition.

[0045] Continue to refer Figure 3 , Figure 3 A process 300 of another embodiment of a method for training a large model for image and text recognition according to the present disclosure is shown. The method for training a large model for image and text recognition includes the following steps:

[0046] Step 301: Obtain a training sample set.

[0047] In this embodiment, the execution subject of the training method of the large image and text recognition model (for example Figure 1 The server 105 shown in the figure will obtain a training sample set, wherein the training samples in the training sample set include a sample image, a prompt word corresponding to the sample image, a face recognition result corresponding to the sample image, and image description information of the sample image. Step 301 is basically the same as step 201 in the above embodiment. The specific implementation method can refer to the above description of step 201, which will not be repeated here.

[0048] Step 302: input the sample image into a pre-trained face recognition model, and output a face recognition result.

[0049] In this embodiment, the execution subject inputs the sample image into the pre-trained face recognition model and outputs the face recognition result, wherein the face recognition model includes: a face feature extraction model and a face feature library, and the face recognition result includes: face feature information and character name information. Specifically, the face feature extraction model can extract the face features of the sample image, output the face feature information, and then determine the character name information based on the face feature information and the face feature library. Thus, the face feature information and the character name are accurately determined based on the face recognition model.

[0050] Step 303, concatenate the semantic feature information and the facial feature information to obtain target feature information.

[0051] In this embodiment, the execution subject will splice the semantic feature information and the facial feature information to obtain the target feature information. That is, the semantic feature information of the picture is spliced ​​with the facial feature information of the person in the picture to obtain the target feature information. By splicing the facial features and the features of the picture itself together, the understanding of the main content of the picture can be guaranteed without losing the information of the important people in the picture.

[0052] Step 304, concatenate the prompt and the character name information to obtain the target text information.

[0053] In this embodiment, the execution entity will splice the prompt and the character name information to obtain the target text information. Since the prompt is the information input by the user for the current picture, and the character name information is the text information describing the person in the picture, the two text information are spliced ​​together and integrated into a fixed format to obtain complete text information, that is, the target text information. For example, if the prompt is "What's in the picture?" and the character name information is "Zhang San", then the target text information is "What's in the picture? It is known that the picture contains Zhang San."

[0054] Step 305 , taking the target feature information and the target text information as input and the image description information as output, training the pre-built large language model to obtain a large image-text recognition model.

[0055] In this embodiment, the above-mentioned execution entity will take the target feature information and the target text information as input, and the image description information as output, and train the pre-built large language model to obtain a large image and text recognition model.

[0056] from Figure 3 It can be seen that Figure 2 Compared with the corresponding embodiments, the training method of the image and text recognition large model in this embodiment uses the face recognition model to obtain the facial feature information and character name information corresponding to the sample image, and trains the image and text recognition large model based on the above information, thereby improving the face recognition accuracy and recognition accuracy of the image and text recognition large model.

[0057] Continue to refer Figure 4 , Figure 4 A process 400 of another embodiment of a method for training a large model for image and text recognition according to the present disclosure is shown. The method for training a large model for image and text recognition includes the following steps:

[0058] Step 401: Obtain a training sample set.

[0059] In this embodiment, the execution subject of the training method of the large image and text recognition model (for example Figure 1 The server 105 shown in the figure will obtain a training sample set, wherein the training samples in the training sample set include a sample image, a prompt word corresponding to the sample image, a face recognition result corresponding to the sample image, and image description information of the sample image. Step 401 is basically the same as step 301 in the above embodiment. The specific implementation method can refer to the above description of step 301, which will not be repeated here.

[0060] Step 402: Input the sample image into a pre-trained facial feature extraction model, and output facial feature information.

[0061] In this embodiment, the execution subject inputs the sample image into a pre-trained facial feature extraction model, and outputs facial feature information of the sample image. The facial feature extraction model is a model that can extract facial features, such as a pre-trained model ResNet50. The facial feature extraction model is used to extract facial features of the sample image, thereby obtaining facial feature information.

[0062] In some optional implementations of the present embodiment, the facial feature extraction model is trained through the following steps: training the initial network model using the first facial data set to obtain a pre-trained model; fine-tuning the pre-trained model using the second facial data set to obtain a facial recognition and classification model; removing the last fully connected layer of the facial classification model to obtain a facial feature extraction model.

[0063] In this implementation, the execution subject first obtains the first face data set and the second face data set, wherein the first face data set is generally a training data set including common face images, which may be DeepGlint-Face, VGGFACE2, etc., and the second face data set is a training data set including face images of public figures, and the second face data set is generally a self-built data set for the current task. Then, the standard ResNet50 is used as the feature extractor of the face image, and Arcface Loss (Additive Angular Margin Loss) is used as the network supervision loss function, and the initial network (ResNet50) is trained using the first face data set to obtain a pre-trained model. Afterwards, the pre-trained model is fine-tuned using the second face data set to obtain a face classification model. Finally, the last fully connected layer of the face classification model is removed, that is, the last fully connected layer of the trained ResNet50 network is removed to obtain a face feature extraction model. Thus, the trained face feature extraction model can accurately extract the feature information of the face.

[0064] In some optional implementations of the present embodiment, the training method of the above-mentioned large image and text recognition model also includes: obtaining a first initial data set and a second initial data set; using a face detection network to generate facial key points of the images in the first initial data set and the second initial data set; aligning the facial key points through similarity transformation processing to obtain an aligned first initial data set and an aligned second initial data set, which are recorded as the first face data set and the second face data set.

[0065] In this implementation, the execution subject will first obtain the initial data set, that is, the first initial data set and the second initial data set, and pre-process the first initial data set and the second initial data set, that is, perform face alignment. Specifically, the face detection network is used to generate the face key points of the pictures in the first initial data set and the second initial data set. The face detection network here can be MTCNN (Multi-task convolutional neural network), that is, the MTCNN model is used to detect the face key points. Then, the normalized aligned face photos after trimming are obtained through similarity transformation, and the size is 256*256, so as to obtain the aligned first initial data set, that is, the first face data set, and the aligned second initial data set, that is, the second face data set. Thus, the initial data sets are subjected to face alignment processing to obtain the first face data set and the second face data set.

[0066] Step 403, determining the person's name information based on the facial feature information and the facial feature library.

[0067] In this embodiment, the above-mentioned execution entity will determine the character name information of the sample image based on the facial feature information and a pre-built facial feature library. The character name information here is the name of the character, and the pre-built facial feature library stores the facial features of a large number of characters, and also stores the names of different characters. Here, the above-mentioned execution entity will match the facial feature information with the facial features in the facial feature library. If the match is successful, the name corresponding to the successfully matched facial feature is the character name information.

[0068] In some optional implementations of this embodiment, step 403 includes: calculating the similarity between the facial feature information and each feature vector in the facial feature library; in response to determining that the similarity exceeds a preset threshold, using the name of the person in the picture corresponding to the feature vector as the person name information.

[0069] In this implementation, the execution subject calculates the similarity between the facial feature information and each facial feature in the facial feature library, and determines whether the similarity value is greater than a preset threshold. If so, the name of the person in the picture corresponding to the feature vector is used as the person name information. The preset threshold can be set according to actual conditions, for example, it can be 0.85, which is not specifically limited in this embodiment. Thus, the person name information is accurately determined.

[0070] In some optional implementations of this embodiment, the facial feature library is obtained based on the following steps: obtaining a third facial data set; using a facial feature extraction model to extract facial features of each image in the third data set to obtain a facial feature library.

[0071] In this implementation, the execution subject will obtain a third face data set, which includes a large number of face images of public figures, historical figures, and various celebrities collected on the Internet. The third face data set can be the second face data set, or it can include other face data except the second face data set. Then, the face feature extraction model is used to extract the face features of each image in the third data set, thereby obtaining a face feature library. As a result, the constructed face feature library contains the face features of a large number of people.

[0072] Step 404, concatenate the semantic feature information and the facial feature information to obtain target feature information.

[0073] In this embodiment, the execution subject will splice the semantic feature information and the facial feature information to obtain the target feature information. That is, the semantic feature information of the picture is spliced ​​with the facial feature information of the person in the picture to obtain the target feature information. By splicing the facial features and the features of the picture itself together, the understanding of the main content of the picture can be guaranteed without losing the information of the important people in the picture.

[0074] Step 405, concatenate the prompt and the character name information to obtain the target text information.

[0075] In this embodiment, the execution entity will splice the prompt and the character name information to obtain the target text information. Since the prompt is the information input by the user for the current picture, and the character name information is the text information describing the person in the picture, the two text information are spliced ​​together and integrated into a fixed format to obtain complete text information, that is, the target text information. For example, if the prompt is "What's in the picture?" and the character name information is "Zhang San", then the target text information is "What's in the picture? It is known that the picture contains Zhang San."

[0076] Step 406, taking the target feature information and the target text information as input and the image description information as output, training the pre-built large language model to obtain a large image-text recognition model.

[0077] In this embodiment, the above-mentioned execution entity will take the target feature information and the target text information as input, and the image description information as output, and train the pre-built large language model to obtain a large image and text recognition model.

[0078] from Figure 4 It can be seen that Figure 3Compared with the corresponding embodiments, the training method of the large image and text recognition model in this embodiment combines the features of the human face with the features of the image itself, which can not only ensure the understanding of the main content of the image, but also will not lose the information of important people in the image, thereby further improving the image and text recognition precision and recognition accuracy of the large image and text recognition model.

[0079] Continue to refer Figure 5 , which shows a process 500 of an embodiment of a method for image and text recognition according to the present disclosure. The method for image and text recognition comprises the following steps:

[0080] Step 501, obtain the image to be identified and the prompt corresponding to the image to be identified.

[0081] In this embodiment, the execution subject of the image and text recognition method (for example Figure 1 The server 105 shown in the figure will obtain the image to be identified and the prompt corresponding to the image to be identified. The image to be identified can be the image that the user wants to identify, and the image to be identified can include information such as people. The prompt is the identification requirement information input by the user, that is, what information the user wants to know about the image to be identified, for example, the prompt can be "What's in the picture?"

[0082] Step 502: determine the face recognition result of the image to be recognized based on the face recognition model.

[0083] In this embodiment, the above-mentioned execution subject determines the face recognition result of the image to be recognized based on the face recognition model. Here, the face recognition model includes a face feature extraction model and a face feature library. The face feature extraction model can extract face features in the image. The face feature library stores a large number of face features of people. The face feature extraction model can be trained by the method described in the above embodiment, and the face feature library can be constructed by the method described in the above embodiment. The face recognition result includes face feature information and person name information.

[0084] Specifically, the execution subject inputs the image to be identified into the face feature extraction model, and then outputs the face feature information of the image to be identified. Then the execution subject matches the face feature information with the face features in the face feature library. If the match is successful, the name corresponding to the successfully matched face feature is the character name information.

[0085] The face recognition model outputs a face feature vector and the name of the corresponding person; or if there is no public figure in the face feature database in the image, a null value will be output.

[0086] Step 503, input the image to be recognized, prompt and face recognition result into the image and text recognition model, and output the image description information of the image to be recognized.

[0087] In this embodiment, the above-mentioned execution subject will input the image to be recognized, prompt and face recognition results into the image and text recognition large model, and output the image description information of the image to be recognized, wherein the image and text recognition large model is trained using the method described in the above embodiment. The image and text recognition large model includes: an image processing module, an image and text semantic alignment module and a large language model. The image processing module is used to extract the image features of the image to be recognized, the image and text semantic alignment module is used to output the text information of the image to be recognized, and the large language model is used to output the corresponding image description information based on the overall information of the image to be recognized.

[0088] The image description information obtained by the image-text recognition model is the answer content corresponding to the prompt, including the complete description information of the image, not just the name of the person. For example, if the prompt is "What's in the picture?", the image description information obtained by the image-text recognition model is "Zhang San in the picture is giving a speech."

[0089] The image and text recognition method provided in the embodiment of the present disclosure first obtains the image to be recognized and the prompt corresponding to the image to be recognized; then, the face recognition result of the image to be recognized is determined based on the face recognition model; finally, the image to be recognized, the prompt and the face recognition result are input into the image and text recognition large model, and the image description information of the image to be recognized is output. The image and text recognition method in this embodiment can use the image and text recognition large model to accurately generate complete description information of the image, thereby improving the efficiency and accuracy of image and text recognition.

[0090] Continue to refer Figure 6 , which shows a process 600 of another embodiment of the image and text recognition method according to the present disclosure. The image and text recognition method comprises the following steps:

[0091] Step 601, obtain the image to be identified and the prompt corresponding to the image to be identified.

[0092] In this embodiment, the execution subject of the image and text recognition method (for example Figure 1 The server 105 shown in the figure will obtain the image to be identified and the prompt corresponding to the image to be identified. Step 601 is basically the same as step 501 in the above embodiment. The specific implementation method can refer to the above description of step 501, which will not be repeated here.

[0093] Optionally, since the facial feature extraction model and the image processing module have different requirements for the size of the input image, the above execution body will change the image to be recognized to a fixed size of 256*256 and 448*448 respectively.

[0094] Step 602: input the image to be identified into a facial feature extraction model, and output facial feature information of the image to be identified.

[0095] In this embodiment, the execution subject will input the image to be identified into the face feature extraction model, that is, the 256*256 image is input into the face feature extraction model, so that the face feature model extracts the face feature information of the image to be identified, and outputs the extracted face feature information. The output is a feature vector with a dimension of 512 and the name of the corresponding person.

[0096] Step 603, based on the facial feature information and the similarity of each feature vector in the facial feature library, determine the name information of the person in the image to be identified.

[0097] In this embodiment, the above-mentioned execution subject will determine the name information of the person in the picture to be identified based on the similarity between the facial feature information and each feature vector in the facial feature library. That is, after obtaining the facial feature information, the above-mentioned execution subject will calculate the similarity between the facial feature information and each facial feature in the facial feature library, and determine whether the similarity value is greater than a preset threshold value. If it is greater, the name of the person in the picture corresponding to the feature vector is used as the name information of the person in the picture to be identified. The preset threshold value can be set according to the actual situation, for example, it can be 0.85, and this is not specifically limited in this embodiment. Thus, the face recognition result of the picture to be identified is obtained based on the above steps.

[0098] Step 604: input the image to be identified into the image processing module, and output the image feature information of the image to be identified.

[0099] In this embodiment, the execution subject inputs the image to be identified into the image processing module, and outputs the image feature information of the image to be identified.

[0100] Step 605: input the image feature information into the image-text semantic alignment module, and output the semantic feature information of the image to be identified.

[0101] In this embodiment, the execution subject inputs the image feature information into the image-text semantic alignment module, and outputs the semantic feature information of the image to be identified.

[0102] Step 606, concatenate the semantic feature information with the facial feature information to obtain target feature information.

[0103] In this embodiment, the execution subject will combine the semantic feature information with the facial feature information to obtain the target feature information, thereby combining the facial features with the features of the image itself, which can ensure the understanding of the main content of the image without losing the information of the important characters.

[0104] Step 607, concatenate the character name information with the prompt to obtain the target text information.

[0105] In this embodiment, the execution subject will concatenate the character name information with the prompt to obtain the target text information.

[0106] Step 608: input the target feature information and the target text information into the large language model, and output the image description information of the image to be identified.

[0107] In this embodiment, the execution subject inputs the target feature information and the target text information into the large language model, thereby outputting the image description information of the image to be identified.

[0108] from Figure 6 It can be seen that Figure 5 Compared with the corresponding embodiments, the image and text recognition method in this embodiment combines the features of the face with the features of the image itself, which can not only ensure the understanding of the main content of the image, but also will not lose the information of important people in the image, thereby further improving the recognition accuracy and recognition accuracy of image and text recognition.

[0109] Continue to refer Figure 7 , Figure 7 An application scenario of the image-text recognition method according to the present disclosure is shown. In this application scenario, for the input image image, the execution subject will first change it to a fixed size, that is, 256*256 and 448*448.

[0110] Then, the 256*256 image is fed into the face recognition model for face recognition (facial recognition), thereby obtaining a 512-dimensional feature vector (face feature vector) and the corresponding person's name "XXX". At the same time, the input prompt "What's in the picture?" corresponding to the image entered by the user is obtained, and the name and the input prompt are concatenated to obtain the target text information "It is known that the picture contains XXX. What's in the picture?".

[0111] At the same time, the original 448*448 image is input into the image encoder in the VIT model to extract the overall features of the image and obtain a feature vector (image feature vector) with a dimension of 768. The 768-dimensional feature vector is input into the Bert model, and after cross-calculation of the self-attention layer and the cross-attention layer, 32 vectors are obtained, each of which is 768 dimensions.

[0112] Afterwards, the above 512-dimensional feature vector (face feature vector) is mapped to 768 dimensions through a simple linear layer (linear layer), and then concatenated with the above 32 vectors (image feature vectors) to obtain a feature vector of size (32+1)*768, which is recorded as the target feature vector.

[0113] Finally, the above target feature vector and target text information are input into the decoding layer of the large language model LLM, so as to output detailed description information of the image.

[0114] Further references Figure 8 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a training device for a large model of image and text recognition. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0115] like Figure 8 As shown, the training device 800 of the large model of image and text recognition of this embodiment includes: a first acquisition module 801, a first output module 802, a second output module 803 and a first training module 804. The first acquisition module 801 is configured to acquire a training sample set, wherein the training samples in the training sample set include a sample image, a prompt word corresponding to the sample image, a face recognition result corresponding to the sample image, and a picture description information of the sample image; the first output module 802 is configured to input the sample image into the image processing module, and output the picture feature information of the sample image; the second output module 803 is configured to input the picture feature information into the image-text semantic alignment module, and output the semantic feature information of the sample image; the first training module 804 is configured to take the semantic feature information, the face recognition result and the prompt as input, and take the picture description information as output, to train the pre-constructed large language model, and obtain the large model of image and text recognition.

[0116] In the training device 800 for the large model of image and text recognition, the specific processing of the first acquisition module 801, the first output module 802, the second output module 803 and the first training module 804 and the technical effects thereof can be referred to in detail. Figure 2The relevant descriptions of steps 201 - 204 in the corresponding embodiment are not repeated here.

[0117] In some optional implementations of the present embodiment, the training device 800 of the above-mentioned large image and text recognition model also includes a face generation module for generating face recognition results, and the face generation module includes: a face generation sub-module, which is configured to input a sample image into a pre-trained face recognition model, and output a face recognition result, wherein the face recognition model includes: a face feature extraction model and a face feature library, and the face recognition result includes: face feature information and character name information.

[0118] In some optional implementations of this embodiment, the face generation submodule includes: an output unit, configured to input the sample image into a face feature extraction model and output face feature information; a determination unit, configured to determine the character name information based on the face feature information and the face feature library.

[0119] In some optional implementations of this embodiment, the determination submodule is further configured to: calculate the similarity between the facial feature information and each feature vector in the facial feature library; in response to determining that the similarity exceeds a preset threshold, use the name of the person in the picture corresponding to the feature vector as the person name information.

[0120] In some optional implementations of this embodiment, the training device 800 of the above-mentioned large image and text recognition model also includes: a splicing module, which is configured to splice the semantic feature information and the facial feature information to obtain the target feature information.

[0121] In some optional implementations of this embodiment, the first training module is further configured to: concatenate the prompt and character name information to obtain the target text information; take the target feature information and the target text information as input, and take the image description information as output, train the pre-built large language model, and obtain a large image and text recognition model.

[0122] In some optional implementations of the present embodiment, the training device 800 of the above-mentioned large image and text recognition model also includes a second training module for training a facial feature extraction model, and the second training module is configured to: train the initial network model using the first facial data set to obtain a pre-trained model; fine-tune the pre-trained model using the second facial data set to obtain a facial recognition model; remove the last fully connected layer of the facial recognition model to obtain a facial feature extraction model.

[0123] In some optional implementations of the present embodiment, the training device 800 of the above-mentioned large image and text recognition model also includes: a second acquisition module, configured to acquire a first initial data set and a second initial data set; a key point generation module, configured to use a face detection network to generate facial key points of images in the first initial data set and the second initial data set; an alignment module, configured to align the facial key points through similarity transformation processing to obtain an aligned first initial data set and an aligned second initial data set, which are recorded as the first face data set and the second face data set.

[0124] In some optional implementations of the present embodiment, the training device 800 of the above-mentioned large image and text recognition model also includes a construction module for constructing a facial feature library, and the construction module is configured to: obtain a third facial data set; use a facial feature extraction model to extract facial features of each picture in the third data set to obtain a facial feature library.

[0125] Further references Fig. 9 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a graphic text recognition device. Figure 5 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0126] like Fig. 9 As shown, the image and text recognition device 900 of this embodiment includes: a third acquisition module 901, a determination module 902 and a third output module 903. Among them, the third acquisition module 901 is configured to obtain the image to be recognized and the prompt corresponding to the image to be recognized; the determination module 902 is configured to determine the face recognition result of the image to be recognized based on the face recognition model; the third output module 903 is configured to input the image to be recognized, the prompt and the face recognition result into the image and text recognition model, and output the image description information of the image to be recognized.

[0127] In this embodiment, in the image and text recognition device 900, the specific processing of the third acquisition module 901, the determination module 902 and the third output module 903 and the technical effects thereof can be referred to in Figure 5 The relevant descriptions of steps 501-503 in the corresponding embodiment are not repeated here.

[0128] In some optional implementations of this embodiment, the face recognition model includes: a face feature extraction model and a face feature library, the face recognition result includes: face feature information and character name information; and the determination module is further configured to: input the picture to be identified into the face feature extraction model, and output the face feature information of the picture to be identified; based on the similarity between the face feature information and each feature vector in the face feature library, determine the character name information of the picture to be identified.

[0129] In some optional implementations of the present embodiment, the large model for image and text recognition includes: an image processing module, an image-text semantic alignment module and a large language model; and the third output module is further configured to: input the picture to be recognized into the image processing module, and output the picture feature information of the picture to be recognized; input the picture feature information into the image-text semantic alignment module, and output the semantic feature information of the picture to be recognized; splice the semantic feature information with the face feature information to obtain the target feature information; splice the character name information with the prompt to obtain the target text information; input the target feature information and the target text information into the large language model, and output the picture description information of the picture to be recognized.

[0130] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0131] Fig.10 A schematic block diagram of an example electronic device 1000 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0132] like Fig.10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0133] A number of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0134] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as the training method of the large model of image and text recognition or the image and text recognition method. For example, in some embodiments, the training method of the large model of image and text recognition or the image and text recognition method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via ROM 1002 and / or the communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the training method of the large model of image and text recognition or the image and text recognition method described above may be executed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute the training method of the large image and text recognition model or the image and text recognition method in any other appropriate manner (for example, by means of firmware).

[0135] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0136] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0137] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0138] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0139] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0140] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0141] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0142] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A training method for a large image and text recognition model, comprising: Acquire a training sample set, wherein the training samples in the training sample set include a sample image, a prompt word corresponding to the sample image, a face recognition result corresponding to the sample image, and image description information of the sample image, wherein the face recognition result includes: face feature information and character name information; Input the sample image into an image processing module, and output image feature information of the sample image; Inputting the image feature information into an image-text semantic alignment module, and outputting the semantic feature information of the sample image; Taking the semantic feature information, the face recognition result and the prompt as input, taking the picture description information as output, training the pre-built large language model, and obtaining a large picture and text recognition model; The step of taking the semantic feature information, the face recognition result and the prompt as input includes: Concatenating the semantic feature information with the facial feature information to obtain target feature information; Concatenate the prompt and the character name information to obtain target text information; The target feature information and the target text information are taken as input.

2. The method according to claim 1, wherein: The face recognition result is obtained by the following steps: The sample image is input into a pre-trained face recognition model, and the face recognition result is output, wherein the face recognition model includes: a face feature extraction model and a face feature library.

3. The method according to claim 2, wherein: The step of inputting the sample image into a pre-trained face recognition model and outputting the face recognition result includes: Input the sample image into the face feature extraction model, and output the face feature information; The character name information is determined based on the facial feature information and the facial feature library.

4. The method according to claim 3, wherein: The determining the character name information based on the facial feature information and the facial feature library includes: Calculating the similarity between the facial feature information and each feature vector in the facial feature library; In response to determining that the similarity exceeds a preset threshold, the name of the person in the picture corresponding to the feature vector is used as the person name information.

5. The method according to claim 3, wherein: The face feature extraction model is trained by the following steps: Using the first face data set to train the initial network model to obtain a pre-trained model; Using the second face data set to fine-tune the pre-trained model to obtain a face classification model; The last fully connected layer of the face classification model is removed to obtain the face feature extraction model.

6. The method according to claim 5, wherein: Also includes: Acquire a first initial data set and a second initial data set; Generate facial key points of the pictures in the first initial data set and the second initial data set using a face detection network; The facial key points are aligned through similarity transformation processing to obtain an aligned first initial data set and an aligned second initial data set, which are recorded as a first facial data set and a second facial data set.

7. The method according to claim 6, wherein: The face feature library is obtained based on the following steps: Get the third face dataset; The facial feature extraction model is used to extract facial features of each image in the third facial data set to obtain the facial feature library.

8. A method for image and text recognition, comprising: Get the image to be identified and the prompt corresponding to the image to be identified; Determine the face recognition result of the image to be recognized based on the face recognition model; The image to be identified, the prompt and the face recognition result are input into a large image and text recognition model, and the image description information of the image to be identified is output, wherein the large image and text recognition model is trained using the method described in any one of claims 1-7.

9. The method according to claim 8, wherein: The face recognition model includes: a face feature extraction model and a face feature library, and the face recognition result includes: face feature information and character name information; and The determining the face recognition result of the image to be recognized based on the face recognition model includes: Input the image to be identified into the face feature extraction model, and output the face feature information of the image to be identified; Based on the similarity between the facial feature information and each feature vector in the facial feature library, the name information of the person in the image to be identified is determined.

10. The method according to claim 9, wherein: The large image-text recognition model includes: an image processing module, an image-text semantic alignment module and a large language model; and The step of inputting the image to be identified, the prompt, and the face recognition result into the image-text recognition model, and outputting the image description information of the image to be identified includes: Input the image to be identified into the image processing module, and output the image feature information of the image to be identified; Inputting the picture feature information into the image-text semantic alignment module, and outputting the semantic feature information of the picture to be identified; Concatenating the semantic feature information with the facial feature information to obtain target feature information; Concatenate the character name information with the prompt to obtain target text information; The target feature information and the target text information are input into the large language model, and the image description information of the image to be identified is obtained as output.

11. A training device for a large model of image and text recognition, comprising: A first acquisition module is configured to acquire a training sample set, wherein the training samples in the training sample set include a sample image, a prompt word corresponding to the sample image, a face recognition result corresponding to the sample image, and image description information of the sample image, wherein the face recognition result includes: face feature information and character name information; A first output module is configured to input the sample image into an image processing module and output image feature information of the sample image; A second output module is configured to input the image feature information into an image-text semantic alignment module, and output the semantic feature information of the sample image; The first training module is configured to take the semantic feature information, the face recognition result and the prompt as input, take the picture description information as output, train the pre-built large language model, and obtain a large picture and text recognition model; The first training module is further configured to: Concatenating the semantic feature information with the facial feature information to obtain target feature information; Concatenate the prompt and the character name information to obtain target text information; The target feature information and the target text information are taken as input.

12. The device according to claim 11, wherein The device also includes a face generation module for generating a face recognition result, and the face generation module includes: The face generation submodule is configured to input the sample image into a pre-trained face recognition model and output the face recognition result, wherein the face recognition model includes: a face feature extraction model and a face feature library.

13. The device according to claim 12, wherein: The face generation submodule comprises: An output unit, configured to input the sample image into the face feature extraction model and output the face feature information; The determination unit is configured to determine the character name information based on the facial feature information and the facial feature library.

14. The device according to claim 13, wherein: The determining unit is further configured to: Calculating the similarity between the facial feature information and each feature vector in the facial feature library; In response to determining that the similarity exceeds a preset threshold, the name of the person in the picture corresponding to the feature vector is used as the person name information.

15. The device according to claim 13, wherein: The apparatus further comprises a second training module for training a facial feature extraction model, wherein the second training module is configured to: Using the first face data set to train the initial network model to obtain a pre-trained model; Using the second face data set to fine-tune the pre-trained model to obtain a face classification model; The last fully connected layer of the face classification model is removed to obtain the face feature extraction model.

16. The device according to claim 15, wherein: Also includes: A second acquisition module is configured to acquire a first initial data set and a second initial data set; A key point generation module, configured to generate face key points of the pictures in the first initial data set and the second initial data set using a face detection network; The alignment module is configured to align the facial key points through similarity transformation processing to obtain an aligned first initial data set and an aligned second initial data set, which are recorded as a first facial data set and a second facial data set.

17. The device according to claim 11, wherein: The apparatus further comprises a building module for building a face feature library, wherein the building module is configured to: Get the third face dataset; The facial feature extraction model is used to extract facial features of each image in the third facial data set to obtain the facial feature library.

18. A graphic recognition device, comprising: A third acquisition module is configured to acquire a picture to be identified and a prompt corresponding to the picture to be identified; A determination module is configured to determine a face recognition result of the image to be recognized based on a face recognition model; The third output module is configured to input the image to be identified, the prompt and the face recognition result into a large image and text recognition model, and output image description information of the image to be identified, wherein the large image and text recognition model is trained using the method described in any one of claims 1-7.

19. The device according to claim 18, wherein: The face recognition model includes: a face feature extraction model and a face feature library, and the face recognition result includes: face feature information and character name information; and The determination module is further configured to: Input the image to be identified into the face feature extraction model, and output the face feature information of the image to be identified; Based on the similarity between the facial feature information and each feature vector in the facial feature library, the name information of the person in the image to be identified is determined.

20. The device according to claim 19, wherein The large image-text recognition model includes: an image processing module, an image-text semantic alignment module and a large language model; and The third output module is further configured to: Input the image to be identified into the image processing module, and output the image feature information of the image to be identified; Inputting the picture feature information into the image-text semantic alignment module, and outputting the semantic feature information of the picture to be identified; Concatenating the semantic feature information with the facial feature information to obtain target feature information; Concatenate the character name information with the prompt to obtain target text information; The target feature information and the target text information are input into the large language model, and the image description information of the image to be identified is obtained as output.

21. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7 or 8-10.

22. A non-transitory computer-readable storage medium storing computer instructions, the computer instructions being used to cause the computer to execute the method of any one of claims 1-7 or 8-10.

23. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1-7 or 8-10.

Images