Training method, device, equipment and storage medium for large image and text recognition model

Abstract

The present disclosure provides a training method, device, equipment and storage medium for a large model of image and text recognition, which relates to the field of artificial intelligence technology, specifically to the technical fields of computer vision, deep learning, large models, etc., and can be applied to scenarios such as artificial intelligence content generation. The method includes: obtaining a training sample set, wherein the training samples in the training sample set include an initial sample image, a prompt word prompt corresponding to the initial sample image, and image description information corresponding to the initial sample image; scaling and segmenting the initial sample image to obtain multiple sample images; inputting the multiple sample images into an image processing conversion module, and outputting the semantic feature information corresponding to the initial sample image; using the semantic feature information and prompt as input, and the image description information as output, training a pre-constructed large language model to obtain a large model of image and text recognition.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically to technical fields such as computer vision, deep learning, and large models, and in particular to a training method, device, equipment, and storage medium for a large image and text recognition model, which can be applied to scenarios such as artificial intelligence content generation. Background Art

[0002] As large language models have demonstrated extraordinary capabilities in various fields, multimodal large language models (MLLM) have gradually shown greater potential. Multimodal large language models can process and understand many different types of data (such as text, images, sounds, etc.) to obtain a more comprehensive and in-depth understanding of information. Among them, the large picture understanding model that understands visual content is the most widely used. By understanding the input image, it can obtain richer input forms and more accurately understand the user's intentions. In the fields of automatic image annotation, video content analysis, real-time monitoring systems, etc., multimodal large language models can provide more accurate and in-depth analysis. At the same time, in applications such as analyzing user behavior and content recommendation, they can also better understand complex user interaction content. Summary of the invention

[0003] The present invention provides a training method, device, equipment and storage medium for a large image and text recognition model.

[0004] According to a first aspect of the present disclosure, a training method for a large model for image and text recognition is provided, comprising: obtaining a training sample set, wherein the training samples in the training sample set include an initial sample image, a prompt word prompt corresponding to the initial sample image, and image description information corresponding to the initial sample image; scaling and segmenting the initial sample image to obtain a plurality of sample images; inputting the plurality of sample images into an image processing conversion module, and outputting semantic feature information corresponding to the initial sample image; taking the semantic feature information and prompt as input, and taking the image description information as output, to train a pre-constructed large language model to obtain the large model for image and text recognition.

[0005] According to a second aspect of the present disclosure, a method for image and text recognition is provided, including: obtaining a picture to be recognized and a prompt corresponding to the picture to be recognized; scaling and segmenting the picture to be recognized to obtain multiple target pictures; inputting the multiple target pictures and the prompt into a large image and text recognition model, and outputting picture description information of the picture to be recognized, wherein the large image and text recognition model is trained using the method described in the first aspect.

[0006] According to a third aspect of the present disclosure, a training device for a large model of image and text recognition is provided, comprising: a first acquisition module, configured to acquire a training sample set, wherein the training samples in the training sample set include an initial sample image, a prompt word prompt corresponding to the initial sample image, and image description information corresponding to the initial sample image; a first segmentation module, configured to scale and segment the initial sample image to obtain a plurality of sample images; an input module, configured to input the plurality of sample images into an image processing conversion module, and output semantic feature information corresponding to the initial sample image; a training module, configured to take the semantic feature information and prompt as input, and take the image description information as output, to train a pre-constructed large language model to obtain the large model of image and text recognition.

[0007] According to a fourth aspect of the present disclosure, a graphic and text recognition device is provided, comprising: a second acquisition module, configured to acquire a picture to be recognized and a prompt corresponding to the picture to be recognized; a second segmentation module, configured to scale and segment the picture to be recognized to obtain multiple target pictures; a second output module, configured to input the multiple target pictures and the prompt into a large graphic and text recognition model, and output picture description information of the picture to be recognized, wherein the large graphic and text recognition model is trained using the method described in the first aspect.

[0008] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in any implementation manner of the first aspect or the second aspect.

[0009] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to execute the method described in any implementation manner of the first aspect or the second aspect.

[0010] According to a seventh aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the computer program implements the method described in any implementation manner of the first aspect or the second aspect.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0013] Figure 1 is an exemplary system architecture diagram in which the present disclosure may be applied;

[0014] Figure 2 is a flowchart of an embodiment of a training method for a large model of image and text recognition according to the present disclosure;

[0015] Figure 3 is a flowchart of another embodiment of a method for training a large model for image and text recognition according to the present disclosure;

[0016] Figure 4 is a flowchart of another embodiment of a training method for a large model of image and text recognition according to the present disclosure;

[0017] Figure 5 is a flow chart of another embodiment of a training method for a large model of image and text recognition according to the present disclosure;

[0018] Figure 6 is a flow chart of an embodiment of the image and text recognition method according to the present disclosure;

[0019] Figure 7 is a flow chart of another embodiment of the image and text recognition method according to the present disclosure;

[0020] Figure 8 is a schematic diagram of an application scenario of the image and text recognition method disclosed in the present invention;

[0021] Fig. 9 It is a structural schematic diagram of an embodiment of a training device for a large model of image and text recognition according to the present disclosure;

[0022] Fig.10 is a structural schematic diagram of an embodiment of an image and text recognition device according to the present disclosure;

[0023] Fig.11 It is a block diagram of an electronic device used to implement the training method of the large image and text recognition model or the image and text recognition method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0025] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0026] Figure 1 An exemplary system architecture 110 is shown to which an embodiment of the method for training a large image and text recognition model or the apparatus for training a large image and text recognition model of the present disclosure can be applied.

[0027] like Figure 1 As shown, the system architecture 110 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0028] The user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send information, etc. Various client applications can be installed on the terminal devices 101, 102, 103.

[0029] The terminal devices 101, 102, 103 may be hardware or software. When the terminal devices 101, 102, 103 are hardware, they may be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc. When the terminal devices 101, 102, 103 are software, they may be installed in the above electronic devices. They may be implemented as multiple software or software modules, or as a single software or software module. No specific limitation is made here.

[0030] The server 105 can provide various services. For example, the server 105 can analyze and process the training sample sets obtained from the terminal devices 101, 102, and 103, and generate processing results (such as a large model for image and text recognition).

[0031] It should be noted that the server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server 105 is software, it can be implemented as multiple software or software modules (for example, for providing distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0032] It should be noted that the training method of the large image and text recognition model provided in the embodiment of the present disclosure is generally executed by the server 105 , and accordingly, the training device of the large image and text recognition model is generally set in the server 105 .

[0033] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0034] Continue to refer Figure 2 , which shows a process 200 of an embodiment of a training method for a large image and text recognition model according to the present disclosure. The training method for a large image and text recognition model includes the following steps:

[0035] Step 201: Obtain a training sample set.

[0036] In this embodiment, the execution subject of the training method of the large image-text recognition model (for example Figure 1 The server 105 shown in the figure obtains a training sample set, wherein the training samples in the training sample set include an initial sample image, a prompt word prompt corresponding to the initial sample image, and image description information corresponding to the initial sample image. The training sample set includes multiple training samples, and each training sample includes an initial sample image, a prompt word prompt corresponding to the initial sample image, and image description information of the initial sample image.

[0037] Here, the initial sample images can be images collected on the Internet, which can be landscape images, face images, food images, etc. The prompt corresponding to the initial sample images is the content entered by the user during the search. For example, for a certain image, the user enters the question "What's on the table?", then "What's on the table" is the prompt corresponding to the image. The image description information of the initial sample images is a complete description of the image, such as what items are on the table and what characteristics each item has.

[0038] Step 202: scaling and segmenting the initial sample image to obtain multiple sample images.

[0039] In this embodiment, the execution subject will scale and segment the initial sample image to obtain multiple sample images. Specifically, the execution subject will first scale the initial sample image to a target size to obtain a target sample image, and then segment the target sample image into N images of a preset size to obtain N sample images, where N = M 2 , M is an integer greater than 1, and N is an integer. That is, the execution subject first scales the initial sample image to a target size, and then divides the image of the target size to obtain N divided images, where the N images are all of the same preset size, and N = M 2 , M=1, 2, 3..., correspondingly, M is 1, 4, 9...

[0040] As an example, in this embodiment, a picture with a resolution of 448*448 is required, so the preset size is 448*448. Then the initial sample picture can be scaled to 1792*1792, where 1792*1792 is the target size, and the picture of the target size can be divided into 2 2 = 4 pictures with a resolution of 448*448, that is, M = 2, N = 4. That is, the relationship between the target size, the preset size, N, and M is: N = M 2 , target size = preset size * N.

[0041] Step 203: input a plurality of sample images into an image processing conversion module, and output semantic feature information corresponding to the initial sample images.

[0042] In this embodiment, the execution subject inputs multiple sample images into the image processing conversion module, and outputs the semantic feature information corresponding to the initial sample image. That is, multiple sample images are simultaneously input into the image processing conversion module, so that the image processing conversion module processes the multiple sample images and outputs the semantic feature information corresponding to the complete initial sample image.

[0043] Specifically, the image processing conversion module includes an image processing module and a feature fusion alignment module. That is, the above-mentioned execution subject will input multiple sample images into the image processing module, so that the image processing module extracts the image feature information of each sample image, thereby outputting the image feature information of each sample image. The image processing module can be a ViT (vision transformer) model. The ViT model is a text image pre-training model pre-trained on open source data. It applies the Transformer to the image classification model, which has good effect and strong scalability. Of course, the image processing module can also be other models with image feature extraction functions, such as the CLIP (Contrastive Language-Image Pre-Training) model, etc., which is not specifically limited in this embodiment.

[0044] Afterwards, the execution entity will splice the image feature information of each sample image, and input the spliced ​​feature information into the feature fusion alignment module, thereby outputting the semantic feature information corresponding to the initial sample image. The feature fusion alignment module can be a Bert (Bidirectional Encoder Representations from Transformers) model. Bert is a pre-trained language model with strong applicability. It only needs to fine-tune the model to be applied to a variety of tasks. In this embodiment, the Bert model is used to semantically compress the spliced ​​image feature information. Of course, the feature fusion alignment module can also be other models with image-text semantic alignment functions, such as a GPT (Generative Pre-Trained Transformer) model, etc., which is not specifically limited in this embodiment.

[0045] Step 204 , taking the semantic feature information and prompt as input and the image description information as output, trains the pre-built large language model to obtain a large image and text recognition model.

[0046] In this embodiment, the execution subject takes the semantic feature information and prompt as input and the image description information as output, trains the pre-built large language model, and obtains the image-text recognition large model. The trained image-text recognition large model can recognize the image and output the complete description information of the image.

[0047] It should be noted that a large language model (large model) is an artificial intelligence model designed to understand and generate human language. By training on large amounts of text data, it can perform a wide range of tasks, including text summarization, translation, sentiment analysis, and more. A large model refers to a deep neural network with more than 1 billion parameters that can process massive amounts of data and complete various complex tasks. With the continuous improvement of computer hardware performance and the continuous optimization of deep learning algorithms, the development of large models is also accelerating. The parameter scale of large models continues to expand, the training time is getting longer and longer, but the performance is also improving. Large models are usually based on deep learning architectures, such as transformers, which enable impressive capabilities in various natural language processing tasks. Common large models can include but are not limited to ChatGPT, GPT-4, ERNIE Bot (Wenxin Yiyan), etc.

[0048] The training method of the large model for image and text recognition provided by the embodiment of the present disclosure is as follows: first, a training sample set is obtained; then, the initial sample image is scaled and segmented to obtain multiple sample images; thereafter, the multiple sample images are input into the image processing conversion module, and the semantic feature information corresponding to the initial sample image is output; finally, the semantic feature information and prompt are used as input, and the image description information is used as output to train the pre-built large language model to obtain the large model for image and text recognition. The training method of the large model for image and text recognition in this embodiment is as follows: the method first scales and segments the initial sample image, and then uses the image processing conversion module to process the multiple sample images after segmentation to obtain the semantic feature information of the initial sample image before segmentation, and finally the semantic feature information and prompt are simultaneously input into the large language model for training to obtain the trained large model for image and text recognition, thereby improving the ability to recognize image details, and further improving the image recognition accuracy of the large model for image and text recognition.

[0049] Continue to refer Figure 3 , Figure 3 A process 300 of another embodiment of a method for training a large model for image and text recognition according to the present disclosure is shown. The method for training a large model for image and text recognition includes the following steps:

[0050] Step 301: Obtain a training sample set.

[0051] In this embodiment, the execution subject of the training method of the large image-text recognition model (for example Figure 1 The server 105 shown in the figure will obtain a training sample set, wherein the training samples in the training sample set include an initial sample image, a prompt word prompt corresponding to the initial sample image, and image description information corresponding to the initial sample image. Step 301 is basically the same as step 201 in the above embodiment. The specific implementation method can refer to the above description of step 201, which will not be repeated here.

[0052] Step 302: scaling the initial sample image to a target size to obtain a target sample image.

[0053] In this embodiment, the execution entity scales the initial sample image to a target size to obtain a scaled image, namely, a target sample image.

[0054] Step 303: divide the target sample image into N images of a preset size to obtain N sample images.

[0055] In this embodiment, the execution subject divides the target sample image into N images of a preset size to obtain N sample images, where N = M 2, M is an integer greater than 1, and N is an integer. That is, the execution subject first scales the initial sample image to a target size, and then divides the image of the target size to obtain N divided images, where the N images have the same preset size, and N = M 2 , M=1, 2, 3..., correspondingly M is 1, 4, 9...

[0056] As an example, in this embodiment, a picture with a resolution of 448*448 is required, so the preset size is 448*448. Then the initial sample picture can be scaled to 1792*1792, where 1792*1792 is the target size, and the picture of the target size can be divided into 2 2 = 4 pictures with a resolution of 448*448, that is, M = 2, N = 4. That is, the relationship between the target size, the preset size, N, and M is: N = M 2 , target size = preset size * N.

[0057] Through the above scaling and segmentation process, the model can learn the detailed information of the image.

[0058] Step 304: input N sample images into the image processing module, and output the image feature information of each sample image.

[0059] In this embodiment, the above-mentioned execution subject will input the segmented N sample images into the image processing module, so that the image processing module extracts the image feature information of each sample image, thereby outputting the image feature information of each sample image. The image processing module can be a ViT (vision transformer) model. The ViT model is a text image pre-training model pre-trained on open source data. It applies the Transformer to the image classification model, which has good effect and strong scalability. Of course, the image processing module can also be other models with image feature extraction functions, such as the CLIP (Contrastive Language-Image Pre-Training) model, etc., which is not specifically limited in this embodiment.

[0060] Step 305: input the image feature information of each sample image into a feature fusion alignment module, and output the semantic feature information corresponding to the initial sample image.

[0061] In the present embodiment, the above-mentioned execution entity will splice the image feature information of each sample image, and input the spliced ​​feature information into the feature fusion alignment module, so as to output the semantic feature information corresponding to the initial sample image. The feature fusion alignment module can be a Bert (Bidirectional Encoder Representations from Transformers) model. Bert is a pre-trained language model with strong applicability. It only needs to fine-tune the model to be applied to a variety of tasks. In the present embodiment, the Bert model is used to perform semantic compression on the spliced ​​image feature information. Of course, the feature fusion alignment module can also be other models with image-text semantic alignment functions, such as the GPT (Generative Pre-Trained Transformer) model, etc., which is not specifically limited in the present embodiment. Thereby, the semantic information of the complete image is generated based on the feature information of each segmented image.

[0062] Step 306, taking the semantic feature information and prompt as input and the image description information as output, training the pre-built large language model to obtain a large image and text recognition model.

[0063] In this embodiment, the execution subject uses the semantic feature information and prompt as input, and the image description information as output, and trains the pre-built large language model to obtain a large image and text recognition model. Step 306 is basically the same as step 204 of the above embodiment. The specific implementation method can refer to the above description of step 204, which will not be repeated here.

[0064] In some optional implementations of the present embodiment, the training method of the above-mentioned image and text recognition large model also includes: using the first data set to pre-train the initial image processing module and the initial feature fusion alignment module to obtain the image processing module and the pre-trained feature fusion alignment module; using the second data set to fine-tune the pre-trained feature fusion alignment module and the initial large language model to obtain the fine-tuned feature fusion alignment module and the fine-tuned large language model; using the third data set to fine-tune the fine-tuned feature fusion alignment module and the fine-tuned large language model again to obtain the feature fusion alignment module and the large language model.

[0065] In this implementation, the execution subject will first use the first data set to pre-train the initial image processing module and the initial feature fusion alignment module, and then use the second data set to fine-tune the pre-trained feature fusion alignment module and the initial large language model. After these two stages of small resolution (256*256) training, a new training process is added, that is, the fine-tuned feature fusion alignment module and the fine-tuned large language model are fine-tuned again using the third data set, so as to obtain the feature fusion alignment module and the large language model. Among them, the first data set is a data set containing many images, the second data set is a small number of data sets that have been finely annotated, and the third data set is a more accurate data set.

[0066] Therefore, only a small amount of training is needed to complete the alignment from low resolution to high resolution, thereby improving the ability to recognize image details.

[0067] from Figure 3 It can be seen that Figure 2 Compared with the corresponding embodiments, the training method of the large image and text recognition model in this embodiment highlights the steps of scaling and segmenting the initial sample images and obtaining semantic feature information, thereby improving the model's ability to recognize image details.

[0068] Continue to refer Figure 4 , Figure 4 A process 400 of another embodiment of a method for training a large model for image and text recognition according to the present disclosure is shown. The method for training a large model for image and text recognition includes the following steps:

[0069] Step 401: Obtain a training sample set.

[0070] In this embodiment, the execution subject of the training method of the large image-text recognition model (for example Figure 1 The server 105 shown in the figure will obtain a training sample set, wherein the training samples in the training sample set include an initial sample image, a prompt word prompt corresponding to the initial sample image, and image description information corresponding to the initial sample image.

[0071] Step 402: scaling the initial sample image to a target size to obtain a target sample image.

[0072] In this embodiment, the execution entity scales the initial sample image to a target size to obtain a target sample image.

[0073] Step 403: divide the target sample image into N images of a preset size to obtain N sample images.

[0074] In this embodiment, the execution subject divides the target sample image into N images of a preset size to obtain N sample images, where N = M2 , M is an integer greater than 1, and N is an integer.

[0075] Step 404: input N sample images into the image processing module, and output the image feature information of each sample image.

[0076] In this embodiment, the execution subject inputs N sample images into the image processing module, and outputs the image feature information of each sample image.

[0077] Steps 401-404 are basically consistent with steps 301-304 of the aforementioned embodiment. For specific implementation methods, reference may be made to the aforementioned description of steps 301-304, which will not be repeated here.

[0078] Step 405: concatenate the image feature information of each sample image to obtain first feature information.

[0079] In this embodiment, the execution entity splices the image feature information of each sample image to obtain the spliced ​​first feature information.

[0080] Step 406: input the first feature information into a feature fusion alignment module, and output the semantic feature information corresponding to the initial sample image.

[0081] In this embodiment, the execution subject inputs the first feature information into the feature fusion alignment module, and outputs the semantic feature information corresponding to the initial sample image.

[0082] By extracting the image feature information of each sample image and then splicing the image feature information of each sample image to obtain the first feature information, the recognition ability of the image details is improved and the integrity of the image information is improved.

[0083] Step 407 , using the semantic feature information and prompt as input and the image description information as output, trains the pre-built large language model to obtain a large image and text recognition model.

[0084] In this embodiment, the execution subject takes the semantic feature information and prompt as input and the image description information as output, trains the pre-built large language model, and obtains a large image and text recognition model.

[0085] Step 407 is basically the same as step 204 of the aforementioned embodiment. The specific implementation method can refer to the aforementioned description of step 204, which will not be repeated here.

[0086] from Figure 4 It can be seen that Figure 3Compared with the corresponding embodiments, the training method of the large image and text recognition model in this embodiment highlights the steps of splicing the image feature information of each sample image and generating semantic feature information based on the spliced ​​feature information, thereby improving the ability to recognize image details and improving the integrity of image information.

[0087] Continue to refer Figure 5 , Figure 5 A process 500 of another embodiment of a method for training a large model for image and text recognition according to the present disclosure is shown. The method for training a large model for image and text recognition includes the following steps:

[0088] Step 501: Obtain a training sample set.

[0089] In this embodiment, the execution subject of the training method of the large image-text recognition model (for example Figure 1 The server 105 shown in the figure will obtain a training sample set, wherein the training samples in the training sample set include an initial sample image, a prompt word prompt corresponding to the initial sample image, and image description information corresponding to the initial sample image.

[0090] Step 501 is basically the same as step 401 of the aforementioned embodiment. For the specific implementation method, reference may be made to the aforementioned description of step 401 and will not be repeated here.

[0091] Step 502: input the initial sample image into a facial feature extraction model, and output facial feature information of the initial sample image.

[0092] In this embodiment, the above-mentioned execution subject will input the initial sample image into the facial feature extraction model, and output the facial feature information of the initial sample image. That is, the application scenario of this embodiment is to recognize facial photos, that is, the initial sample image is a picture containing a face. Here, the initial sample image containing a face will be first input into the facial feature extraction model, so that the facial feature extraction model extracts the facial features in the initial sample image, and outputs the extracted facial feature information. The facial feature extraction model is a model that can extract facial features, such as a pre-trained model ResNet50.

[0093] Step 503, based on the facial feature information and the pre-built facial feature library, determine the character name information of the initial sample image.

[0094] In this embodiment, the execution subject determines the character name information of the sample image based on the facial feature information and the pre-built facial feature library. The character name information here is the name of the character, and the pre-built facial feature library stores the facial features of a large number of characters and the names of different characters. Here, the execution subject matches the facial feature information with the facial features in the facial feature library. If the match is successful, the name corresponding to the successfully matched facial feature is the character name information. Thus, the facial feature information and character name information of the initial sample image are obtained.

[0095] Step 504: scale the initial sample image to a target size to obtain a target sample image.

[0096] Step 505 , dividing the target sample image into N images each having a preset size, to obtain N sample images.

[0097] Step 506: input N sample images into the image processing module, and output the image feature information of each sample image.

[0098] Step 507: input the image feature information of each sample image into a feature fusion alignment module, and output the semantic feature information corresponding to the initial sample image.

[0099] Steps 504-507 are basically consistent with steps 302-305 of the aforementioned embodiment. For specific implementation methods, reference may be made to the aforementioned description of steps 302-305, which will not be repeated here.

[0100] Step 508: Concatenate the facial feature information and the semantic feature information to obtain second feature information.

[0101] In this embodiment, the execution subject will splice the facial feature information and the semantic feature information to obtain the second feature information. That is, the semantic feature information of the picture is spliced ​​with the facial feature information of the person in the picture to obtain the second feature information. By splicing the facial features and the features of the picture itself together, the understanding of the main content of the picture can be guaranteed without losing the information of the important people in the picture.

[0102] Step 509, concatenate the prompt and the character name information to obtain the target text information.

[0103] In this embodiment, the execution entity will splice the prompt and the character name information to obtain the target text information. Since the prompt is the information input by the user for the current picture, and the character name information is the text information describing the person in the picture, the two text information are spliced ​​together and integrated into a fixed format to obtain complete text information, that is, the target text information. For example, if the prompt is "What's in the picture?" and the character name information is "Zhang San", then the target text information is "What's in the picture? It is known that the picture contains Zhang San."

[0104] Step 510, taking the second feature information and the target text information as input, taking the image description information as output, training the pre-built large language model, and obtaining a large image-text recognition model.

[0105] In this embodiment, the execution subject takes the second feature information and the target text information as input, takes the image description information as output, trains the pre-built large language model, and obtains a large image-text recognition model.

[0106] from Figure 5 It can be seen that Figure 4 Compared with the corresponding embodiments, the training method of the large image and text recognition model in this embodiment combines the features of the human face with the features of the image itself, which can not only ensure the understanding of the main content of the image, but also will not lose the information of important people in the image, thereby further improving the image and text recognition precision and recognition accuracy of the large image and text recognition model.

[0107] Continue to refer Figure 6 , which shows a process 600 of an embodiment of a method for image and text recognition according to the present disclosure. The method for image and text recognition comprises the following steps:

[0108] Step 601, obtain the image to be identified and the prompt corresponding to the image to be identified.

[0109] In this embodiment, the execution subject of the image and text recognition method (for example Figure 1 The server 105 shown in the figure will obtain the image to be identified and the prompt corresponding to the image to be identified. The image to be identified can be the image that the user wants to identify, and the image to be identified can include information such as people. The prompt is the identification requirement information input by the user, that is, what information the user wants to know about the image to be identified, for example, the prompt can be "What's in the picture?"

[0110] Step 602: scaling and segmenting the image to be identified to obtain multiple target images.

[0111] In this embodiment, the execution subject will scale and segment the image to be identified to obtain multiple target images. That is, the execution subject will first scale the image to be identified to the target size to obtain a target sample image, and then segment the target sample image into N images of preset sizes to obtain N sample images, where N = M 2 , M is an integer greater than 1, and N is an integer.

[0112] Step 603: input multiple target images and prompts into the image and text recognition model, and output the image description information of the image to be recognized.

[0113] In this embodiment, the above-mentioned execution subject will input the segmented multiple target images and prompt into the large image and text recognition model, and output the image description information of the image to be recognized, wherein the large image and text recognition model is trained using the method described in the above embodiment. The large image and text recognition model includes: an image processing module, a feature fusion alignment module and a large language model. The image processing module is used to extract the image features of multiple target images, the feature fusion alignment module is used to output the semantic feature information corresponding to the target features after the image features of multiple target images are spliced, and the large language model is used to output the complete image description information corresponding to the image to be recognized.

[0114] The image description information obtained by the large image-text recognition model is the answer content corresponding to the prompt, which is the complete description information of the image, including the detailed information in the image. For example, if the prompt is "What is on the table?", the image description information obtained by the large image-text recognition model is "There is a pen holder on the table, and there are 5 pens in the pen holder."

[0115] The image and text recognition method provided by the disclosed embodiment first obtains the image to be recognized and the prompt corresponding to the image to be recognized; then, the image to be recognized is scaled and segmented to obtain multiple target images; finally, the multiple target images and the prompt are input into the image and text recognition large model, and the image description information of the image to be recognized is output. The image and text recognition method in this embodiment can use the image and text recognition large model to accurately generate complete description information of the image, improve the recognition ability of the image details, and thus improve the efficiency and accuracy of image and text recognition.

[0116] Continue to refer Figure 7 , which shows a process 700 of another embodiment of the image and text recognition method according to the present disclosure. The image and text recognition method comprises the following steps:

[0117] Step 701, obtain the image to be identified and the prompt corresponding to the image to be identified.

[0118] In this embodiment, the execution subject of the image and text recognition method (for example Figure 1 The server 105 shown in the figure will obtain the picture to be identified and the prompt corresponding to the picture to be identified.

[0119] Step 702: scaling and segmenting the image to be identified to obtain multiple target images.

[0120] In this embodiment, the execution entity above will scale and segment the image to be identified to obtain multiple target images.

[0121] Steps 701-702 are basically consistent with steps 601-602 of the aforementioned embodiment. For specific implementation methods, reference may be made to the aforementioned description of steps 601-602, which will not be repeated here.

[0122] In some optional implementations of this embodiment, step 702 includes: scaling the image to be identified to a target size to obtain a first image; dividing the first image into N images of a preset size to obtain N target images, where N=M. 2 , M is an integer greater than 1, and N is an integer.

[0123] In this implementation, the execution subject first scales the image to be identified to a target size, and then segments the image of the target size to obtain N segmented images, where the N images have the same preset size, and N=M. 2 , M=1, 2, 3..., correspondingly M is 1, 4, 9...

[0124] Step 703: input multiple target images into an image processing module, and output image feature information of each target image.

[0125] In this embodiment, the execution subject inputs a plurality of target images into the image processing module, so that the image processing module extracts the image feature information of each target image, thereby outputting the image feature information of each target image.

[0126] Step 704: input the image feature information of each target image into a feature fusion alignment module, and output the semantic feature information corresponding to the image to be identified.

[0127] In this embodiment, the execution entity inputs the image feature information of each target image into the feature fusion alignment module, and outputs the semantic feature information corresponding to the image to be identified.

[0128] In some optional implementations of this embodiment, step 704 includes: splicing the image feature information of each target image to obtain first feature information; inputting the first feature information into a feature fusion alignment module, and outputting semantic feature information corresponding to the image to be identified.

[0129] In this implementation, the execution subject will first splice the image feature information of each target image to obtain the spliced ​​first feature information, and then input the first feature information into the feature fusion alignment module to output the complete semantic feature information corresponding to the image to be identified.

[0130] Step 705: input the semantic feature information and prompt into the large language model, and output the image description information of the image to be identified.

[0131] In this embodiment, the execution subject inputs the semantic feature information and prompt into the large language model, thereby outputting the image description information of the image to be identified.

[0132] from Figure 7 It can be seen that Figure 6 Compared with the corresponding embodiments, the image and text recognition method in this embodiment splices the image feature information of multiple target images together, and generates corresponding complete semantic feature information based on the spliced ​​image feature information, thereby improving the ability to recognize image details, thereby further improving the recognition accuracy and recognition accuracy of image and text recognition.

[0133] Continue to refer Figure 8 , Figure 8 An application scenario of the image and text recognition method according to the present disclosure is shown. In this application scenario, the execution subject first obtains the initial image, then scales the initial image to 1792*1792, and divides the scaled image into 4 target images, and the size of the target image is 448*448. Then, the execution subject inputs the 4 target images into the VIT image feature extractor, and outputs the image features of each target image, that is, image features*4. After that, the execution subject inputs the obtained 4 image features into the feature fusion alignment module to splice the 4 image features to obtain the target image features, and outputs the semantic feature information of the target image features (that is, the aligned feature information). Finally, the aligned feature information is input into the LLM (Large Language Model) image description generation model, and the image description information corresponding to the initial image is output.

[0134] Further references Fig. 9 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a training device for a large model of image and text recognition. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0135] like Fig. 9As shown, the training device 900 of the large model of image and text recognition of this embodiment includes: a first acquisition module 901, a first segmentation module 902, an input module 903 and a training module 904. The first acquisition module 901 is configured to acquire a training sample set, wherein the training samples in the training sample set include an initial sample image, a prompt word prompt corresponding to the initial sample image, and image description information corresponding to the initial sample image; the first segmentation module 902 is configured to scale and segment the initial sample image to obtain multiple sample images; the input module 903 is configured to input multiple sample images into the image processing conversion module, and output the semantic feature information corresponding to the initial sample image; the training module 904 is configured to take the semantic feature information and prompt as input, and take the image description information as output, to train the pre-constructed large language model to obtain the large model of image and text recognition.

[0136] In the training device 900 for the large model of image and text recognition, the specific processing of the first acquisition module 901, the first segmentation module 902, the input module 903 and the training module 904 and the technical effects thereof can be referred to in detail. Figure 2 The relevant descriptions of steps 201 - 204 in the corresponding embodiment are not repeated here.

[0137] In some optional implementations of this embodiment, the first segmentation module is further configured to: scale the initial sample image to a target size to obtain a target sample image; segment the target sample image into N images of a preset size to obtain N sample images, where N = M 2 , M is an integer greater than 1, and N is an integer.

[0138] In some optional implementations of the present embodiment, the image processing conversion module includes: an image processing module and a feature fusion alignment module; and the input module includes: a first input submodule, configured to input N sample images into the image processing module, and output image feature information of each sample image; a second input submodule, configured to input the image feature information of each sample image into the feature fusion alignment module, and output semantic feature information corresponding to the initial sample image.

[0139] In some optional implementations of this embodiment, the second input submodule is further configured to: splice the image feature information of each sample image to obtain first feature information; input the first feature information into the feature fusion alignment module, and output the semantic feature information corresponding to the initial sample image.

[0140] In some optional implementations of the present embodiment, the training device 900 for the above-mentioned large image and text recognition model also includes: a pre-training module, configured to use the first data set to pre-train the initial image processing module and the initial feature fusion alignment module to obtain the image processing module and the pre-trained feature fusion alignment module; a first fine-tuning module, configured to use the second data set to fine-tune the pre-trained feature fusion alignment module and the initial large language model to obtain the fine-tuned feature fusion alignment module and the fine-tuned large language model; a second fine-tuning module, configured to use the third data set to fine-tune the fine-tuned feature fusion alignment module and the fine-tuned large language model again to obtain the feature fusion alignment module and the large language model.

[0141] In some optional implementations of the present embodiment, the training device 900 of the above-mentioned large image and text recognition model also includes: a first output module, configured to input the initial sample image into the facial feature extraction model, and output the facial feature information of the initial sample image; a determination module, configured to determine the character name information of the initial sample image based on the facial feature information and a pre-built facial feature library.

[0142] In some optional implementations of this embodiment, the training device 900 for the above-mentioned large image and text recognition model also includes: a splicing module, which is configured to splice the facial feature information and the semantic feature information to obtain the second feature information.

[0143] In some optional implementations of this embodiment, the training module is further configured to: concatenate the prompt and character name information to obtain the target text information; take the second feature information and the target text information as input, and take the image description information as output, train the pre-built large language model, and obtain a large image and text recognition model.

[0144] Further references Fig.10 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a graphic text recognition device. Figure 6 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0145] like Fig.10As shown, the image and text recognition device 1000 of this embodiment includes: a second acquisition module 1001, a second segmentation module 1002, and a second output module 1003. Among them, the second acquisition module 1001 is configured to obtain the image to be recognized and the prompt corresponding to the image to be recognized; the second segmentation module 1002 is configured to scale and segment the image to be recognized to obtain multiple target images; the second output module 1003 is configured to input the multiple target images and the prompt into the image and text recognition model, and output the image description information of the image to be recognized.

[0146] In this embodiment, in the image and text recognition device 1000, the specific processing of the second acquisition module 1001, the second segmentation module 1002 and the second output module 1003 and the technical effects thereof can be referred to in Figure 6 The relevant descriptions of steps 601-603 in the corresponding embodiment are not repeated here.

[0147] In some optional implementations of this embodiment, the second segmentation module is further configured to: scale the image to be identified to a target size to obtain a first image; segment the first image into N images of a preset size to obtain N target images, where N = M 2 , M is an integer greater than 1, and N is an integer.

[0148] In some optional implementations of the present embodiment, the large model for image and text recognition includes: an image processing module, a feature fusion alignment module and a large language model; and the second output module includes: a third input submodule, configured to input multiple target images into the image processing module, and output image feature information of each target image; a fourth input submodule, configured to input the image feature information of each target image into the feature fusion alignment module, and output semantic feature information corresponding to the image to be recognized; a fifth input submodule, configured to input the semantic feature information and prompt into the large language model, and output image description information of the image to be recognized.

[0149] In some optional implementations of this embodiment, the fourth input submodule is further configured to: splice the image feature information of each target image to obtain first feature information; input the first feature information into the feature fusion alignment module, and output the semantic feature information corresponding to the image to be identified.

[0150] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0151] Fig.11A schematic block diagram of an example electronic device 1100 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0152] like Fig.11 As shown, the device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the device 1100 can also be stored. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0153] A number of components in the device 1100 are connected to the I / O interface 1105, including: an input unit 1106, such as a keyboard, a mouse, etc.; an output unit 1107, such as various types of displays, speakers, etc.; a storage unit 1108, such as a disk, an optical disk, etc.; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1109 allows the device 1100 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0154] The computing unit 1101 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1101 performs the various methods and processes described above, such as the training method of the large model of image and text recognition or the image and text recognition method. For example, in some embodiments, the training method of the large model of image and text recognition or the image and text recognition method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1100 via ROM 1102 and / or the communication unit 1109. When the computer program is loaded into RAM 1103 and executed by the computing unit 1101, one or more steps of the training method of the large model of image and text recognition or the image and text recognition method described above may be executed. Alternatively, in other embodiments, the computing unit 1101 may be configured to execute the training method of the large image and text recognition model or the image and text recognition method in any other appropriate manner (for example, by means of firmware).

[0155] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0156] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0157] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0158] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0159] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0160] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0161] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0162] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A training method for a large image and text recognition model, comprising: Acquire a training sample set, wherein the training samples in the training sample set include an initial sample picture, a prompt word prompt corresponding to the initial sample picture, and picture description information corresponding to the initial sample picture; Scaling and segmenting the initial sample image to obtain multiple sample images; Inputting the plurality of sample images into an image processing conversion module, and outputting semantic feature information corresponding to the initial sample images; The semantic feature information, the prompt, the facial feature information of the initial sample image, and the character name information of the initial sample image are taken as input, and the image description information is taken as output, and a pre-constructed large language model is trained to obtain the large image-text recognition model, wherein the facial feature information is obtained based on a facial feature extraction model, and the character name information is obtained based on the facial feature information and a pre-constructed facial feature library.

2. The method according to claim 1, wherein: The initial sample image is scaled and segmented to obtain a plurality of sample images, including: Scaling the initial sample image to a target size to obtain a target sample image; The target sample image is divided into N images of a preset size to obtain N sample images, where N = M 2 , M is an integer greater than 1, and N is an integer.

3. The method according to claim 2, wherein: The image processing conversion module includes: an image processing module and a feature fusion alignment module; and The step of inputting the plurality of sample images into an image processing conversion module and outputting semantic feature information corresponding to the initial sample images includes: Input the N sample images into the image processing module, and output image feature information of each sample image; The image feature information of each sample image is input into the feature fusion alignment module, and the semantic feature information corresponding to the initial sample image is output.

4. The method according to claim 3, wherein: The step of inputting the image feature information of each sample image into the feature fusion alignment module and outputting the semantic feature information corresponding to the initial sample image includes: splicing the picture feature information of each sample picture to obtain first feature information; The first feature information is input into the feature fusion alignment module, and the semantic feature information corresponding to the initial sample image is output.

5. The method according to claim 3, wherein: Also includes: Pre-training an initial image processing module and an initial feature fusion alignment module using the first data set to obtain the image processing module and the pre-trained feature fusion alignment module; Using the second data set, fine-tuning the pre-trained feature fusion alignment module and the initial large language model to obtain a fine-tuned feature fusion alignment module and a fine-tuned large language model; The fine-tuned feature fusion alignment module and the fine-tuned large language model are fine-tuned again using a third data set to obtain the feature fusion alignment module and the large language model.

6. The method according to claim 1, wherein: Also includes: Inputting the initial sample image into a facial feature extraction model, and outputting facial feature information of the initial sample image; Based on the facial feature information and a pre-built facial feature library, the character name information of the initial sample image is determined.

7. The method according to claim 1, wherein: The method uses the semantic feature information, the prompt, the facial feature information of the initial sample image, and the character name information of the initial sample image as input, uses the image description information as output, trains the pre-built large language model, and obtains the image-text recognition large model, including: Concatenating the facial feature information and the semantic feature information to obtain second feature information; Concatenate the prompt and the character name information to obtain target text information; The second feature information and the target text information are used as input, and the image description information is used as output, and a pre-built large language model is trained to obtain the image-text recognition large model.

8. A method for image and text recognition, comprising: Get the image to be identified and the prompt corresponding to the image to be identified; Scaling and segmenting the image to be identified to obtain multiple target images; The multiple target images and the prompt are input into a large image-text recognition model, and the image description information of the image to be recognized is output, wherein the large image-text recognition model is trained using the method described in any one of claims 1-7.

9. The method according to claim 8, wherein: The step of scaling and segmenting the image to be identified to obtain multiple target images includes: Scaling the image to be identified to a target size to obtain a first image; The first image is divided into N images of a preset size to obtain N target images, where N = M 2 , M is an integer greater than 1, and N is an integer.

10. The method according to claim 9, wherein: The large model for image and text recognition includes: an image processing module, a feature fusion alignment module and a large language model; and The step of inputting the plurality of target images and the prompt into the image-text recognition model and outputting the image description information of the image to be recognized includes: Input the multiple target images into the image processing module, and output image feature information of each target image; Inputting the image feature information of each target image into the feature fusion alignment module, and outputting the semantic feature information corresponding to the image to be identified; The semantic feature information and the prompt are input into the large language model, and the image description information of the image to be identified is obtained as output.

11. The method according to claim 10, wherein: The step of inputting the image feature information of each target image into the feature fusion alignment module and outputting the semantic feature information corresponding to the image to be identified includes: splicing the image feature information of each target image to obtain first feature information; The first feature information is input into the feature fusion alignment module, and the semantic feature information corresponding to the image to be identified is output.

12. A training device for a large model of image and text recognition, comprising: A first acquisition module is configured to acquire a training sample set, wherein the training samples in the training sample set include an initial sample image, a prompt word prompt corresponding to the initial sample image, and image description information corresponding to the initial sample image; A first segmentation module is configured to scale and segment the initial sample image to obtain a plurality of sample images; An input module, configured to input the plurality of sample images into an image processing conversion module, and output semantic feature information corresponding to the initial sample images; The training module is configured to take the semantic feature information, the prompt, the facial feature information of the initial sample image, and the character name information of the initial sample image as input, and take the image description information as output, to train a pre-constructed large language model to obtain the image-text recognition large model, wherein the facial feature information is obtained based on a facial feature extraction model, and the character name information is obtained based on the facial feature information and a pre-constructed facial feature library.

13. The device according to claim 12, wherein: The first segmentation module is further configured to: Scaling the initial sample image to a target size to obtain a target sample image; The target sample image is divided into N images of a preset size to obtain N sample images, where N = M 2 , M is an integer greater than 1, and N is an integer.

14. The device according to claim 13, wherein: The image processing conversion module includes: an image processing module and a feature fusion alignment module; and The input module comprises: A first input submodule is configured to input the N sample images into the image processing module, and output image feature information of each sample image; The second input submodule is configured to input the image feature information of each sample image into the feature fusion alignment module, and output the semantic feature information corresponding to the initial sample image.

15. The device according to claim 14, wherein: The second input submodule is further configured to: splicing the picture feature information of each sample picture to obtain first feature information; The first feature information is input into the feature fusion alignment module, and the semantic feature information corresponding to the initial sample image is output.

16. The device according to claim 14, wherein: Also includes: A pre-training module is configured to pre-train an initial image processing module and an initial feature fusion alignment module using the first data set to obtain the image processing module and the pre-trained feature fusion alignment module; A first fine-tuning module is configured to fine-tune the pre-trained feature fusion alignment module and the initial large language model using the second data set to obtain a fine-tuned feature fusion alignment module and a fine-tuned large language model; The second fine-tuning module is configured to use a third data set to fine-tune the fine-tuned feature fusion alignment module and the fine-tuned large language model again to obtain the feature fusion alignment module and the large language model.

17. The device according to claim 12, wherein: Also includes: A first output module is configured to input the initial sample image into a face feature extraction model and output face feature information of the initial sample image; The determination module is configured to determine the character name information of the initial sample image based on the facial feature information and a pre-built facial feature library.

18. The device according to claim 12, wherein: The training module is further configured to: Concatenating the facial feature information and the semantic feature information to obtain second feature information; Concatenate the prompt and the character name information to obtain target text information; The second feature information and the target text information are used as input, and the image description information is used as output, and a pre-built large language model is trained to obtain the image-text recognition large model.

19. A device for image and text recognition, comprising: A second acquisition module is configured to acquire a picture to be identified and a prompt corresponding to the picture to be identified; A second segmentation module is configured to scale and segment the image to be identified to obtain multiple target images; The second output module is configured to input the multiple target images and the prompt into a large image-text recognition model, and output image description information of the image to be recognized, wherein the large image-text recognition model is trained using the method described in any one of claims 1-7.

20. The device according to claim 19, wherein The second segmentation module is further configured to: Scaling the image to be identified to a target size to obtain a first image; The first image is divided into N images of a preset size to obtain N target images, where N = M 2 , M is an integer greater than 1, and N is an integer.

21. The device according to claim 20, wherein: The large model for image and text recognition includes: an image processing module, a feature fusion alignment module and a large language model; and The second output module comprises: A third input submodule is configured to input the plurality of target images into the image processing module, and output image feature information of each target image; A fourth input submodule is configured to input the image feature information of each target image into the feature fusion alignment module, and output the semantic feature information corresponding to the image to be identified; The fifth input submodule is configured to input the semantic feature information and the prompt into the large language model, and output the image description information of the image to be identified.

22. The device according to claim 21, wherein The fourth input submodule is further configured to: splicing the image feature information of each target image to obtain first feature information; The first feature information is input into the feature fusion alignment module, and the semantic feature information corresponding to the image to be identified is output.

23. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7 or 8-11.

24. A non-transitory computer-readable storage medium storing computer instructions, the computer instructions being used to cause the computer to execute the method of any one of claims 1-7 or 8-11.

25. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1-7 or 8-11.

Images