Target identification and positioning method and device based on multi-modal large language model, and product

Abstract

The invention provides a target recognition and positioning method, device and product based on a multi-modal large language model, and relates to the field of large language model recognized.The method comprises the steps that a coordinate Token is added into a word segmentation device of a visual language large model, a full connection layer is added to the output end, and a to-be-trained multi-modal large language model is obtained; respectively inputting the text content in the training image and the corresponding question text into a word segmentation device to obtain a text coding feature and a question coding feature; inputting the training image into a to-be-trained multi-modal large language model to obtain image features; obtaining answer text content and position coordinates thereof through a to-be-trained multi-modal large language model based on the image features and the question coding features; and carrying out model training on the basis of the answer text content, the text coding features, the position coordinates of the text content and the position coordinates of the answer text content to obtain a trained multi-modal large language model, so that a text positioning function is added under the condition of not changing an image recognition function of the multi-modal large language model.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of large language model recognition, and in particular to a method, device and product for target recognition and positioning based on a multimodal large language model. Background Art

[0002] At present, multimodal large models are developing rapidly and have shown great potential in practical applications. They have certain image target recognition capabilities in various scenarios. However, all current multimodal large models can only output target categories or text content. If you want to output the precise location of the target while outputting the target category or text content, this is often very difficult or even impossible.

[0003] However, when obtaining the target category or text content, it is often urgent to obtain the corresponding position in many scenarios: taking text recognition as an example, the large model recognizes a piece of text content in the image, but the recognition result does not meet the requirements. If there is a positioning function, it can be easily known where the recognition result that does not meet the requirements appears in the image, so as to correct the error and make corrections; or, if the large model cannot directly recognize the content of some ancient characters or rare characters, but can recognize common characters, then when encountering unrecognizable characters, mark and output the text position, so that it can be distinguished or perform subsequent special recognition processing; no matter what the scenario, the precise positioning of the target provides great help for understanding image recognition and handling image anomalies. In addition, the recognition and positioning of targets have huge market value both in terms of business and improving customer experience. Summary of the invention

[0004] The embodiments of the present invention provide a method, device and product for target recognition and positioning based on a multimodal large language model, so as to add a text positioning function without changing the image recognition function of the multimodal large language model.

[0005] A first aspect of an embodiment of the present invention provides a method for target recognition and positioning based on a multimodal large language model, the method comprising:

[0006] Add coordinate tokens to the word segmenter of the visual language model, and add a fully connected layer to the output of the visual language model to obtain a multimodal language model to be trained. The fully connected layer is used to increase the position output and change the output dimension.

[0007] Acquire real training data, the real training data including: a training image, text content in the training image, position coordinates of the text content in the training image, and question text corresponding to the training image;

[0008] Inputting the text content in the training image into the word segmenter, and obtaining text encoding features according to a vocabulary or a code book;

[0009] Inputting the question text corresponding to the training image into the word segmenter, and obtaining the question coding features according to the vocabulary or code book;

[0010] Inputting the training image into the multimodal large language model to be trained to obtain image features;

[0011] Obtaining answer text content and position coordinates of the answer text content in the training image based on the image features and the question encoding features through the multimodal large language model to be trained;

[0012] Based on the answer text content and the text encoding features, as well as the position coordinates of the text content in the training image and the position coordinates of the answer text content in the training image, the multimodal large language model to be trained is trained to obtain a trained multimodal large language model;

[0013] The target image and the question text corresponding to the target image are input into the trained multimodal large language model to obtain the target text content of the target image and the position coordinates of the target text content in the target image.

[0014] A second aspect of an embodiment of the present invention provides a target recognition and positioning device based on a multimodal large language model, the device comprising:

[0015] The initial model acquisition module is used to add coordinate tokens to the word segmenter of the visual language model, and add a fully connected layer to the output end of the visual language model to obtain a multimodal language model to be trained. The fully connected layer is used to increase the position output and change the output dimension.

[0016] A training data acquisition module is used to acquire real training data, wherein the real training data includes: a training image, text content in the training image, position coordinates of the text content in the training image, and a question text corresponding to the training image;

[0017] A first word segmentation module, used for inputting the text content in the training image into the word segmenter, and obtaining text coding features according to a vocabulary or a code book;

[0018] A second word segmentation module, used for inputting the question text corresponding to the training image into the word segmenter, and obtaining the question coding feature according to the vocabulary or code book;

[0019] An image processing module, used for inputting the training image into the multimodal large language model to be trained to obtain image features;

[0020] A first model prediction module is used to obtain the answer text content and the position coordinates of the answer text content in the training image based on the image features and the question encoding features through the multimodal large language model to be trained;

[0021] A model training module, used for training the multimodal large language model to be trained based on the answer text content and the text encoding features, as well as the position coordinates of the text content in the training image and the position coordinates of the answer text content in the training image, to obtain a trained multimodal large language model;

[0022] The second model prediction module is used to input the target image and the question text corresponding to the target image into the trained multimodal large language model to obtain the target text content of the target image and the position coordinates of the target text content in the target image.

[0023] A third aspect of an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the computer program is executed by the processor, the target recognition and positioning method based on a multimodal large language model as described in the first aspect of the embodiment of the present invention is implemented.

[0024] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the target recognition and positioning method based on a multimodal large language model of the first aspect of the embodiment of the present invention is implemented.

[0025] In the target recognition and positioning method based on the multimodal large language model provided in the embodiment of the present invention, the current visual language large model is improved: not only is a coordinate Token added to the word segmenter of the visual language large model, but a fully connected layer is also added to the output end of the visual language large model to change the output dimension of the visual language large model, so that the visual language large model (i.e., the multimodal large language model) can understand the location information of the image content and output it. While outputting the target text content in the target image, the visual language large model can also output the position coordinates of the target text content in the target image, thereby accurately locating the position of the text content in the image without changing the image recognition function of the multimodal large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0027] Figure 1 is a flow chart of a method for target recognition and positioning based on a multimodal large language model according to an embodiment of the present invention;

[0028] Figure 2 is a schematic diagram of model training of a multimodal large language model shown in one embodiment of the present invention;

[0029] Figure 3 It is a structural block diagram of a target recognition and positioning device based on a multimodal large language model provided by an embodiment of the present invention;

[0030] Figure 4 It is a schematic diagram of an electronic device shown in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] Generally speaking, for target recognition and positioning, the traditional non-large model target positioning and detection method is used: target positioning first obtains multiple possible text candidate areas, namely candidate positioning, through image preprocessing and then extracting image features; then differentiable binarization is used to determine the precise boundary, namely the precise position coordinates; then, according to the positioning area, further feature extraction and text classification or target classification are performed; although many methods provide end-to-end technical solutions, this dual-task divided into two steps is still the optimal technical solution for traditional non-large models.

[0033] The disadvantage is that this method is often used to solve single-scene tasks; different tasks require a certain batch of data sets to achieve the ideal effect, that is, the generalization is insufficient, and tasks in different scenes often need to be retrained and evaluated. For example, when encountering fonts or targets that are not in the scene, the model often cannot predict at all; in addition, this technical solution is divided into two steps, and its disadvantages compared to the single step are self-evident.

[0034] Therefore, in order to at least partially solve one or more of the above-mentioned problems and other potential problems, an embodiment of the present invention proposes a target recognition and positioning method based on a multimodal large language model. In this method, the current visual language large model is improved: not only is a coordinate Token added to the word segmenter of the visual language large model, but a fully connected layer is also added to the output end of the visual language large model to change the output dimension of the visual language large model, so that the visual language large model (i.e., the multimodal large language model) can understand the location information of the image content and output it. While outputting the target text content in the target image, the visual language large model can also output the position coordinates of the target text content in the target image, thereby accurately locating the position of the text content in the image without changing the image recognition function of the multimodal large language model.

[0035] Hereinafter, specific examples of the present solution will be described in more detail with reference to the accompanying drawings.

[0036] refer to Figure 1 , Figure 1 FIG. 1 is a flow chart showing a method for target recognition and positioning based on a multimodal large language model according to an embodiment of the present invention. Figure 1 As shown, the target recognition and positioning method based on the multimodal large language model of this embodiment may include the following steps:

[0037] Step S11: Add the coordinate Token to the word segmenter of the large visual language model, and add a fully connected layer to the output end of the large visual language model to obtain a large multimodal language model to be trained. The fully connected layer is used to increase the position output and change the output dimension.

[0038] In this embodiment, improvements can be made on the basis of any large visual language model. It is understandable that the large visual language model is a large visual language model (Vision-Language Large Models, VLLMs), which has gradually become a research hotspot in the intersection of computer vision and natural language processing (NLP). This type of model can not only process complex visual information, but also understand and generate natural language text, realizing a seamless connection between vision and language.

[0039] The visual language large model is a multimodal large language model, wherein the multimodality in this embodiment refers to two modes, image and text, i.e., text description and related images as input, and then understanding the association between image content and text; and large language models (LLMs) refer to natural language processing (NLP) models with a large number of parameters trained by deep learning technology, and large language models can process and generate natural language text. In a preferred embodiment, this embodiment is an improvement on the currently optimal visual language large model (e.g., Qwen large language model), which has strong generalization and can be applied in many scenarios.

[0040] In this embodiment, each different position coordinate is directly defined as a different category. Based on this, this embodiment adds a coordinate token to the word segmenter of the visual language model, that is, adds a coordinate category to the word segmenter, so that the visual language model can understand that a single coordinate value is also a category. For example, you can add coordinate categories 0 to 2048 to the word segmenter, that is, limit the predicted coordinate value to be between [0, 2048].

[0041] In order to change the output dimension and number of units of the large visual language model, this embodiment adds a fully connected layer at the output end of the large visual language model to increase the position output and change the output dimension. Among them, the fully connected layer (Fully ConnectedLayer) is a network layer in a neural network. Each neuron in the fully connected layer is connected to all neurons in the previous layer, and the output dimension and the number of output units can be changed. In this embodiment, the fully connected layer is used to increase the position output and change the output dimension, that is, to change the output dimension of the large visual language model, that is, to increase the position output on the basis of the original output, thereby changing the output dimension.

[0042] In this embodiment, a coordinate Token is added to the word segmenter of the visual language large model, and a fully connected layer is added to the output end of the visual language large model to obtain the multimodal large language model to be trained. The coordinate Token is added to the word segmenter, and the visual language large model with a fully connected layer added to the output end is the multimodal large language model to be trained.

[0043] Step S12: Acquire real training data, where the real training data includes: a training image, text content in the training image, position coordinates of the text content in the training image, and question text corresponding to the training image.

[0044] In this embodiment, it is necessary to obtain real training data during the training process of the multimodal large language model to be trained in order to train the multimodal large language model to be trained. In this embodiment, the real training data includes: training images, question texts and answer texts, and the answer texts (i.e., labels) of the real training data are divided into text content in the training images and position coordinates of the text content in the training images. In other words, the real training data includes at least: training images, text content in the training images, position coordinates of the text content in the training images, and question texts corresponding to the training images. Among them, the text content in the training images is the correct text content in the training images for the question text corresponding to the training images; the position coordinates of the text content in the training images are the correct position coordinates in the training images for the question text corresponding to the training images.

[0045] The real training data in this embodiment includes multiple training images, each of which corresponds to the text content in the training image, the position coordinates of the text content in the training image, and the question text corresponding to the training image. In other words, a training image, the text content in the training image, the position coordinates of the text content in the training image, and the question text corresponding to the training image constitute a set of real training data, so that the multimodal large language model to be trained can be trained through multiple sets of real training data.

[0046] Step S13: input the text content in the training image into the word segmenter, and obtain text encoding features according to a vocabulary or a code book.

[0047] In this embodiment, the text content in the training image is input into the word segmenter of the multimodal large language model to be trained, the input word segmenter is decomposed and converted into an understandable form, and then the text encoding features are obtained according to the vocabulary or code book.

[0048] Step S14: input the question text corresponding to the training image into the word segmenter, and obtain the question coding features according to the vocabulary or code book.

[0049] In this embodiment, similarly, the question text corresponding to the training image is input into the word segmenter of the multimodal large language model to be trained, the input word segmenter is decomposed and converted into an understandable form, and then the question encoding feature is obtained according to the vocabulary or code book.

[0050] Step S15: inputting the training image into the multimodal large language model to be trained to obtain image features.

[0051] In this embodiment, the training image needs to be input into the multimodal large language model to be trained to obtain image features.

[0052] Step S16: obtaining the answer text content and the position coordinates of the answer text content in the training image based on the image features and the question encoding features through the multimodal large language model to be trained.

[0053] In this embodiment, the answer text content and the position coordinates of the answer text content in the training image can be obtained by processing the multimodal large language model to be trained based on the image features and the question coding features. The answer text content is the text content predicted and output by the multimodal large language model to be trained, and the position coordinates of the answer text content in the training image are the position coordinates of the answer text content predicted and output by the multimodal large language model to be trained. In one example, the image features and the question coding features can be spliced, and the spliced ​​image features and the question coding features can be processed by the multimodal large language model to be trained to obtain the answer text content and the position coordinates of the answer text content in the training image.

[0054] Step S17: Based on the answer text content and the text encoding features, as well as the position coordinates of the text content in the training image and the position coordinates of the answer text content in the training image, the multimodal large language model to be trained is trained to obtain a trained multimodal large language model.

[0055] In this embodiment, after obtaining the answer text content and the position coordinates of the answer text content in the training image, the multimodal large language model to be trained can be trained based on the answer text content and text encoding features, as well as the position coordinates of the text content in the training image and the position coordinates of the answer text content in the training image to obtain a trained multimodal large language model.

[0056] Among them, during the training process, the multimodal large language model to be trained continuously learns based on the text content in the training image and the position coordinates of the text content in the training image to obtain the answer text, and then understands the obtained answer text based on the word segmenter with the coordinate Token added, from which the position coordinates and text content are understood, and the answer text content and the position coordinates of the answer text content in the training image are split from the answer text for output, and finally a trained multimodal large language model that can realize target recognition and target positioning is obtained.

[0057] In this embodiment, target recognition refers to identifying the category or text content of the corresponding target in the image. The text content in this method includes but is not limited to Chinese, English, numbers and symbols. Target positioning requires determining the exact position of an object (such as text) in the image through the image content. This method involves locating the bounding box of the target in the image to mark the position, and it is necessary to determine the four coordinate points of the text bounding box, which are also closely related to the center point coordinates, width and height of the text.

[0058] Step S18: input the target image and the question text corresponding to the target image into the trained multimodal large language model to obtain the target text content of the target image and the position coordinates of the target text content in the target image.

[0059] In this embodiment, after obtaining a trained multimodal large language model, the target image and the question text corresponding to the target image can be input into the trained multimodal large language model to obtain the target text content of the target image output by the trained multimodal large language model and the position coordinates of the target text content in the target image. The target image is an image to be subjected to target recognition and positioning, such as the target image can be an image to be subjected to text recognition and positioning.

[0060] In this embodiment, the current visual language large model is improved: not only is a coordinate Token added to the word segmenter of the visual language large model, but a fully connected layer is also added to the output end of the visual language large model to change the output dimension of the visual language large model, so that the improved visual language large model (i.e., the multimodal large language model) can understand the location information of the image content and output it. While outputting the target text content in the target image, the visual language large model can also output the position coordinates of the target text content in the target image, thereby accurately locating the position of the text content in the image without changing the image recognition function of the multimodal large language model.

[0061] In combination with the above embodiments, in one implementation, the present invention also provides a method for target recognition and positioning based on a multimodal large language model. Specifically, in this embodiment, the above step S17 may include steps S21 to S23:

[0062] Step S21: Calculate a first loss based on the answer text content and the text encoding feature.

[0063] In this embodiment, after obtaining the output of the multimodal large language model to be trained, it is necessary to calculate the difference between the output predicted value and the true value, and optimize and update the model parameters of the multimodal large language model to be trained through the loss function Loss. Specifically, based on the text encoding features corresponding to the answer text content (i.e., predicted value) output by the multimodal large language model to be trained and the text content in the training image (i.e., true value), the first loss is calculated through the first loss function.

[0064] Step S22: Calculate a second loss based on the position coordinates of the text content in the training image and the position coordinates of the answer text content in the training image.

[0065] In this embodiment, it is also necessary to calculate the second loss through a second loss function based on the position coordinates of the answer text content in the training image (i.e., the predicted value) output by the multimodal large language model to be trained and the position coordinates of the text content in the training image (i.e., the true value).

[0066] In this embodiment, the first loss function and the second loss function can be the same or different loss functions, and there is no limitation on this.

[0067] Step S23: Based on the first loss and the second loss, the multimodal large language model to be trained is trained to obtain a trained multimodal large language model.

[0068] In this embodiment, the multimodal large language model to be trained can be trained based on the calculated first loss and second loss, and the model parameters can be gradually updated until a trained multimodal large language model is obtained.

[0069] In combination with the above embodiments, in one implementation, the present invention also provides a method for target recognition and positioning based on a multimodal large language model. Specifically, in this method, the above step S23 may specifically include step S31:

[0070] Step S31: Based on the first loss, the second loss and the loss weight, the multimodal large language model to be trained is trained to obtain a trained multimodal large language model.

[0071] In this embodiment, the first loss is the cross-entropy loss, and the second loss is the Iou loss. Among them, the cross-entropy loss is a loss function commonly used in classification problems. It measures the difference between the probability distribution predicted by the model and the probability distribution of the true label. Cross-entropy loss is very effective in almost all classification problems, especially in the training process of neural networks, it can help the model learn how to reduce prediction errors during training and improve the performance of classification tasks. IoU (Intersection over Union) loss is a loss function in the field of computer vision, especially in target detection and positioning. It measures the degree of overlap between the predicted bounding box and the true bounding box. The calculation formula is IoU = (the intersection area of ​​the predicted bounding box and the true bounding box / the union area of ​​the predicted bounding box and the true bounding box), so the value range of the iou value is between 0 and 1. It is not difficult to understand from the formula that the higher the value, the greater the degree of overlap of the bounding boxes, that is, the more accurate the prediction.

[0072] That is to say, this embodiment is based on the text encoding features corresponding to the answer text content output by the multimodal large language model to be trained and the text content in the training image, and the first loss is calculated through the cross entropy loss function (i.e., the first loss function), that is, the cross entropy loss is calculated. And, based on the position coordinates of the answer text content output by the multimodal large language model to be trained in the training image and the position coordinates of the text content in the training image, the second loss is calculated through the IoU loss function (i.e., the second loss function), that is, the Iou loss is calculated. In an optional example, the IoU loss function can be a Wise-Iou loss function, which is a bounding box loss based on a dynamic non-monotonic focusing mechanism that is improved on the basis of the original Iou.

[0073] Then, based on the calculated first loss, second loss and the pre-set loss weight, the total loss is obtained, and then the multimodal large language model to be trained is trained based on the total loss, and the model parameters are gradually updated. Finally, after multiple rounds of training, the total loss value drops to the lowest point. At this time, the model parameters are fixed to obtain the final prediction model, that is, the trained multimodal large language model.

[0074] In addition, in one embodiment, after the total loss value drops to the lowest point and the model parameters are fixed, the model with the fixed model parameters can be tested through a test set test, and the model that passes the test is used as a trained multimodal large language model.

[0075] In this embodiment, while retaining the cross entropy loss CELoss, the Iou loss is added, and a loss weight λ is introduced for this purpose. The loss weight is pre-set to control the proportion of the first loss and the second loss, λ∈[0,1]. In one embodiment, the loss function calculation formula of the total loss is Loss=λ(Iou-Loss)+1-λ(CELoss); wherein Iou-Loss is the second loss, i.e., Iou loss; CELoss is the first loss, i.e., cross entropy loss; the closer λ is to 1, the higher the proportion of the positioning loss Iou, and the closer λ is to 0, the higher the proportion of the text content cross entropy loss.

[0076] In combination with the above embodiments, in another implementation, the embodiment of the present invention further provides a method for target recognition and positioning based on a multimodal large language model. Specifically, in this embodiment, the "obtaining real training data" in the above step S12 may specifically include steps S41 to S44:

[0077] Step S41: Acquire an initial training image.

[0078] In this embodiment, a plurality of initial training images may be acquired, where the initial training images are original, unprocessed images.

[0079] Step S42: performing data enhancement on the initial training image to obtain the training image.

[0080] In this embodiment, it is necessary to perform random data enhancement on the initial training image to obtain a training image that can be used for model training. The data enhancement in this embodiment includes at least one of the following: adding Gaussian noise, sharpening, perspective transformation, increasing the brightness, contrast and hue saturation of the image. In an optional embodiment, for an initial training image, multiple random data enhancements can be performed to obtain multiple training images, thereby achieving the purpose of expanding the training sample.

[0081] Step S43: Obtain the initial question text corresponding to the training image.

[0082] In this embodiment, after obtaining the training image, the initial question text corresponding to the training image may be obtained, and the initial question text is the original unprocessed question text.

[0083] Step S44: randomly perform semantic enhancement on the initial question text corresponding to the training image through a large language model to obtain the question text corresponding to the training image.

[0084] In this embodiment, the initial question text corresponding to the training image can be semantically randomly enhanced by a large language model, thereby obtaining the question text corresponding to the training image for model training.

[0085] In combination with the above embodiments, in another implementation, the present invention also provides a method for target recognition and positioning based on a multimodal large language model. Specifically, in this embodiment, in addition to the above steps, it can also include step S51 and step S52:

[0086] Step S51: input the first image and the question text corresponding to the first image into the trained multimodal large language model to obtain the first text content of the first image and the position coordinates of the first text content in the first image.

[0087] In this embodiment, after obtaining a trained multimodal large language model, the trained multimodal large language model can be used to correct the image that does not meet the requirements. For example, the trained multimodal large language model recognizes a text content in the image, but the text content has a recognition result that does not meet the requirements. If there is a positioning function, it can be easily known where the recognition result that does not meet the requirements appears in the image, so as to perform error correction and correction.

[0088] Specifically, the first image and the question text corresponding to the first image can be input into the trained multimodal large language model to obtain the first text content corresponding to the first image and the position coordinates of the first text content in the first image. The first text content is the text recognition result of the trained multimodal large language model for the first image.

[0089] Step S52: when the first text content of the first image does not meet the requirements corresponding to the first image, the first text content in the first image is corrected based on the position coordinates of the first text content in the first image to obtain a first image that meets the requirements.

[0090] In this embodiment, it can be determined whether the first text content of the first image meets the requirements corresponding to the first image. If the first text content of the first image does not meet the requirements corresponding to the first image, the first text content in the first image is corrected and corrected based on the position coordinates of the first text content in the first image to obtain a first image that meets the requirements.

[0091] In one example, the requirements corresponding to the first image may include: the category corresponding to the first image. For example, the category corresponding to the first image is fruit, but the first text content of the first image obtained by the model includes "mobile phone", and it is determined that "mobile phone" does not conform to the category of "fruit". Therefore, based on the position coordinates of the "mobile phone" in the first image obtained by the model, the "mobile phone" in the first image can be corrected to obtain a first image that meets the requirements.

[0092] In combination with the above embodiments, in another implementation, the present invention also provides a method for target recognition and positioning based on a multimodal large language model. Specifically, in this embodiment, in addition to the above steps, steps S61 to S63 may also be included:

[0093] Step S61: input the second image and the question text corresponding to the second image into the trained multimodal large language model to obtain the second text content of the second image and the position coordinates of the second text content in the second image.

[0094] In this embodiment, after obtaining the trained multimodal large language model, different types of text in the image can be marked in different ways by the trained multimodal large language model. Specifically, the second image and the question text corresponding to the second image can be input into the trained multimodal large language model to obtain the second text content corresponding to the second image and the position coordinates of the second text content in the second image. Among them, the second text content is the text recognition result of the trained multimodal large language model for the second image, and the second image includes at least any of the following: ancient characters, uncommon characters, and commonly used characters.

[0095] Step S62: when the second text content is ancient characters and / or rare characters, based on the position coordinates of the second text content in the second image, the second text content is framed in the second image in a first manner.

[0096] In this embodiment, the second text content can be judged to determine whether the second text content is a common character, which is a character in the modern Chinese common character table. If it is determined that the second text content is not a common character, the second text content is determined to be an ancient character and / or an uncommon character.

[0097] In one example, whether the second text content is a common character can be determined by determining whether the second text content is content that cannot be directly recognized: when it is determined that the second text content is not content that cannot be directly recognized, the second text content is determined to be a common character; when it is determined that the second text content is content that cannot be directly recognized, the second text content is determined not to be a common character, and the second text content is determined to be an ancient character and / or an uncommon character.

[0098] When it is determined that the second text content is ancient characters and / or rare characters, the second text content can be framed in the second image in a first manner based on the position coordinates of the second text content in the second image, so as to mark the ancient characters and / or rare characters in the second image in the first manner.

[0099] Step S63: when the second text content is a commonly used character, based on the position coordinates of the second text content in the second image, the second text content is framed and selected in the second image in a second manner.

[0100] In this embodiment, when it is determined that the second text content is a commonly used character, the second text content can be framed in the second image in a second manner based on the position coordinates of the second text content in the second image, so as to mark the commonly used characters in the second image in a first manner. Among them, the first manner of this embodiment is different from the second manner, so as to distinguish ancient characters and / or uncommon characters from commonly used characters in the second image, so as to perform subsequent special recognition processing on the second image, such as further special recognition processing on the ancient characters and / or uncommon characters in the second image. It should be noted that this embodiment does not make specific restrictions on the first manner and the second manner, and the two manners can be different.

[0101] In one embodiment, if Figure 2 As shown, Figure 2 FIG. 1 is a schematic diagram of a model training of a multimodal large language model shown in an embodiment of the present invention. Figure 2 In the example, Text is the training text. Text is split into Question Text and Answer Text. Then Answer Text is split into Position coordinates and Target Content, so that they can be processed separately later.

[0102] Secondly, for the distinguished Target Content (such as text content), the Tokenizer of the multimodal large language model to be trained is used to decompose the content and convert it into an understandable form, and then the corresponding category Input Id of the text content is obtained according to the vocabulary or code book (that is, the aforementioned text encoding feature); and, similarly, for Question Text (question text), that is Figure 2 The Semantic Text in the text is also decomposed by the Tokenizer of the multimodal large language model to be trained and converted into an understandable form. Then, the corresponding category Input Id of the text content (i.e., the aforementioned problem encoding feature) is obtained according to the vocabulary or code book.

[0103] For the distinguished Position coordinates, in order to make the multimodal large language model to be trained understand that a single coordinate value is also a category, the word segmenter of the multimodal large language model to be trained should also add coordinate categories (i.e., add coordinate tokens) before model training. For example, add coordinate categories 0 to 2048, i.e., limit the predicted coordinate values ​​to be between [0, 2024]. In this way, the input target content (Target Content) and the target position information (Position coordinates) are distinguished, which is convenient for subsequent processing and calculation.

[0104] Next, for the training images (i.e. Figure 2 In the multimodal large language model to be trained, perform Vision Encoding, Vision Tower, Feature Extraction, and Image Embedding in sequence to obtain image features. In addition, perform Text Embedding on Input Id (question encoding feature).

[0105] Then, after splicing (ConCat) the image features after Image Embedding and the problem encoding features of Text Embedding, they are processed by adding a fully connected layer to the multimodal large language model to be trained (Large Language Model), adding a fully connected layer to increase the Position output, and changing the output dimension and number of units of the large language model. The final model output includes text content Id_Out and position coordinates Pos_Out. In this way, this embodiment makes comprehensive improvements to the model output, processing, and output to increase the model positioning capability.

[0106] Finally, after obtaining the model output, it is necessary to calculate the difference between the output and the true value, and optimize and update the model parameters through the loss function Loss until a trained multimodal large language model is obtained. In this step, this embodiment adds Iou loss while retaining the cross entropy loss CELoss, and the Wise-Iou loss function is used here; after modifying the loss function hyperparameters, the loss weight λ is also introduced to control the proportion of the two losses. In the present invention, λ∈[0,1], the closer λ is to 1, the higher the proportion of positioning loss Iou, and the closer λ is to 0, the higher the proportion of target content cross entropy loss. Finally, our loss function calculation formula is Loss=λ(Iou-Loss)+1-λ(CELoss). Among them, the cross entropy loss CELoss is calculated based on the text content Id_Out and Input Id (text encoding features, and the Wise-Iou Loss is calculated based on the position coordinates Pos_Out and Positioncoordinates (position coordinates).

[0107] In this way, through the improvement and optimization of the aforementioned steps, the model can fully understand the location information. The trained model can accurately locate the target area through the input images and questions, answer the target content and accurately output the target's location coordinates.

[0108] In combination with the above embodiments, in a specific example, according to the needs (based on the image input by the user and answering the questions about the image, for example, inputting a picture of an ID card and needing to output the ID card name and the location information of the name field), first, prepare a number of initial samples, including: images with text, input questions and the correct corresponding output text results and the coordinate position of the text, and 5,000 initial samples in this format.

[0109] Secondly, training samples are prepared based on several initial samples: 5,000 initial samples are subjected to data enhancement, and the randomly added data enhancement includes adding Gaussian noise, sharpening, perspective transformation, as well as image brightness, contrast and hue saturation; then the input question is subjected to semantic random enhancement through a large language model; finally, a total of 50,000 enhanced samples are obtained, and 45,000 are randomly selected as training sets at a ratio of 9 to 1, and 5,000 are used as test sets.

[0110] Then, the model is trained based on the training samples: after the training test samples are prepared, the model training is performed. Using the method of any of the aforementioned embodiments, the input text and the correct output result are distinguished into target content information and location coordinate information, and the coordinate Token is added to the word segmenter; the processed input is learned by the modified model to obtain the content ID output and the coordinate Pos output, and the model parameters are gradually updated by calculating the loss difference between the set Iou+ cross entropy loss and the correct output; finally, after multiple rounds of training, the loss value is reduced to the lowest point, and the effect of the model is also optimal when tested by the test set, that is, it can be used as the final prediction model.

[0111] Finally, model prediction and result display are performed based on the final prediction model: According to the final prediction model, the questions and images input by the user are used as the model input for prediction. The model finally predicts and outputs accurate content information and coordinate information based on the learned parameters.

[0112] In this embodiment, a coordinate position module is added to the visual language model: the coordinate information is separated from the real information, and the coordinate token is added to the word segmenter. Finally, the model output is modified and the Iou loss function is added to achieve the output of both the target content information and the target coordinate position information. In this way, in addition to solving special information extraction problems such as image location information, it also has strong flexibility and substitutability, that is, the location information extraction module can be flexibly removed or replaced at any time and deployed in other personalized target extraction scenarios.

[0113] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0114] Based on the same inventive concept, an embodiment of the present invention provides a target recognition and positioning device 300 based on a multimodal large language model. Figure 3 , Figure 3 FIG. 1 is a block diagram of a target recognition and positioning device based on a multimodal large language model provided by an embodiment of the present invention. Figure 3 As shown, the target recognition and positioning device 300 based on the multimodal large language model includes:

[0115] The initial model acquisition module 301 is used to add the coordinate token to the word segmenter of the visual language large model, and add a fully connected layer to the output end of the visual language large model to obtain a multimodal large language model to be trained, wherein the fully connected layer is used to increase the position output and change the output dimension;

[0116] The training data acquisition module 302 is used to acquire real training data, wherein the real training data includes: a training image, text content in the training image, position coordinates of the text content in the training image, and question text corresponding to the training image;

[0117] The first word segmentation module 303 is used to input the text content in the training image into the word segmenter and obtain text coding features according to a vocabulary or a code book;

[0118] The second word segmentation module 304 is used to input the question text corresponding to the training image into the word segmenter, and obtain the question coding feature according to the vocabulary or code book;

[0119] An image processing module 305 is used to input the training image into the multimodal large language model to be trained to obtain image features;

[0120] A first model prediction module 306, configured to obtain the answer text content and the position coordinates of the answer text content in the training image based on the image features and the question encoding features through the multimodal large language model to be trained;

[0121] A model training module 307 is used to train the multimodal large language model to be trained based on the answer text content and the text encoding features, as well as the position coordinates of the text content in the training image and the position coordinates of the answer text content in the training image, to obtain a trained multimodal large language model;

[0122] The second model prediction module 308 is used to input the target image and the question text corresponding to the target image into the trained multimodal large language model to obtain the target text content of the target image and the position coordinates of the target text content in the target image.

[0123] Optionally, the model training module 307 includes:

[0124] A first loss calculation module, used for calculating a first loss based on the answer text content and the text encoding feature;

[0125] A second loss calculation module, used for calculating a second loss based on the position coordinates of the text content in the training image and the position coordinates of the answer text content in the training image;

[0126] A loss training module is used to train the multimodal large language model to be trained based on the first loss and the second loss to obtain a trained multimodal large language model.

[0127] Optionally, the first loss is a cross entropy loss, and the second loss is an Iou loss; the loss training module includes:

[0128] The loss training submodule is used to train the multimodal large language model to be trained based on the first loss, the second loss and the loss weight to obtain a trained multimodal large language model.

[0129] Optionally, the training data acquisition module 302 includes:

[0130] An initial data acquisition module, used to acquire initial training images;

[0131] An image enhancement module, used for performing data enhancement on the initial training image to obtain the training image; the data enhancement includes at least one of the following: adding Gaussian noise, sharpening, perspective transformation, increasing the brightness, contrast and hue saturation of the image;

[0132] An initial text acquisition module, used to acquire the initial question text corresponding to the training image;

[0133] The text enhancement module is used to perform random semantic enhancement on the initial question text corresponding to the training image through a large language model to obtain the question text corresponding to the training image.

[0134] Optionally, the target recognition and positioning device 300 based on a multimodal large language model further includes:

[0135] A first prediction module, configured to input a first image and a question text corresponding to the first image into the trained multimodal large language model to obtain a first text content of the first image and a position coordinate of the first text content in the first image;

[0136] The image correction module is used to correct the first text content in the first image based on the position coordinates of the first text content in the first image to obtain a first image that meets the requirements when the first text content in the first image does not meet the requirements corresponding to the first image.

[0137] Optionally, the target recognition and positioning device 300 based on a multimodal large language model further includes:

[0138] A second prediction module is used to input the second image and the question text corresponding to the second image into the trained multimodal large language model to obtain the second text content of the second image and the position coordinates of the second text content in the second image; the second image includes at least any one of the following: ancient characters, rare characters, and common characters;

[0139] A first frame selection module is used to frame the second text content in the second image in a first manner based on the position coordinates of the second text content in the second image when the second text content is an ancient text and / or an uncommon character;

[0140] The second selection module is used to select the second text content in the second image in a second manner based on the position coordinates of the second text content in the second image when the second text content is a commonly used character; the first manner is different from the second manner.

[0141] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the target recognition and positioning method based on a multimodal large language model as described in any of the above embodiments of the present invention are implemented.

[0142] Based on the same inventive concept, another embodiment of the present invention provides an electronic device 400, such as Figure 4 shown. Figure 4 4 is a schematic diagram of an electronic device shown in an embodiment of the present invention. The electronic device includes a memory 402, a processor 401, and a computer program stored in the memory and executable on the processor, and when executed by the processor, the steps in the target recognition and positioning method based on a multimodal large language model described in any of the above embodiments of the present invention are implemented.

[0143] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0144] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0145] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, embodiments of the present invention may take the form of complete hardware embodiments, complete software embodiments, or embodiments combining software and hardware. Furthermore, embodiments of the present invention may take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0146] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0147] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0148] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0149] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0150] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0151] The above is a detailed introduction to the target recognition and positioning method, device and product based on a multimodal large language model provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A target recognition and positioning method based on a multimodal large language model, characterized in that: The method comprises: Add coordinate tokens to the word segmenter of the visual language model, and add a fully connected layer to the output of the visual language model to obtain a multimodal language model to be trained. The fully connected layer is used to increase the position output and change the output dimension. Acquire real training data, the real training data including: a training image, text content in the training image, position coordinates of the text content in the training image, and question text corresponding to the training image; Inputting the text content in the training image into the word segmenter, and obtaining text encoding features according to a vocabulary or a code book; Inputting the question text corresponding to the training image into the word segmenter, and obtaining the question coding features according to the vocabulary or code book; Inputting the training image into the multimodal large language model to be trained to obtain image features; Obtaining answer text content and position coordinates of the answer text content in the training image based on the image features and the question encoding features through the multimodal large language model to be trained; Based on the answer text content and the text encoding features, as well as the position coordinates of the text content in the training image and the position coordinates of the answer text content in the training image, the multimodal large language model to be trained is trained to obtain a trained multimodal large language model; The target image and the question text corresponding to the target image are input into the trained multimodal large language model to obtain the target text content of the target image and the position coordinates of the target text content in the target image.

2. The target recognition and positioning method based on a multimodal large language model according to claim 1, characterized in that: Based on the answer text content and the text encoding features, as well as the position coordinates of the text content in the training image and the position coordinates of the answer text content in the training image, the multimodal large language model to be trained is trained to obtain a trained multimodal large language model, including: Calculate a first loss based on the answer text content and the text encoding feature; Calculating a second loss based on the position coordinates of the text content in the training image and the position coordinates of the answer text content in the training image; Based on the first loss and the second loss, the multimodal large language model to be trained is trained to obtain a trained multimodal large language model.

3. The target recognition and positioning method based on a multimodal large language model according to claim 2 is characterized in that: The first loss is a cross entropy loss, and the second loss is an Iou loss; based on the first loss and the second loss, the multimodal large language model to be trained is trained to obtain a trained multimodal large language model, including: Based on the first loss, the second loss and the loss weight, the multimodal large language model to be trained is trained to obtain a trained multimodal large language model.

4. The target recognition and positioning method based on a multimodal large language model according to claim 1, characterized in that: The obtaining of real training data comprises: Get initial training images; Performing data enhancement on the initial training image to obtain the training image; the data enhancement includes at least one of the following: adding Gaussian noise, sharpening, perspective transformation, increasing the brightness, contrast and hue saturation of the image; Obtaining an initial question text corresponding to the training image; The initial question text corresponding to the training image is semantically randomly enhanced by using a large language model to obtain the question text corresponding to the training image.

5. The target recognition and positioning method based on a multimodal large language model according to any one of claims 1 to 4, characterized in that: The method further comprises: Inputting the first image and the question text corresponding to the first image into the trained multimodal large language model to obtain the first text content of the first image and the position coordinates of the first text content in the first image; When the first text content of the first image does not meet the requirements corresponding to the first image, the first text content in the first image is corrected based on the position coordinates of the first text content in the first image to obtain a first image that meets the requirements.

6. The target recognition and positioning method based on a multimodal large language model according to any one of claims 1 to 4, characterized in that: The method further comprises: Inputting the second image and the question text corresponding to the second image into the trained multimodal large language model to obtain the second text content of the second image and the position coordinates of the second text content in the second image; the second image includes at least any one of the following: ancient characters, rare characters, and common characters; In the case where the second text content is an ancient character and / or an uncommon character, based on the position coordinates of the second text content in the second image, the second text content is framed and selected in the second image in a first manner; When the second text content is a commonly used character, the second text content is framed in the second image in a second manner based on the position coordinates of the second text content in the second image; the first manner is different from the second manner.

7. A target recognition and positioning device based on a multimodal large language model, characterized in that: The device comprises: The initial model acquisition module is used to add coordinate tokens to the word segmenter of the visual language model, and add a fully connected layer to the output end of the visual language model to obtain a multimodal language model to be trained. The fully connected layer is used to increase the position output and change the output dimension. A training data acquisition module is used to acquire real training data, wherein the real training data includes: a training image, text content in the training image, position coordinates of the text content in the training image, and a question text corresponding to the training image; A first word segmentation module, used for inputting the text content in the training image into the word segmenter, and obtaining text coding features according to a vocabulary or a code book; A second word segmentation module, used for inputting the question text corresponding to the training image into the word segmenter, and obtaining the question coding feature according to the vocabulary or code book; An image processing module, used for inputting the training image into the multimodal large language model to be trained to obtain image features; A first model prediction module is used to obtain the answer text content and the position coordinates of the answer text content in the training image based on the image features and the question encoding features through the multimodal large language model to be trained; A model training module, used for training the multimodal large language model to be trained based on the answer text content and the text encoding features, as well as the position coordinates of the text content in the training image and the position coordinates of the answer text content in the training image, to obtain a trained multimodal large language model; The second model prediction module is used to input the target image and the question text corresponding to the target image into the trained multimodal large language model to obtain the target text content of the target image and the position coordinates of the target text content in the target image.

8. The target recognition and positioning device based on a multimodal large language model according to claim 7, characterized in that: The model training module includes: A first loss calculation module, used for calculating a first loss based on the answer text content and the text encoding feature; A second loss calculation module, used for calculating a second loss based on the position coordinates of the text content in the training image and the position coordinates of the answer text content in the training image; A loss training module is used to train the multimodal large language model to be trained based on the first loss and the second loss to obtain a trained multimodal large language model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the computer program is executed by the processor, the target recognition and positioning method based on a multimodal large language model as described in any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the target recognition and positioning method based on a multimodal large language model as described in any one of claims 1 to 6 is implemented.

Images