A large model-based multi-scene text recognition method and system

Abstract

The application provides a large model-based multi-scene text recognition method and system, and belongs to the technical field of multi-scene text recognition; the method comprises the following steps: constructing a training data set based on multi-scene image samples; training a prompt engine by using the training data set, obtaining an image to be recognized, extracting an image visual feature vector of the image to be recognized and inputting the image visual feature vector into the trained prompt engine to obtain a prompt word template identifier of the image to be recognized; inputting a target prompt word and the image to be recognized into a text recognition model, and outputting a text recognition result by the text recognition model; by introducing the trainable prompt engine, the most relevant prompt word is screened out from the visual features of the input image by the trainable prompt engine, and the prompt word is used to provide scene-based and targeted recognition guidance for the text recognition model, so that the technical problem that the generalization ability of a single text recognition model is weak in various scenes is effectively solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-scene text recognition, and particularly relates to a multi-scene text recognition method and system based on a large model. BACKGROUND

[0002] With the development of information technology, text recognition technology has experienced iterative evolution from traditional methods to deep learning driven methods. Early text recognition technology is mainly based on traditional image processing and pattern recognition methods, which realizes text extraction through image preprocessing, character segmentation, feature extraction and classifier recognition, such as character recognition methods based on template matching and character classification methods based on contour features. However, this kind of method has high dependence on image quality, and when facing complex situations such as light change, shooting angle offset, background noise interference and font style diversity, the recognition accuracy decreases significantly, and it is necessary to manually design special feature extraction rules and classification models for different scenes, which has poor scene adaptability and is difficult to meet the actual application requirements in multiple scenes.

[0003] To solve the limitations of traditional methods, deep learning technology is widely used in the field of text recognition, and a series of text recognition schemes based on convolutional neural network (CNN), recurrent neural network (RNN), long short-term memory network (LSTM) and other models have appeared. This kind of method automatically learns deep features in images through neural network, replaces traditional manual feature design, and significantly improves the recognition performance in complex scenes. Although the deep learning driven text recognition method has made a qualitative leap compared with the traditional method, there are still obvious shortcomings in multi-scene adaptation. On the one hand, existing deep learning models are mostly trained and optimized for specific scenes, such as customized models specially designed for license plate recognition and bill recognition. When applied to new scenes that have not been trained, due to the difference in visual features between scenes (such as text layout, background complexity, font characteristics, etc.), the model has insufficient generalization ability, and the recognition accuracy decreases sharply. If a special model is trained for each scene, not only a large number of labeled samples are needed, but also the number of models is redundant, increasing the deployment and maintenance cost. On the other hand, with the rise of large language models (LLM), its powerful semantic understanding and context modeling ability provides a new technical idea for text recognition, and some researches try to combine visual features with large models to optimize the recognition results by prompting words. However, in the current related scheme, the prompt words are mostly preset by humans according to scene experience, and lack of dynamic adaptation ability to scene features. When the input image scene changes, the fixed prompt words cannot match the scene requirements, and cannot fully exert the semantic advantages of the large model, resulting in limited recognition performance. SUMMARY

[0004] The application aims to provide a large model-based multi-scene text recognition method and system, which trains a prompt engine by collecting multi-scene image samples, makes the prompt engine match a target prompt word according to a visual feature vector of an image to be recognized, and then inputs the target prompt word and the image to be recognized into a text recognition model to output a text recognition result, thereby realizing multi-scene text recognition.

[0005] To achieve the above-mentioned purpose, the application provides a large model-based multi-scene text recognition method, which comprises the following steps: collecting multi-scene image samples, and constructing a training data set based on the multi-scene image samples; training a prompt engine by using the training data set, so that the prompt engine can output a corresponding prompt word template identifier according to an image visual feature vector of an input image; obtaining an image to be recognized, extracting an image visual feature vector of the image to be recognized, and inputting the image visual feature vector into the trained prompt engine to obtain a prompt word template identifier of the image to be recognized; querying a target prompt word from a prompt word template library according to the prompt word template identifier of the image to be recognized; and inputting the target prompt word and the image to be recognized into a text recognition model, so that the text recognition model outputs a text recognition result, thereby completing text recognition of a corresponding scene.

[0006] Optionally, the training data set is composed of a plurality of training samples, and each training sample comprises an image visual feature vector and a corresponding prompt word template identifier.

[0007] Optionally, the step of constructing a training data set based on the multi-scene image samples comprises the following steps: labeling a prompt word template identifier for each image sample, wherein the prompt word template identifier is associated with a predefined prompt word template of a scene to which the image sample belongs; using a pre-trained visual feature extraction model to perform visual feature extraction operation on all the labeled image samples, so as to obtain image visual feature vectors corresponding to the image samples respectively; and combining the image visual feature vector of each image sample and the labeled prompt word template identifier as a key-value pair, and forming a training data set by all the key-value pairs.

[0008] Optionally, the step of performing visual feature extraction operation on all the labeled image samples to obtain image visual feature vectors corresponding to the image samples respectively comprises the following steps: pre-processing each image sample, wherein the pre-processing at least comprises adjusting the image size to a fixed input size set by the pre-trained visual feature extraction model; inputting the pre-processed image sample into the visual feature extraction model; extracting a feature tensor from an output end of a global average pooling layer of the pre-trained visual feature extraction model; and performing flattening processing on the feature tensor to obtain an image visual feature vector with a fixed dimension.

[0009] Optionally, the training of the prompt engine using the training dataset comprises: constructing a classifier network, and taking the classifier network as the prompt engine, the classifier network comprising an input layer, at least one fully connected hidden layer, and an output layer; the number of neurons of the input layer is consistent with the dimension of the image visual feature vector, and the number of neurons of the output layer is the same as the total number of categories of the predefined prompt word templates; the output layer adopts a normalized exponential activation function to output a probability distribution corresponding to each category of the prompt word templates; the training dataset is divided into a training subset and a validation subset according to a preset ratio, the classifier network is iteratively trained using the training subset, and a performance indicator of the validation subset is taken as a trigger judgment condition of an early stopping mechanism; when the trigger judgment condition of the early stopping mechanism is met or a preset maximum number of iterations is reached, the training is terminated, and the trained prompt engine is obtained.

[0010] Optionally, the iterative training of the classifier network using the training subset comprises: inputting the image visual feature vector of one data batch in the training subset into the classifier network; the classifier network sequentially passes through the input layer, the at least one fully connected hidden layer, and the output layer to perform nonlinear transformation on the input image visual feature vector, and generates a predicted probability distribution vector from the output layer; a deviation between the predicted probability distribution vector and a one-hot encoding vector corresponding to a true prompt word template identifier is calculated, and the deviation is taken as a loss value of the current data batch; using an error back propagation algorithm, a gradient of the loss value with respect to the parameters of the classifier network is calculated; using a gradient descent-based optimizer, each parameter of the classifier network is adjusted according to the calculated gradient; the above steps are repeated to traverse all data batches in the training subset to complete one training cycle; a plurality of training cycles are repeatedly performed until the early stopping mechanism based on the performance of the validation subset is triggered or the preset maximum number of iterations is reached.

[0011] Optionally, the extraction of the image visual feature vector of the to-be-recognized image and the input of the image visual feature vector into the trained prompt engine to obtain the prompt word template identifier of the to-be-recognized image comprises: pre-processing the to-be-recognized image, the pre-processing at least comprising adjusting the size of the to-be-recognized image to a fixed input size set by the visual feature extraction model; inputting the pre-processed to-be-recognized image into the visual feature extraction model, the visual feature extraction model outputting an image visual feature vector corresponding to the to-be-recognized image; inputting the image visual feature vector corresponding to the to-be-recognized image into the trained prompt engine, the prompt engine outputting the prompt word template identifier of the to-be-recognized image.

[0012] Optionally, the querying the target prompt word from the prompt word template library comprises: performing a key-value matching operation in the pre-stored prompt word template library with the prompt word template identifier as the key to obtain the target prompt word.

[0013] Optionally, the inputting the target prompt word and the image to be recognized into the text recognition model, and the text recognition model outputting a text recognition result, comprises: constructing the target prompt word and the image to be recognized into combined data, and inputting the combined data into the text recognition model, and the text recognition model outputting a text recognition result.

[0014] In another aspect, the present application provides a large model-based multi-scene text recognition system for implementing a large model-based multi-scene text recognition method, which comprises a control module, the control module comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the computer program to implement the large model-based multi-scene text recognition method.

[0015] The above technical solution uses the visual features of the input image to filter out the prompt word most relevant to the current scene through the trainable prompt engine, and uses the prompt word to provide scene-based and targeted recognition guidance for the text recognition model, effectively solving the technical problem of weak generalization ability of a single text recognition model in diverse scenes.

[0016] Other features and advantages of the present application will be described in detail in the following specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings are included to provide a further understanding of the present application, and constitute a part of the specification, and are used together with the following specific embodiments to explain the present application, but do not constitute a limitation on the present application. In the drawings:

[0018] Figure 1 is a large model-based multi-scene text recognition method flowchart.

[0019] Figure 2 is a multi-scene image sample labeling and auditing flowchart.

[0020] Figure 3 is a prompt engine training iteration flowchart. DETAILED DESCRIPTION

[0021] The following will be described in detail in conjunction with the accompanying drawings Figure 1 - the accompanying drawings Figure 3 The specific embodiments of the present application are described in detail. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application.

[0022] It should be noted that the acquisition, transmission, storage, use, processing, etc. of data in the technical solutions of the present application comply with the relevant provisions of national laws and regulations. In the embodiments of the present application, some industry existing solutions such as software, components, models, etc. may be mentioned, which should be considered as exemplary, and the purpose is only to illustrate the feasibility in the implementation of the technical solutions of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0023] The present inventors found in the process of implementing the present application that the text recognition methods in the prior art, whether they are special models designed for specific scenarios or single models pursuing universality, are difficult to effectively solve the domain adaptation problem in multiple scenarios while taking into account flexibility and low cost, thereby leading to deficiencies in generalization ability and recognition accuracy in complex real environments.

[0024] Embodiment 1

[0025] Reference Figures 1-3 For the first embodiment of the present application, the embodiment provides a multi-scene text recognition method based on a large model, referring to Figure 1 , comprising:

[0026] S100: Collecting multi-scene image samples and constructing a training data set based on the multi-scene image samples.

[0027] In the preferred embodiments of the present application, first, the specific coverage of multi-scenarios is defined, and the core categories of multi-scene image samples are divided according to text carrier attributes and application scenarios, including but not limited to natural scenes (such as street signs, billboards, product packaging, menus, license plates), document scenes (such as printed documents, handwritten notes, forms, receipts, contract scans), special scenes (such as low-light images, skewed distortion images, blurred images, reflective images, multi-language mixed images, small font images). Then, each type of scene is further subdivided into subtypes, for example, "street signs" in natural scenes can be subdivided into traffic signs, street signs, scenic guide signs, etc.; "receipts" in document scenes can be subdivided into invoices, tickets, receipts, etc., thereby ensuring the comprehensiveness and typicality of scene coverage. Then, through public data sets, web crawling or actual shooting, etc., original images covering the above defined scenes are collected to obtain a multi-scene image sample set. Next, the number of samples for various scenes is controlled according to specific use requirements, and the collected original images are preliminarily filtered to eliminate invalid samples that are completely blurred, have no valid text, or have text regions that are severely occluded (occlusion area exceeds 40% of the text region), and retain valid samples with clear text and clear scene features.

[0028] In the embodiments of the present application, the training data set is composed of multiple training samples, and each training sample includes an image visual feature vector and its corresponding prompt word template identifier.

[0029] In the embodiments of the present application, constructing the training data set based on the multi-scene image samples includes: labeling each image sample with a prompt word template identifier, the prompt word template identifier being associated with a predefined prompt word template of the scene to which the image sample belongs; performing visual feature extraction operation on all the labeled image samples using a pre-trained visual feature extraction model to obtain image visual feature vectors corresponding to the image samples respectively; and combining the image visual feature vector of each image sample with the prompt word template identifier labeled for the image sample as a key-value pair, and the training data set is constituted by all the key-value pairs.

[0030] Performing visual feature extraction operation on all the labeled image samples to obtain image visual feature vectors corresponding to the image samples respectively includes: pre-processing each image sample, the pre-processing at least including adjusting the image size to a fixed input size set by the pre-trained visual feature extraction model; inputting the pre-processed image sample into the visual feature extraction model; extracting a feature tensor from an output end of a global average pooling layer of the pre-trained visual feature extraction model; and performing flattening processing on the feature tensor to obtain an image visual feature vector of a fixed dimension.

[0031] The pre-processing of each image sample further includes:

[0032] Size normalization: adjusting the image size to a fixed input size (such as 224x224) set by the pre-trained visual feature extraction model, and adopting bilinear interpolation method to maintain the image proportion during the adjustment process to avoid text deformation.

[0033] Image enhancement: performing targeted enhancement operation for defects in different scenes, histogram equalization algorithm is used to improve brightness for low-light images; text line direction is detected by Hough transform for inclined distortion images, and rotation correction is performed; non-local mean denoising algorithm is used to optimize the definition for blurred images.

[0034] Channel and format unification: converting the image to RGB three-channel format (if it is a grayscale image, channel replication is performed), and normalizing the pixel value to the interval [0, 1] to adapt to the input requirements of the model.

[0035] In the preferred embodiment of the present application, a dedicated pre-defined prompt template is designed for each type of scene subtype, and a prompt template library is constructed. The pre-defined prompt template needs to include scene feature description, text attribute constraint, recognition task requirement and other key information, for example, the traffic sign scene template: "Please recognize the text content on the traffic sign in the image, the text is printed, may contain arrow symbols or English abbreviations, and the text content and symbol information need to be accurately extracted". Then a unique prompt template identifier is assigned to each prompt template, the identifier uses the coding rule of "scene category-scene subtype-template serial number", such as "natural scene-traffic sign-001", then an one-to-one corresponding association relationship between the identifier and the prompt template is established, and stored in the template index table.

[0036] Further, the "Prompt Template Identifier Annotation Manual" is formulated to clearly define the matching rules of each scene subtype and the corresponding prompt template, for example, "images containing license plate outline, blue background white text / yellow background black text features, annotated as 'natural scene-license plate-001' identifier". Referring to Figure 2 , the pre-trained scene classification model is used to pre-annotate the image samples, output the candidate identifier, and then a 3-person annotation team is used to manually review and correct the pre-annotation results, and record ambiguous samples during the annotation process; finally, cross-audit is performed on the annotated samples, and batches with an audit pass rate of less than 90% need to be re-annotated, so as to ensure the accuracy of the annotation.

[0037] In the preferred embodiment of the present application, the base model for image visual feature extraction selects a deep visual model pre-trained on a general image dataset, such as EfficientNet-B4 of high-efficiency convolutional architecture or ViT-Base based on attention mechanism. In terms of parameter configuration, the network layer parameters of the first 80% of the model are frozen to retain the general feature extraction capability obtained by pre-training, and only the top network layer is unfrozen for subsequent targeted fine-tuning, to obtain a visual feature extraction model.

[0038] Further, the pre-processed image sample is input into the visual feature extraction model, the feature tensor is extracted from the global average pooling layer output end of the visual feature extraction model, and the feature tensor is subjected to flattening processing to obtain an initial feature vector. L2 normalization processing is performed on the initial feature vector to constrain the vector length to the [0, 1] interval, and an image visual feature vector with fixed dimensions is obtained.

[0039] Then, the image visual feature vector of each image sample is combined with its labeled prompt word template identifier as a key-value pair. Subsequently, a de-duplication process is performed on all key-value pairs, repeated samples are removed by calculating the cosine similarity between feature vectors (with a similarity threshold of 0.90), and abnormal samples (such as samples with feature vectors significantly deviating from the corresponding scene clustering center) are detected and removed using the Isolation Forest algorithm. Finally, the processed key-value pairs are divided into an initial training subset and an initial validation subset in proportion (e.g., 8:2), which together constitute a complete training dataset, and are stored in LMDB format to improve the reading efficiency of subsequent training.

[0040] The above scheme achieves comprehensive coverage of multiple scenes by layering the core categories and subtypes according to carrier attributes and application scenarios, allowing the training data to accurately match real and diverse application requirements. By customizing exclusive prompt word templates for each scene subtype and binding unique identifiers, the data and scene requirements are accurately associated, providing highly targeted training materials for the prompt engine.

[0041] S200: Train the prompt engine using the training dataset, so that the prompt engine can output the corresponding prompt word template identifier according to the image visual feature vector of the input image.

[0042] In an embodiment of the present application, referring to Figure 3 Training the prompt engine using the training dataset includes: constructing a classifier network and taking the classifier network as the prompt engine, the classifier network including an input layer, at least one fully connected hidden layer (the hidden layer can also be set to 2 or 3 layers), and an output layer; the number of neurons in the input layer is consistent with the dimension of the image visual feature vector, and the number of neurons in the output layer is the same as the total number of pre-defined prompt word template categories; the output layer uses a normalized exponential activation function to output the probability distribution of each prompt word template category; the training dataset is divided into a training subset and a validation subset in a predetermined proportion, the training subset is used to iteratively train the classifier network, and the performance indicators of the validation subset are used as trigger judgment conditions for early stopping mechanism; when the trigger judgment conditions for early stopping mechanism are met or the maximum number of iterations is reached, the training is terminated, and the trained prompt engine is obtained.

[0043] The training of the classifier network using the training subset comprises: inputting the image visual feature vector of one data batch in the training subset into the classifier network; the classifier network sequentially passes through the input layer, at least one fully connected hidden layer, and the output layer, performs nonlinear transformation on the input image visual feature vector, and generates a predicted probability distribution vector from the output layer; a deviation between the predicted probability distribution vector and a one-hot encoded vector corresponding to the real prompt word template identifier is calculated, and the deviation is taken as a loss value of the current data batch; a gradient of the loss value with respect to the parameters of the classifier network is calculated using an error back propagation algorithm; an optimizer based on gradient descent is used to adjust the parameters of the classifier network according to the calculated gradient; the above steps are repeated to traverse all data batches in the training subset to complete one training cycle; after completing one training cycle, if the early stopping mechanism is not triggered and the maximum iteration number is not reached, the learning rate is updated according to the preset cosine annealing learning rate scheduling strategy, and then the next training cycle is entered, and the multiple training cycles are repeatedly executed until the early stopping mechanism based on the performance of the validation subset is triggered or the preset maximum iteration number is reached.

[0044] In the preferred embodiment of the present application, when training the parameter configuration of the prompt engine, an optimizer based on gradient descent (AdamW optimizer, i.e., Adam optimizer with weight decay) is used. The initial learning rate is set based on the task complexity (the total number of prompt word templates in the prompt word template library), the number of network parameters, and the batch size. The learning rate scheduling adopts a cosine annealing strategy, the decay period is set to a fixed proportion of the total training rounds, the minimum learning rate is decayed to a corresponding order of magnitude of the initial learning rate, and periodic learning rate oscillation is used to avoid the model falling into local optimum. The batch size is dynamically adjusted based on hardware computing resources such as GPU memory bandwidth and capacity, which not only ensures that the memory occupation of single forward and backward propagation is controlled within the hardware limit, but also considers the statistical effectiveness of gradient estimation. The total training rounds are set to a sufficient number of iterations covering the complete convergence curve of the model, and the early stopping mechanism is combined to avoid overtraining. The loss function selects cross-entropy loss, which quantifies the KL divergence between the multi-class probability distribution output by the prompt engine (classifier network) and the real label (one-hot encoded form) as the core optimization objective of gradient back propagation.

[0045] Further preferably, to evaluate the generalization performance of the prompt engine and prevent overfitting, after each round of training, the feature vectors of the validation subset are input into the classifier network in the current training state, and the accuracy and macro average F1 score of the validation set are calculated as core evaluation indicators. The early stopping strategy is set as follows: set the early stopping tolerance to 10 rounds, if the improvement of the macro average F1 score of the validation set after 10 consecutive rounds of training does not exceed 0.1%, early stopping is triggered and the training process is terminated; if early stopping is not triggered, the training continues until the maximum number of training rounds (such as 100 rounds) is reached. The model weight with the optimal macro average F1 score of the validation set during training is saved dynamically as the final converged prompt engine model.

[0046] The above scheme selects a mature and simple classifier network as the prompt engine, which can efficiently complete the accurate mapping of visual feature vectors to prompt word template identifiers. The combination of AdamW optimizer, cosine annealing learning rate scheduling, cross-entropy loss function and early stopping mechanism not only conforms to the best practice of deep learning, but also accelerates model convergence while effectively avoiding overfitting problems. And through continuous parameter iteration adjustment during training, the prompt engine can fully adapt to the feature differences of multiple scenes, greatly improving the recognition and adaptation ability of different scene images.

[0047] S300: Obtain an image to be recognized, extract an image visual feature vector of the image to be recognized and input it into the trained prompt engine to obtain a prompt word template identifier of the image to be recognized.

[0048] In the embodiments of the present application, the image to be recognized is preprocessed, and the preprocessing at least includes adjusting the size of the image to be recognized to a fixed input size set by the visual feature extraction model; the preprocessed image to be recognized is input into the visual feature extraction model, and the visual feature extraction model outputs an image visual feature vector corresponding to the image to be recognized; the image visual feature vector corresponding to the image to be recognized is input into the trained prompt engine, and the prompt engine outputs a prompt word template identifier of the image to be recognized.

[0049] In the preferred embodiment of the present application, the pre-processing method of the image to be identified is consistent with the pre-processing method of the image sample described above. The pre-processed image tensor is input into the visual feature extraction model configured in the training stage. The visual feature extraction model outputs a fixed-dimension high-dimensional visual feature vector corresponding to the input image to be identified. The fixed-dimension high-dimensional visual feature vector is subjected to flattening processing to convert it into a one-dimensional initial feature vector. Then, L2 normalization is performed on the initial feature vector to constrain the vector length to the interval [0, 1], thereby obtaining an image visual feature vector corresponding to the image to be identified (consistent with the format of the image visual feature vector in the training stage). Then, the image visual feature vector corresponding to the image to be identified is input into the prompt engine trained. The prompt engine outputs a multi-class probability distribution vector through the Softmax activation function (normalized exponential activation function), and the vector length is equal to the total number of predefined prompt word templates (such as a 30-dimensional vector corresponding to 30 prompt word templates). Then, the index with the maximum probability value in the probability distribution vector is taken, and the index corresponds to the predefined prompt word template identifier (such as index 0 corresponding to "natural scene-traffic sign-001"). Finally, the prompt engine outputs the prompt word template identifier of the image to be identified.

[0050] The above scheme can preferentially select the template with the highest matching degree with the image to be identified through Softmax probability sorting, thereby significantly improving the matching accuracy of the scene and the template. The unique prompt word template identifier output can provide a clear index for subsequent rapid query of the target prompt word, thereby ensuring smooth connection of the whole process.

[0051] S400: Query the target prompt word from the prompt word template library according to the prompt word template identifier of the image to be identified.

[0052] In the embodiments of the present application, the key-value matching operation is performed in the pre-stored prompt word template library with the prompt word template identifier as the key, thereby obtaining the target prompt word.

[0053] In the preferred embodiment of the present application, the pre-stored prompt word template library adopts a "key-value" mapping structure, the key of which is a unique prompt word template identifier (such as "natural scene-traffic sign-001"), and the value is the corresponding structured prompt word. The prompt word needs to clearly define the core constraints of the text recognition task (such as "recognize the white text on the outdoor traffic sign, ignore the background blur area, and preferentially ensure the recognition accuracy of numbers and arrow symbols"). In order to ensure the response speed of key-value matching and adapt to the real-time inference requirements of single or batch images, the prompt word template library is stored in the form of an in-memory dictionary, a lightweight database, or a JSON file, etc. In the construction stage of the prompt word template library, the uniqueness of all template identifiers has been pre-verified to avoid matching ambiguity caused by duplicate keys. Furthermore, the correspondence between the identifiers and the prompt words is associated through the establishment of an identifier index table, thereby further improving the query efficiency.

[0054] Taking the above example, the key-value retriever is connected to the output end of the prompt engine at the front end, and is connected to the retrieval entrance of the prompt template library at the rear end. The key-value retriever performs an exact key-value matching operation in the prompt template library with the prompt template identifier of the image to be recognized output by the prompt engine as the retrieval key, and directly locates the corresponding prompt entry. Subsequently, the key-value retriever performs matching result judgment. If the corresponding key-value pair is retrieved, the key-value retriever extracts the prompt word under the entry as the candidate target prompt word, and then the prompt word verifier performs format integrity verification on the candidate target prompt word. After verification, the final target prompt word is output by the result outputter. If no matching result is retrieved (such as identifier error or missing corresponding entry in the template library), the key-value retriever triggers the built-in exception handling module, and the exception handling module outputs the alarm information of “no matching prompt word” and records the exception log.

[0055] The above scheme uses an efficient storage format such as an in-memory dictionary and a lightweight database for the prompt template library, and is matched with an identifier index table, which greatly improves the query response speed and can well adapt to real-time application scenarios. Moreover, through the exact key-value matching and format verification mechanism, it is ensured that the target prompt word obtained is completely adapted to the scene to be recognized, and the problem of template misuse is avoided. The target prompt word obtained can provide scene-based and targeted recognition guidance for text recognition.

[0056] S500: input the target prompt word and the image to be recognized into the text recognition model, and the text recognition model outputs a text recognition result, thereby completing text recognition of the corresponding scene.

[0057] In an embodiment of the present application, the target prompt word and the image to be recognized are constructed as combined data, and the combined data is input into the text recognition model, and the text recognition model outputs a text recognition result.

[0058] In a preferred embodiment of the present application, the target prompt word and the image to be recognized are packaged into “image-prompt word” combined data according to the interface requirements of the text recognition model (such as the Qwen3-VL series), and then the combined data is transmitted into the text recognition model through a preset interface by the data integrator and triggers inference. The text recognition model analyzes the scene constraints and recognition requirements in the target prompt word, and then processes the image to be recognized and focuses on the core recognition target. Finally, a structured text recognition result is generated and output.

[0059] The above scheme uses the combined input mode of image plus target prompt word, so that the text recognition model can accurately grasp the scene characteristics and recognition requirements, effectively focus on the core recognition target, and reduce the interference of irrelevant information. The text recognition model can process low-light, inclined and other multi-scene defects under the guidance of the prompt word, and significantly improve the text recognition accuracy in complex scenes.

[0060] The application further provides a large model-based multi-scene text recognition system for implementing the large model-based multi-scene text recognition method.

[0061] The application further provides a storage medium having a program stored thereon, the program being executed by a processor to implement the large model-based multi-scene text recognition method.

[0062] The application further provides a processor for running a program, wherein the program is executed to implement the large model-based multi-scene text recognition method.

[0063] The application further provides a device including a processor, a memory, and a program stored in the memory and executable on the processor, the processor being executed to implement the large model-based multi-scene text recognition method. The device herein can be a server, a PC, a PAD, a mobile phone, or the like.

[0064] The application further provides a computer program product adapted to implement the large model-based multi-scene text recognition method when executed on a data processing device.

[0065] Those skilled in the art should understand that embodiments of the present application can provide methods, systems, or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) containing computer-usable program code.

[0066] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks. Figure 1 The devices that implement the functions specified in one or more flows or blocks.

[0067] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the Figure 1 function specified in the flow or flows and / or blocks Figure 1 of the block or blocks.

[0068] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the Figure 1 function specified in the flow or flows and / or blocks Figure 1 of the block or blocks.

[0069] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0070] The memory can include non-persistent memory and / or volatile memory, e.g., random access memory (RAM) and / or non-volatile memory, e.g., read-only memory (ROM) or flash memory (flash RAM). The memory is an example of computer-readable media.

[0071] Computer-readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD), or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media, such as modulated data signals and carrier waves.

[0072] It should also be noted that the terms "comprising", "comprises" or other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0073] The above embodiments are only used to illustrate the present application, but not to limit it. Instead of the above, various modifications and changes can be made to the application by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application shall fall into the scope of the claims of the application.

Claims

1. A multi-scene text recognition method based on a large model, characterized in that, include: Collecting multi-scene image samples and constructing a training dataset based on the multi-scene image samples includes: labeling each image sample with a prompt word template identifier, wherein the prompt word template identifier is associated with a predefined prompt word template of the scene to which the image sample belongs; using a pre-trained visual feature extraction model to perform visual feature extraction operations on all labeled image samples to obtain image visual feature vectors corresponding to each image sample; combining the image visual feature vector of each image sample with its labeled prompt word template identifier into key-value pairs, and using all the key-value pairs to constitute the training dataset; The prompting engine is trained using the training dataset, enabling it to output corresponding prompt word template identifiers based on the image visual feature vectors of the input image. The step of training the prompt engine using the training dataset includes: A classifier network is constructed and used as the prompting engine. The classifier network includes an input layer, at least one fully connected hidden layer, and an output layer. The number of neurons in the input layer is the same as the dimension of the image visual feature vector, and the number of neurons in the output layer is the same as the total number of categories of the predefined prompt word templates; The output layer uses a normalized exponential activation function to output the probability distribution of each prompt word template category; The training dataset is divided into a training subset and a validation subset according to a preset ratio. The classifier network is trained iteratively using the training subset, and the performance index of the validation subset is used as the trigger condition for the early stopping mechanism. When the early stopping mechanism is triggered or the preset maximum number of iterations is reached, training terminates and the trained prompt engine is obtained. The step of iteratively training the classifier network using the training subset includes: The visual feature vectors of one batch of images from the training subset are input into the classifier network. The classifier network sequentially passes through an input layer, at least one fully connected hidden layer, and an output layer to perform a nonlinear transformation on the input image visual feature vector, and the output layer generates a predicted probability distribution vector. Calculate the deviation between the predicted probability distribution vector and the one-hot encoding vector corresponding to the true prompt word template identifier, and use this deviation as the loss value of the current data batch; The gradient of the loss value with respect to the classifier network parameters is calculated using the backpropagation algorithm. An optimizer based on gradient descent is used to adjust the parameters of the classifier network according to the calculated gradient; Repeat the above steps to traverse all data batches in the training subset to complete one training cycle; Repeat the training cycle multiple times until an early stopping mechanism based on the performance of the validation subset is triggered or the preset maximum number of iterations is reached; The image to be identified is obtained, the image visual feature vector of the image to be identified is extracted and input into the trained prompt engine to obtain the prompt word template identifier of the image to be identified; Based on the prompt word template identifier of the image to be recognized, the target prompt word is retrieved from the prompt word template library; The target prompt word and the image to be recognized are input into the text recognition model, and the text recognition model outputs the text recognition result to complete the text recognition for the corresponding scene.

2. The multi-scene text recognition method based on a large model according to claim 1, characterized in that, The training dataset consists of multiple training samples, each of which includes an image visual feature vector and its corresponding prompt word template identifier.

3. The multi-scene text recognition method based on a large model according to claim 1, characterized in that, The step of performing visual feature extraction operations on all labeled image samples to obtain image visual feature vectors corresponding to each image sample includes: Each of the image samples is preprocessed, and the preprocessing includes at least adjusting the image size to the fixed input size set by the pre-trained visual feature extraction model; The preprocessed image samples are input into the visual feature extraction model; Extract feature tensors from the output of the global average pooling layer of the pre-trained visual feature extraction model; Flattening is performed on the feature tensor to obtain a fixed-dimensional image visual feature vector.

4. The multi-scene text recognition method based on a large model according to claim 1, characterized in that, The step of extracting the image visual feature vector of the image to be recognized and inputting it into the trained prompt engine to obtain the prompt word template identifier of the image to be recognized includes: The image to be recognized is preprocessed, and the preprocessing includes at least adjusting the size of the image to be recognized to a fixed input size set by the visual feature extraction model; The preprocessed image to be identified is input into the visual feature extraction model, and the visual feature extraction model outputs an image visual feature vector corresponding to the image to be identified. The visual feature vector of the image to be identified is input into the trained prompting engine, and the prompting engine outputs the prompt word template identifier of the image to be identified.

5. The multi-scene text recognition method based on a large model according to claim 1, characterized in that, The step of retrieving the target prompt word from the prompt word template library includes: using the prompt word template identifier as the key, performing a key-value matching operation in the pre-stored prompt word template library to obtain the target prompt word.

6. The multi-scene text recognition method based on a large model according to claim 1, characterized in that, The step of inputting the target prompt word and the image to be recognized into a text recognition model, and the text recognition model outputting a text recognition result, includes: constructing a combined data set of the target prompt word and the image to be recognized, inputting the combined data set into the text recognition model, and the text recognition model outputting a text recognition result.

7. A multi-scene text recognition system based on a large model, characterized in that, The system includes a control module, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the computer program to implement the multi-scene text recognition method based on a large model according to any one of claims 1-6.

Images