Text detection method and device, storage medium, electronic equipment and program product

Abstract

The invention relates to a text detection method and device, a storage medium, electronic equipment and a program product. Obtaining a to-be-detected text image; inputting the text image into a target detection model to obtain a target text detection result; wherein the target detection model comprises a large language model and a text detection model, the output end of the large language model is coupled with the input end of the text detection model, and the target detection model is a model obtained after instruction fine adjustment is conducted through multiple feature description instructions and sample images. The feature description instruction represents feature information of a text in the sample image.

Description

Technical Field

[0001] The present disclosure relates to the field of text detection, and in particular, to a text detection method, device, storage medium, electronic device, and program product. Background Art

[0002] With the continuous advancement of smart education, its application is becoming increasingly widespread. In smart education scenarios, optical character recognition (OCR) of text images is a crucial step. For example, in electronic examination marking, OCR can be used to scan and identify the content on the examination paper. This recognized content is then uploaded to the review system, where it is then compared with the pre-set judgment and comparison content, allowing for automatic review and judgment of the examination paper. Summary of the Invention

[0003] The purpose of the present disclosure is to provide a text detection method, device, storage medium, electronic device and program product.

[0004] To achieve the above objectives, the present disclosure provides a text detection method, the method comprising:

[0005] Get the text image to be detected;

[0006] After inputting the text image into the target detection model, a target text detection result is obtained;

[0007] Among them, the target detection model includes a large language model and a text detection model, the output end of the large language model is coupled with the input end of the text detection model, and the target detection model is a model that has been fine-tuned using multiple feature description instructions and sample images, and the feature description instructions represent the feature information of the text in the sample image.

[0008] Optionally, after inputting the text image into the target detection model, obtaining a target text detection result includes:

[0009] After inputting the text image into the large language model, obtaining a first feature map output by the large language model and feature description information of the text in the text image;

[0010] After the first feature map and the feature description information are input into the text detection model, the target text detection result is obtained.

[0011] Optionally, the text detection model includes a DBNet network, and the DBNet network includes a ResNet network and a feature pyramid network FPN connected to the ResNet network; after inputting the first feature map and the feature description information into the text detection model, obtaining the target text detection result includes:

[0012] Inputting the first feature map and the feature description information into the ResNet network to obtain a plurality of second feature maps of different scales;

[0013] After performing feature enhancement on the plurality of second feature maps through the FPN, a plurality of feature enhancement maps of different scales are obtained;

[0014] After performing feature fusion on the multiple feature enhancement maps of different scales, a fused feature map is obtained;

[0015] After determining a text detection probability map and a text detection threshold map corresponding to the text image according to the fused feature map, the target text detection result is determined according to the text detection probability map and the text detection threshold map.

[0016] Optionally, the target detection model is pre-tuned by performing instruction fine-tuning in the following manner:

[0017] For each sample image, construct a feature description instruction corresponding to the sample image, where different sample images correspond to different feature description instructions;

[0018] For each of the sample images, inputting the sample image and the feature description instruction corresponding to the sample image into a pre-trained large language model to obtain a sample feature map output by the pre-trained large language model and sample feature description information corresponding to the sample image;

[0019] After inputting the sample feature map and the sample feature description information into a pre-trained text detection model, a model output for text detection is obtained;

[0020] According to the model output of the text detection and the text detection label corresponding to the sample image, the parameters of the pre-trained large language model and the pre-trained text detection model are fine-tuned through a preset loss function to obtain the target detection model.

[0021] Optionally, the preset loss function includes a cross-entropy loss function, the model output includes a pixel probability sequence corresponding to the sample image obtained by model prediction, each element in the pixel probability sequence represents the probability that the corresponding pixel belongs to a text pixel of the text to be detected; the text detection label includes a pixel annotation sequence belonging to the text pixel in the sample image;

[0022] The object detection model is obtained by fine-tuning the parameters of the pre-trained large language model and the pre-trained text detection model according to the model output of the text detection and the text detection label corresponding to the sample image through a preset loss function, including:

[0023] According to the pixel point probability sequence and the pixel annotation sequence, after calculating the target loss value through the cross entropy loss function, the parameters of the pre-trained large language model and the pre-trained text detection model are fine-tuned according to the target loss value to obtain the target detection model.

[0024] Optionally, the text feature described in the feature description instruction includes at least one of the following:

[0025] Text style, character category corresponding to the text, whether there is background interference, text position, text size, text direction, text context, preset types of characters in the text, and the recognition scenario corresponding to the text.

[0026] Through the above technical solution, a text image to be detected is obtained; after the text image is input into a target detection model, a target text detection result is obtained; wherein, the target detection model includes a large language model and a text detection model, the output end of the large language model is coupled to the input end of the text detection model, and the target detection model is a model that has been fine-tuned using multiple feature description instructions and sample images, wherein the feature description instructions represent the feature information of the text in the sample image. Text detection is performed on the text image to be detected by the target detection model. Since the architecture of the target detection model uses a large language model and a text detection model, the universal characteristics of the large language model can be utilized, that is, it performs better in a wide range of tasks, not just for specific tasks. After fine-tuning the model using multiple feature description instructions, the target detection model can adaptively recognize text with different attribute features, thereby improving the versatility of the detection model in OCR recognition and avoiding the arduous task of repeatedly training small models for different scenarios and annotating data.

[0027] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following detailed description, they are used to explain the present disclosure but do not constitute a limitation of the present disclosure. In the accompanying drawings:

[0029] Figure 1 The figure is a flowchart of a text detection method according to an exemplary embodiment.

[0030] Figure 2 is based on Figure 1 The illustrated embodiment shows a flowchart of a text detection method.

[0031] Figure 3 The figure is a schematic diagram showing the architecture of a DBNet network according to an exemplary embodiment.

[0032] Figure 4 is based on Figure 1 The illustrated embodiment shows a flowchart of a model training method.

[0033] Figure 5 The figure is a block diagram of a text detection device according to an exemplary embodiment.

[0034] Figure 6 is based on Figure 5 The embodiment shown is a block diagram of a text detection device.

[0035] Figure 7 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0036] The following describes the specific embodiments of the present disclosure in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present disclosure and are not intended to limit the present disclosure.

[0037] The present disclosure is mainly used in scenarios where OCR detection of text in text images is performed. Most OCR detection models in related technologies are obtained through training based on a single scenario, that is, data is annotated one by one for each scenario to train a specific model. This will bring a lot of inconveniences in actual application. First, model training requires a large amount of annotated data, which is time-consuming and labor-intensive. In addition, in actual use, the same task may not be available for the model simply because the object has changed. For example, when training a model for correcting oral arithmetic questions, the training data is annotated printed data. After the training is completed, the model can be used to detect printed mathematical formulas. However, when encountering handwritten mathematical formulas, even if the mathematical formulas are the same, there will be detection misalignment or confusion between handwriting and print, resulting in errors in the final correction results. Therefore, the versatility of the OCR detection models obtained through training for a single scenario in related technologies is low. In addition, the labor cost of annotating training data is high.

[0038] Another related technology involves data augmentation, which creates new training samples by making small changes to the original training data (such as rotation, scaling, and cropping), or model ensembles, which combine predictions from multiple models to improve generalization. However, data augmentation methods rely heavily on the heterogeneity of the source data and the feasibility of the augmentation scheme. Model ensembles, however, are difficult to train due to the mathematical description of the problem and the coupling of models, making training expensive and unreliable.

[0039] To solve the above problems, the present disclosure provides a file detection method, device, storage medium, electronic device and program product. Specific implementation methods of the present disclosure are described in detail below with reference to the accompanying drawings.

[0040] Figure 1 FIG. 1 is a flowchart of a text detection method according to an exemplary embodiment. Figure 1 As shown, the method includes the following steps:

[0041] In step S101 , a text image to be detected is obtained.

[0042] The text image refers to a text image that requires OCR recognition. For example, in the scenario of automatically correcting oral arithmetic problems, the text image may include an image of the oral arithmetic equation to be corrected. In the scenario of intelligent office, the text image may include images of bills, contracts, forms, etc. that require character recognition.

[0043] Taking the recognition of oral arithmetic expressions as an example, oral arithmetic expressions include handwritten and printed expressions. Unlike the related art, the detection model trained using handwritten training samples can only be used to detect oral arithmetic expressions in handwritten form, and the detection model trained using printed training samples can only be used to detect oral arithmetic expressions in printed form. The target detection model provided by the present disclosure can adaptively recognize and detect oral arithmetic expressions in handwritten and printed form.

[0044] In step S102, after the text image is input into the target detection model, a target text detection result is obtained, wherein the target detection model includes a large language model and a text detection model, the output end of the large language model is coupled with the input end of the text detection model, and the target detection model is a model that has been fine-tuned using a variety of feature description instructions and sample images, and the feature description instructions represent the feature information of the text in the sample image.

[0045] A large language model (LLM) refers to a deep learning model trained using large amounts of text data. It can generate natural language text or understand the meaning of text. Common large language models include ChatGPT and LLaMA. The text detection model can include the DBNet detection model.

[0046] In the present disclosure, the model architecture of the target detection model used for text detection includes a large language model and a text detection model. The output end of the large language model is coupled with the input end of the text detection model. Compared with the detection model in the related art that is only applicable to a single scenario, the present disclosure can be based on the general characteristics of the large language model, that is, it performs better in a wide range of tasks, not just for specific tasks. After fine-tuning the model using multiple feature description instructions, the target detection model can adaptively identify texts with different attribute features, thereby improving the versatility of the detection model for OCR recognition.

[0047] The feature description instruction refers to a natural language description of the attribute features of the text in the corresponding input sample image (or referred to as "text features"). The feature description instruction can be in the form of text or speech, and the present disclosure does not limit the specific form of the feature description instruction.

[0048] For example, the text features of the sample image described by the feature description instruction include at least one of the following: text style, corresponding character category, presence of background interference, text position, text size, text orientation, text context, pre-set character types within the text, and the corresponding recognition scenario. Text style instructs the model to focus on different font types, such as handwriting, print, and cursive, enabling the model to distinguish and recognize text of different styles. Character category instructs the model to learn features of specific character categories, such as numbers, letters, and Chinese characters, to help the model better recognize various types of text. Background interference instructs the model to handle complex background interference, such as shadows, noise, and occlusion, thereby enhancing model robustness. Text position may include the position of each character within the image text. Text position instructs the model to focus on text location information, such as horizontal, vertical, and tilted text, to improve the model's ability to locate text. Text size instructs the model to handle text of different sizes, including large, medium, and small, ensuring the model can adapt to text of varying sizes. Text orientation instructs the model to handle text of different orientations, such as horizontal, vertical, and diagonal, enhancing model flexibility. The text context is used to instruct the model to learn contextual information, such as words, sentences, paragraphs, etc., which helps to improve the overall recognition rate. The preset types of characters in the text refer to pre-set special characters, such as punctuation marks, ligatures, abbreviations, etc. After the preset types of characters in the text are input into the model, the model can be instructed to learn such special characters to enrich the model's knowledge base. The recognition scenario corresponding to the text refers to the scenario corresponding to the current text detection task. The recognition scenario may include, for example, a text recognition task and a task type, etc., wherein the text recognition task may be, for example, test paper grading, and the task type may include information such as the subject, grade, and specific question type corresponding to the test paper. The above lists commonly used text features. In actual application scenarios, the text features described by the feature description instruction may also include speed requirements for text detection, as well as common text recognition errors (such as missing strokes, overlapping characters, etc.). The text recognition errors can be used to instruct the model to learn to automatically correct common recognition errors.

[0049] In this way, the target detection model obtained by fine-tuning the instructions using the above-mentioned multiple feature description instructions has the ability to adaptively detect text images with different text features.

[0050] Using the above method, text detection is performed on the text image to be detected through the target detection model. Since the architecture of the target detection model uses a large language model and a text detection model, the general characteristics of the large language model can be utilized, that is, it performs better in a wide range of tasks, not just for specific tasks. After fine-tuning the model using multiple feature description instructions, the target detection model can adaptively recognize text with different attribute features, thereby improving the versatility of the detection model for OCR recognition, and also improving the accuracy and detection efficiency of the OCR system, while avoiding the arduous task of repeated training of small models for different scenarios and labeling data.

[0051] Figure 2 is based on Figure 1 The embodiment shown is a flowchart of a text detection method, as shown in FIG. Figure 2 As shown, step S102 includes the following sub-steps:

[0052] In step S1021, after the text image is input into the large language model, a first feature map output by the large language model and feature description information of the text in the text image are obtained.

[0053] After the text image is input into the large language model, the large language model can encode the text image into the first feature map. Since the large language model inherently has the ability to understand the meaning of language text, after fine-tuning the model's instructions using multiple feature description instructions, the large language model is able to automatically understand the text features in the input text image. Therefore, after the text image is input into the large language model, the large language model can output feature description information of the text in the text image.

[0054] Among them, the feature description information can be used to describe the attribute characteristics of the text in the text image (such as whether the font style is handwritten or printed), character category (such as numbers, letters, Chinese characters, etc.), character position information, and whether the character has background interference and other features.

[0055] After the text image is input into the large language model, not only the first feature map of the text image is output, but also the feature description information of the text in the text image can be output. In this way, after the first feature map and the feature description information are subsequently input into the text detection model, it is equivalent to providing the text detection model with text feature prompts. Compared with the text detection model that simply performs text detection based on text images, the accuracy and efficiency of text detection can be significantly improved.

[0056] It should be noted that the text image may be pre-processed before being input into the large language model. Pre-processing the text image includes operations such as clarity enhancement, noise removal, and character segmentation to improve image quality and facilitate subsequent text detection and recognition.

[0057] In step S1022, the first feature map and the feature description information are input into the text detection model to obtain the target text detection result.

[0058] The text detection model may include a DBNet network model. The target text detection result includes the position information of the text in the image and the specific text content. The position information can be represented by the vertex coordinates of the text box where the text is located.

[0059] In one possible implementation, the DBNet network includes a ResNet network and an FPN (Feature Pyramid Network) connected to the ResNet network.

[0060] In this step, the first feature map and feature description information can be input into the ResNet network to obtain multiple second feature maps of different scales; then, the multiple second feature maps are feature enhanced by FPN to obtain multiple feature enhanced maps of different scales; after feature fusion of the multiple feature enhanced maps of different scales, a fused feature map is obtained; after determining the text detection probability map and text detection threshold map corresponding to the text image according to the fused feature map, the target text detection result is determined according to the text detection probability map and the text detection threshold map.

[0061] Here, the first feature map and feature description information are input into the ResNet network. Based on the first feature map, the ResNet network can generate multiple second feature maps of different scales based on the feature description information. Compared with feature maps generated without feature description information, these second feature maps represent text features in the image more explicitly and comprehensively, thereby helping to improve the accuracy and efficiency of text detection.

[0062] For example, Figure 3 FIG. 1 is a schematic diagram showing an architecture of a DBNet network according to an exemplary embodiment. Figure 3As shown, by inputting the first feature map and feature description information into the ResNet network, second feature maps of five scales, 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32, can be obtained. After performing feature enhancement on multiple second feature maps through FPN, feature enhancement maps of three scales, 1 / 4, 1 / 8, and 1 / 16, are obtained. Feature fusion is then performed on these three scales of feature enhancement maps and the second feature map that has been upsampled 8 times, resulting in a fused feature map of 1 / 4. Then, based on the fused feature map, a text detection probability map and a text detection threshold map are output, and a text detection result can be output. Since the ResNet network generates multiple second feature maps of different scales based on the first feature map based on the prompts of the feature description information, the text detection probability map and text detection threshold map generated based on the multi-scale second feature maps are also obtained by fine-tuning based on the prompts of the feature description information. Therefore, the accuracy of the text detection probability map and text detection threshold map is better, thereby improving the accuracy of text detection.

[0063] It should be noted that in the text detection model disclosed in the present invention, FPN is used to enhance the feature representation of text images, but the enhanced feature enhancement maps of multiple scales are fused to obtain a fused feature map, which can make full use of the information of feature maps at different levels and help improve the robustness and accuracy of the model.

[0064] Figure 4 is based on Figure 1 The embodiment shown is a flow chart of a model training method, as shown in FIG. Figure 4 As shown in the figure, the target detection model can be obtained by pre-tuning the instructions through the following steps:

[0065] In step S401 , for each sample image, a feature description instruction corresponding to the sample image is constructed, and different sample images correspond to different feature description instructions.

[0066] The feature description instruction can be a natural language description of the text features in the sample image. For example, the feature description instruction can include multi-dimensional descriptions of the text features in the sample image, such as "the text in the image is handwritten, not printed," "the type of the text in the image is digital," or "the text in the image is horizontally distributed within the image." In actual model fine-tuning scenarios, the instruction set of the feature description instruction can be enriched based on actual business needs to meet different text detection requirements.

[0067] It should be noted that if the purpose of fine-tuning the model is to enable the fine-tuned model to adaptively detect handwriting and print, then the sample images used to fine-tune the model need to include images of handwritten text and images of printed text. In this way, after fine-tuning the parameters of the pre-trained large language model and the pre-trained text detection model using sample images obtained by mixing these two types of images, the target detection model that can adaptively recognize handwriting and print can be obtained.

[0068] In step S402, for each of the sample images, the sample image and the feature description instruction corresponding to the sample image are input into a pre-trained large language model to obtain a sample feature map output by the pre-trained large language model and sample feature description information corresponding to the sample image.

[0069] The pre-trained large language model may be, for example, an LLaMA model.

[0070] In step S403, the sample feature map and the sample feature description information are input into a pre-trained text detection model to obtain a model output for text detection.

[0071] The pre-trained text detection model may be, for example, a DBNet network model.

[0072] In this step, the sample feature description information is input into the pre-trained text detection model. This sample feature description information can be regarded as a natural language prompt for the text features in the sample image. In the related art, when performing text detection on text images based on text detection models, natural language prompts for text features are not used. The use of natural language prompts in the present disclosure can make the model more flexible and customizable, and can adjust its prediction behavior according to different prompts. This allows users to control the detection behavior of the model through simple natural language instructions, thereby adapting to different text detection scenarios without retraining the model.

[0073] In step S404, according to the model output of the text detection and the text detection label corresponding to the sample image, the parameters of the pre-trained large language model and the pre-trained text detection model are fine-tuned through a preset loss function to obtain the target detection model.

[0074] The model output is the pixel probability sequence corresponding to the sample image predicted by the model (referring to the pre-trained large language model and pre-trained text detection model to be fine-tuned). Each element in the pixel probability sequence represents the probability that the corresponding pixel belongs to a text pixel of the text to be detected. The text detection label includes a sequence of pixel annotations of text pixels in the sample image. The text detection label (i.e., the true value) is a series of text pixels in the sample image that have been pre-annotated for the sample image.

[0075] Because both the model output and the text detection labels are in sequence form, before fine-tuning the model's instructions in this disclosure, the loss function determined for training is a function that measures the difference between the predicted sequence and the true sequence. In one possible implementation, the preset loss function can be a cross-entropy loss function.

[0076] In the process of executing this step, the target loss value can be calculated according to the pixel point probability sequence and the pixel annotation sequence through the cross entropy loss function, and the parameters of the pre-trained large language model and the pre-trained text detection model can be fine-tuned based on the back propagation algorithm according to the target loss value to obtain the target detection model.

[0077] It should be noted that, considering that the pre-trained large language model is a deep learning model pre-trained using a large amount of text data, the model has strong versatility. Therefore, in the instruction fine-tuning stage of the model, the present disclosure only uses a small number of sample images and their corresponding feature description instructions to fine-tune the model to adaptively identify text with specific different attribute features, without the need to annotate a large number of sample images. In addition, the present disclosure combines the fine-tuning and generalization capabilities of the LLaMA large model with the DBNet detection model, that is, using the instruction set of the large model to indirectly perform adaptive adjustments to text detection, so that the detection application can be quickly migrated to different scenarios, avoiding the repeated training of small models and the heavy task of annotating data.

[0078] It should also be noted that during instruction fine-tuning, appropriate hyperparameters must be set for the model, including, for example, batch size and learning rate. The batch size refers to the number of samples used in each gradient update step. Although a smaller batch size may make the training process more unstable, it can also better explore the parameter space. Therefore, in this disclosure, in order to enable the fine-tuned model to obtain better generalization capabilities, a smaller batch size can be selected. For example, the batch size can be set to 16 or 32. In addition, the learning rate determines the speed at which the model parameters are updated. A larger learning rate may make the training converge faster, but it may also cause the training process to oscillate around the optimal solution; a smaller learning rate can make the training process more stable, but it may require more iterations to converge. In this disclosure, the learning rate setting can start with a small value, such as 1e-4 or 5e-5, and then gradually increase until the optimal value is found.

[0079] In addition to these optimized parameter settings, instruction fine-tuning also needs to consider balanced data distribution. That is, when selecting sample images, it is necessary to configure a balanced configuration of sample images corresponding to different recognition objects or recognition scenarios.

[0080] In addition, in order to further enhance the generalization ability of the model, during the stage of fine-tuning the model instructions, a reward mechanism can be used for small sample sample images to achieve text detection capabilities in educational scenarios based on large language models and file detection models. Among them, the principle of the reward mechanism is to use a reward function to encourage the model to learn more general representations on small sample data. One possible method is to design rewards based on the performance of the model on the validation set. The model parameters can also be tuned using a small sample thinking chain method (for example, CoT-Influx can be used to enable the model to perform inference learning on small sample examples). The specific implementation method can also refer to the description in the relevant literature and is not specifically limited here.

[0081] The object detection model trained using the aforementioned fine-tuning method is suitable for a variety of OCR scenarios involving mixed handwritten and printed text, including some mixed English and mathematical symbols. The model is also highly robust, supporting service deployment on high-performance servers. Deploying services based on a multi-process service framework meets high concurrency requirements and ensures robust service calls.

[0082] Figure 5 is a block diagram of a text detection device according to an exemplary embodiment. Figure 5 As shown, the device includes:

[0083] An acquisition module 501 is used to acquire a text image to be detected;

[0084] A detection module 502 is configured to input the text image into a target detection model to obtain a target text detection result;

[0085] Among them, the target detection model includes a large language model and a text detection model, the output end of the large language model is coupled with the input end of the text detection model, and the target detection model is a model that has been fine-tuned using multiple feature description instructions and sample images, and the feature description instructions represent the feature information of the text in the sample image.

[0086] Optionally, the detection module 502 is used to input the text image into the large language model, obtain the first feature map output by the large language model and the feature description information of the text in the text image; and input the first feature map and the feature description information into the text detection model to obtain the target text detection result.

[0087] Optionally, the text detection model includes a DBNet network, and the DBNet network includes a ResNet network and a feature pyramid network FPN connected to the ResNet network; the detection module 502 is used to input the first feature map and the feature description information into the ResNet network to obtain multiple second feature maps of different scales; after performing feature enhancement on the multiple second feature maps through the FPN, multiple feature enhancement maps of different scales are obtained; after performing feature fusion on the multiple feature enhancement maps of different scales, a fused feature map is obtained; after determining the text detection probability map and the text detection threshold map corresponding to the text image according to the fused feature map, the target text detection result is determined according to the text detection probability map and the text detection threshold map.

[0088] Optionally, Figure 6 is based on Figure 5 The embodiment shown is a block diagram of a text detection device, such as Figure 6 As shown, the device also includes:

[0089] The model fine-tuning module 503 is configured to pre-fine-tune the target detection model by:

[0090] For each sample image, construct a feature description instruction corresponding to the sample image, where different sample images correspond to different feature description instructions;

[0091] For each of the sample images, inputting the sample image and the feature description instruction corresponding to the sample image into a pre-trained large language model to obtain a sample feature map output by the pre-trained large language model and sample feature description information corresponding to the sample image;

[0092] After inputting the sample feature map and the sample feature description information into a pre-trained text detection model, a model output for text detection is obtained;

[0093] According to the model output of the text detection and the text detection label corresponding to the sample image, the parameters of the pre-trained large language model and the pre-trained text detection model are fine-tuned through a preset loss function to obtain the target detection model.

[0094] Optionally, the preset loss function includes a cross-entropy loss function, the model output includes a pixel probability sequence corresponding to the sample image obtained by model prediction, each element in the pixel probability sequence represents the probability that the corresponding pixel belongs to a text pixel of the text to be detected; the text detection label includes a pixel annotation sequence belonging to the text pixel in the sample image;

[0095] The model fine-tuning module 503 is used to calculate the target loss value through the cross entropy loss function according to the pixel point probability sequence and the pixel annotation sequence, and then fine-tune the parameters of the pre-trained large language model and the pre-trained text detection model according to the target loss value to obtain the target detection model.

[0096] Optionally, the text feature described in the feature description instruction includes at least one of the following:

[0097] Text style, character category corresponding to the text, whether there is background interference, text position, text size, text direction, text context, preset types of characters in the text, and the recognition scenario corresponding to the text.

[0098] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0099] Figure 7 FIG. 1 is a block diagram of an electronic device according to an exemplary embodiment. Figure 7 As shown, the electronic device 700 may include: a processor 701 , a memory 702 , and may further include one or more of a multimedia component 703 , an input / output (I / O) interface 704 , and a communication component 705 .

[0100] The processor 701 is used to control the overall operation of the electronic device 700 to complete all or part of the steps in the above-mentioned text detection method. The memory 702 is used to store various types of data to support the operation of the electronic device 700. Such data may include, for example, instructions for any application or method operating on the electronic device 700, as well as application-related data, such as contact information, sent and received messages, pictures, audio, video, etc. The memory 702 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The multimedia component 703 may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may be further stored in the memory 702 or sent through the communication component 705. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 704 provides an interface between the processor 701 and other interface modules. The above-mentioned other interface modules may be a keyboard, a mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 705 is used for wired or wireless communication between the electronic device 700 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G or 4G, or a combination of one or more thereof, so the corresponding communication component 705 may include: a Wi-Fi module, a Bluetooth module, an NFC module.

[0101] In an exemplary embodiment, the electronic device 700 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-mentioned text detection method.

[0102] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided. When the program instructions are executed by a processor, the steps of the above-mentioned text detection method are implemented. For example, the computer-readable storage medium may be the aforementioned memory 702 including the program instructions. The program instructions may be executed by the processor 701 of the electronic device 700 to perform the above-mentioned text detection method.

[0103] In another exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program that can be executed by a processor. When the computer program is executed by the processor, the steps of the above text detection method are implemented.

[0104] The preferred embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings. However, the present disclosure is not limited to the specific details of the above embodiments. Within the technical concept of the present disclosure, various simple modifications can be made to the technical solutions of the present disclosure, and these simple modifications all fall within the scope of protection of the present disclosure.

[0105] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any appropriate manner without contradiction. In order to avoid unnecessary repetition, the present disclosure will not further describe various possible combinations.

[0106] In addition, the various embodiments of the present disclosure may be arbitrarily combined, and as long as they do not violate the concept of the present disclosure, they should also be regarded as the contents disclosed by the present disclosure.

Claims

1. A text detection method, characterized in that: The method comprises: Get the text image to be detected; After inputting the text image into the target detection model, a target text detection result is obtained; Among them, the target detection model includes a large language model and a text detection model, the output end of the large language model is coupled with the input end of the text detection model, and the target detection model is a model that has been fine-tuned using multiple feature description instructions and sample images, and the feature description instructions represent the feature information of the text in the sample image.

2. The method according to claim 1, characterized in that After inputting the text image into the target detection model, obtaining the target text detection result includes: After inputting the text image into the large language model, obtaining a first feature map output by the large language model and feature description information of the text in the text image; After the first feature map and the feature description information are input into the text detection model, the target text detection result is obtained.

3. The method according to claim 2, characterized in that The text detection model includes a DBNet network, and the DBNet network includes a ResNet network and a feature pyramid network FPN connected to the ResNet network; after inputting the first feature map and the feature description information into the text detection model, obtaining the target text detection result includes: Inputting the first feature map and the feature description information into the ResNet network to obtain a plurality of second feature maps of different scales; After performing feature enhancement on the plurality of second feature maps through the FPN, a plurality of feature enhancement maps of different scales are obtained; After performing feature fusion on the multiple feature enhancement maps of different scales, a fused feature map is obtained; After determining a text detection probability map and a text detection threshold map corresponding to the text image according to the fused feature map, the target text detection result is determined according to the text detection probability map and the text detection threshold map.

4. The method according to claim 1, wherein The target detection model is obtained by pre-tuning the instructions in the following way: For each sample image, construct a feature description instruction corresponding to the sample image, where different sample images correspond to different feature description instructions; For each of the sample images, inputting the sample image and the feature description instruction corresponding to the sample image into a pre-trained large language model to obtain a sample feature map output by the pre-trained large language model and sample feature description information corresponding to the sample image; After inputting the sample feature map and the sample feature description information into a pre-trained text detection model, a model output for text detection is obtained; According to the model output of the text detection and the text detection label corresponding to the sample image, the parameters of the pre-trained large language model and the pre-trained text detection model are fine-tuned through a preset loss function to obtain the target detection model.

5. The method according to claim 4, characterized in that The preset loss function includes a cross-entropy loss function, the model output includes a pixel probability sequence corresponding to the sample image obtained by model prediction, each element in the pixel probability sequence represents the probability that the corresponding pixel belongs to a text pixel of the text to be detected; the text detection label includes a pixel annotation sequence belonging to the text pixel in the sample image; The object detection model is obtained by fine-tuning the parameters of the pre-trained large language model and the pre-trained text detection model according to the model output of the text detection and the text detection label corresponding to the sample image through a preset loss function, including: According to the pixel point probability sequence and the pixel annotation sequence, after calculating the target loss value through the cross entropy loss function, the parameters of the pre-trained large language model and the pre-trained text detection model are fine-tuned according to the target loss value to obtain the target detection model.

6. The method according to any one of claims 1 to 5, characterized in that The text features described in the feature description instruction include at least one of the following: Text style, character category corresponding to the text, whether there is background interference, text position, text size, text direction, text context, preset types of characters in the text, and the recognition scenario corresponding to the text.

7. A text detection device, characterized in that: The device comprises: An acquisition module, used to acquire a text image to be detected; A detection module, configured to input the text image into a target detection model to obtain a target text detection result; Among them, the target detection model includes a large language model and a text detection model, the output end of the large language model is coupled with the input end of the text detection model, and the target detection model is a model that has been fine-tuned using multiple feature description instructions and sample images, and the feature description instructions represent the feature information of the text in the sample image.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

9. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images