Scene text retrieval method and device based on automatic detection screening and storage medium

Abstract

This invention relates to a scene text retrieval method, apparatus, and storage medium based on automatic detection and screening. It extracts features from scene images and query text and calculates feature similarity. The method comprises an image branching model, an automatic detection and screening model, and a text branching model. First, the image branching model detects text-like instances in the scene image and extracts their features. Then, the automatic detection and screening model removes all non-text instances from the text-like instances, retaining true text features. The query text is then fed into the text branching model to extract features. The training of the above models is guided by a distributional similarity metric loss function and a multi-granularity text prediction loss function. During inference, the similarity between the features generated by the image branching model and the features generated by the text branching model is calculated. The desired scene text image instance is retrieved based on the similarity score. Compared with existing technologies, this invention uses an automatic detection and screening model to remove non-text instances that interfere with retrieval and improves the similarity learning process by improving the loss function, thereby enhancing the accuracy of similarity calculation and achieving more efficient scene text retrieval.

Description

Technical Field

[0001] This invention relates to the field of scene image retrieval, and in particular to a method, apparatus, and storage medium for scene text retrieval based on automatic detection and screening. Background Technology

[0002] The goal of scene-based text retrieval is to find images in an image library that contain a given query text and pinpoint the exact location of the text within those images. Most methods addressing this problem focus on learning better feature mapping spaces and similarity calculation methods. While mapping images and text to a common feature space can improve the accuracy of feature similarity calculation, this cross-modal feature space learning always faces the challenge of modal margin. Existing similarity calculation methods primarily rely on cosine similarity calculation of features or on recognizing text instances in an image as text and directly calculating the similarity between the image text and the query text. The use of these two methods in various models is generally accepted and applicable.

[0003] Text instance detection plays a crucial role in scene text retrieval tasks, yet it is often overlooked in existing technologies. Although instance detection modules are trained alongside scene text retrieval models, commonly used anchor-free detection modules still detect many background instances—non-text instances lacking textual information. In subsequent similarity learning, these non-text instances are defaulted to text instances by the model, interfering with optimization and learning, ultimately impacting retrieval performance. Therefore, an automatic bounding box filtering model can help further filter out non-text instances, improving model performance. Furthermore, due to the sequential nature of text, extracted text features contain rich sequence information; while image features contain more semantic, textural, or spatial information. This difference in feature information also hinders the learning of a common feature space. Compared to enriching the texture and spatial information of text, enhancing the sequence features of images is more reasonable and easier. A common approach is to predict the specific character corresponding to an image instance; this fine-grained prediction method requires the model to accurately identify characters and process their order. Other methods enrich sequence information from a coarse-grained perspective, such as predicting the length of the text corresponding to the image instance or the number of characters in the corresponding text. These methods can compensate for the missing sequence information in image features to some extent, but it is difficult to capture more comprehensive sequence information by only performing coarse-grained or fine-grained processing. Summary of the Invention

[0004] The purpose of this invention is to provide a method, device, and storage medium for scene text retrieval based on automatic detection and screening.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] A scene text retrieval method based on automatic detection and screening includes:

[0007] The system receives the input query text and inputs it into the scene text retrieval model, wherein the scene text retrieval model includes an image branching model, a text branching model, and an automatic detection and screening model.

[0008] The query text is fed into a trained text branching model to obtain the text sequence features of the query text;

[0009] Obtain the image sequence features of each image in the target image library;

[0010] Calculate the similarity between the image sequence features of each image in the target image library and the text sequence features of the query text, and return the image containing the query text based on the similarity calculation results;

[0011] The training process of the scene text retrieval model includes:

[0012] Step S1: Preprocess and augment the image and text datasets;

[0013] Step S2: The preprocessed and augmented scene image data is fed into the image branching model to detect text-like instances in each image, and then fed into the automatic detection and screening model to remove interfering non-text instances, thus obtaining the image sequence features of each image in the image dataset;

[0014] Step S3: Feed each text in the text dataset into the text branching model to obtain text sequence features;

[0015] Step S4: Constrain the similarity calculation results of image sequence features and text sequence features from a distributional perspective using a distributional similarity metric loss function, aligning the similarity distribution with the text edit distance score distribution. The distributional similarity metric loss function L... dis for:

[0016]

[0017] Where: N is the number of data points. For the i-th image sequence feature, For the j-th text sequence feature, The text corresponding to the features of the i-th image sequence. W is the text corresponding to the feature of the j-th text sequence. D For the trainable weights, ED(·) is the regularized edit distance function. for and The similarity calculation results are as follows:

[0018]

[0019] Where: ||·|| is the Manhattan distance function.

[0020] Furthermore, in step S2, the automatic detection and screening model uses a text classification loss function to distinguish non-text instances detected by the image branching model. The text classification loss function L cls Specifically:

[0021]

[0022] Where: Y i E represents whether the i-th instance is text or non-text. i This represents the prediction result of the automatic detection and screening model for the i-th instance; M is the number of all instances detected by the image branching model.

[0023] Furthermore, the image branching model uses a multi-granularity text prediction loss function to predict the characters, text length, and number of characters of the ground truth text corresponding to the image sequence features, and compares them with the ground truth to enhance sequence information. The multi-granularity text prediction loss function L... txt for:

[0024]

[0025] Among them: W V and b V To predict trainable weights and biases in characters, V i W represents the truth value of the text character corresponding to the i-th instance, and is a one-hot vector. L and b L To predict trainable weights and biases within a text length, L i W represents the text length corresponding to the i-th instance. C and b C To predict the trainable weights and biases in the frequency of text characters, C i This represents the number of text characters corresponding to the i-th instance.

[0026] Furthermore, the image branching model includes:

[0027] The text instance detection module is used to detect all text-like instances in an image and extract instance features;

[0028] The sequence feature module is used to obtain image sequence features based on the instance features of text instances in the image;

[0029] The basic network structure of the text instance detection module is as follows:

[0030] S = FPN(Res(I))

[0031] Where: Res represents the residual network, consisting of 5 sub-network modules, all of which are convolutional networks; FPN represents the pyramid network, consisting of 3 convolutional network modules; I represents the input image; and S represents the extracted image features, which are used to generate all text-like instance detection boxes.

[0032]

[0033] Among them: W B and b B For trainable weights and biases; B = {b i |i=1…N} is the set of all generated text instance detection boxes, b i Given the detection box for the i-th instance, extract the feature set R = {R_i} of N instances from the feature S based on the detection box. i |i=1…N}, where Ri is the feature of the i-th instance;

[0034] The image sequence features obtained by the sequence feature module are:

[0035]

[0036] Wherein: Conv Ii BiLSTM represents the i-th convolutional module in the sequence feature module. I A is a bidirectional long short-term memory network module. i Instance features for text instances.

[0037] Furthermore, the automatic detection and screening model filters out all non-text instances in the feature set of the instances and removes them. The structure of the automatic detection and screening model is as follows:

[0038]

[0039] in: and E represents trainable weights and biases. i This represents the probability that each instance output by the automatic detection and screening model is a text instance.

[0040] Furthermore, the loss function L of the text instance detection module det for:

[0041]

[0042] Among them: lb i Represents prediction box b iThe corresponding truth boxes; x∩y represents the area of ​​the intersection of boxes x and y, and x∪y represents the area of ​​the union of boxes x and y.

[0043] Furthermore, the text sequence features obtained by the text branching model are:

[0044]

[0045] Wherein: Conv Ti BiLSTM represents the i-th convolutional module in the sequence feature module. T W is a bidirectional long short-term memory network module. T and b T T represents the trainable weights and biases. i This represents the text corresponding to the i-th instance.

[0046] Furthermore, the preprocessing of the image dataset includes image color adjustment, image size scaling, image pixel normalization, and image tensor quantization, while the preprocessing of the text dataset includes one-hot vectorization.

[0047] A scene text retrieval device based on automatic detection and screening includes a memory, a processor, and a program stored in the memory, wherein the processor executes the program to implement the method described above.

[0048] A storage medium having a program stored thereon, which, when executed, implements the method described above.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] (1) An automatic detection and screening model is proposed to screen out non-text instances and eliminate interference, while retaining text instances, thus ensuring the purity of the model similarity calculation and further improving the accuracy of the model similarity calculation.

[0051] (2) Using the distribution similarity metric loss function to guide the model’s similarity learning process from the distribution perspective, helping the model understand the differences between the true values ​​of feature similarity and edit distance in the distribution, and improving the model’s generalization in similarity calculation.

[0052] (3) Use multi-granularity text prediction loss function to enrich the sequence information of the image, assist the model in generating features from different perspectives, so that the features contain diverse sequence information, reduce the complexity of similarity learning with text features, and accelerate the training of the model. Attached Figure Description

[0053] Figure 1 This is a flowchart of the scene text retrieval method of the present invention;

[0054] Figure 2This is a diagram of the scenario text retrieval model architecture of the present invention. Detailed Implementation

[0055] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0056] A scene text retrieval method based on automatic detection and screening, such as Figure 1 and Figure 2 As shown, it includes:

[0057] The system receives the input query text and inputs it into the scene text retrieval model, which includes an image branching model, a text branching model, and an automatic detection and screening model.

[0058] The query text is fed into a trained text branching model to obtain the text sequence features of the query text;

[0059] Obtain the image sequence features of each image in the target image library;

[0060] Calculate the similarity between the image sequence features of each image in the target image library and the text sequence features of the query text, and return the image containing the query text based on the similarity calculation results;

[0061] The training process of the scene text retrieval model includes:

[0062] Step S1: Preprocess and augment the image and text datasets. The preprocessing of the image dataset includes image color adjustment, image size scaling, image pixel normalization, and image tensor quantization. The preprocessing of the text dataset includes one-hot vectorization.

[0063] Step S2: The preprocessed and augmented scene image data is fed into the image branching model to detect text-like instances in each image, and then fed into the automatic detection and screening model to remove interfering non-text instances, thus obtaining the image sequence features of each image in the image dataset;

[0064] Step S3: Feed each text in the text dataset into the text branching model to obtain text sequence features;

[0065] Step S4: The training process is divided into pre-training and fine-tuning. First, pre-training is performed on a synthetic dataset, then fine-tuning is done on a real dataset. The distributional similarity loss function constrains the similarity calculation results of image sequence features and text sequence features from a distributional perspective, aligning the similarity distribution with the text edit distance score distribution. The distributional similarity loss function L... dis for:

[0066]

[0067] Where: N is the number of data points. For the i-th image sequence feature, For the j-th text sequence feature, The text corresponding to the features of the i-th image sequence. W is the text corresponding to the feature of the j-th text sequence. D For the trainable weights, ED(·) is the regularized edit distance function. for and The similarity calculation results are as follows:

[0068]

[0069] Where: ||·|| is the Manhattan distance function.

[0070] Among them, the automatic detection and screening model uses a text classification loss function to distinguish non-text instances detected by the image branching model. The text classification loss function L cls Specifically:

[0071]

[0072] Where: Y i E represents whether the i-th instance is text or non-text. i This represents the prediction result of the automatic detection and screening model for the i-th instance; M is the number of all instances detected by the image branching model.

[0073] The image branching model uses a multi-granularity text prediction loss function to predict the characters, text length, and number of characters in the ground truth text corresponding to the features of the image sequence, and compares them with the ground truth to enhance sequence information. The multi-granularity text prediction loss function L... txt for:

[0074]

[0075] Among them: W V and b V To predict trainable weights and biases in characters, V i W represents the truth value of the text character corresponding to the i-th instance, and is a one-hot vector. L and b L To predict trainable weights and biases within a text length, L i W represents the text length corresponding to the i-th instance. C and b C To predict the trainable weights and biases in the frequency of text characters, C iThis represents the number of text characters corresponding to the i-th instance.

[0076] Image branching models include:

[0077] The text instance detection module is used to detect all text-like instances in an image and extract instance features;

[0078] The sequence feature module is used to obtain image sequence features based on the instance features of text instances in the image;

[0079] The basic network structure of the text instance detection module is as follows:

[0080] S = FPN(res(I))

[0081] Where: Res represents the residual network, consisting of 5 sub-network modules, all of which are convolutional networks; FPN represents the pyramid network, consisting of 3 convolutional network modules; I represents the input image; and S represents the extracted image features, which are used to generate all text-like instance detection boxes.

[0082]

[0083] Among them: W B and b B For trainable weights and biases; B = {b i |i=1…N} is the set of all generated text instance detection boxes, b i Given the detection box for the i-th instance, extract the feature set R = {R_i} of N instances from the feature S based on the detection box. i |i=1…N},R i Features of the i-th instance;

[0084] The image sequence features obtained by the sequence feature module are:

[0085]

[0086] Wherein: Conv Ii BiLSTM represents the i-th convolutional module in the sequence feature module. I A is a bidirectional long short-term memory network module. i Instance features for text instances.

[0087] The automatic detection and screening model filters out all non-text instances from the feature set of instances and removes them. The structure of the automatic detection and screening model is as follows:

[0088]

[0089] in: and E represents trainable weights and biases. i This represents the probability that each instance output by the automatic detection and screening model is a text instance.

[0090] The loss function L of the text instance detection module det for:

[0091]

[0092] Among them: lb i Represents prediction box b i The corresponding truth boxes; x∩y represents the area of ​​the intersection of boxes x and y, and x∪y represents the area of ​​the union of boxes x and y.

[0093] The text sequence features obtained by the text branching model are:

[0094]

[0095] Wherein: Conv Ti BiSTM represents the i-th convolutional module in the sequence feature module. T W is a bidirectional long short-term memory network module. T and b T T represents the trainable weights and biases. i This represents the text corresponding to the i-th instance.

[0096] The method provided in this application is illustrated in a specific case as follows:

[0097] (1) Preprocessing and data augmentation of image and text datasets.

[0098] The dataset includes several scene images and all text appearing in the images. Preprocessing and data augmentation methods are consistent throughout pre-training and fine-tuning. In this process, images first undergo color adjustments such as brightness, contrast, saturation, and hue. Specifically, the brightness and contrast are randomly increased by a factor of 0.875–1.125, and the saturation and hue are increased by a factor of 0.5–1.5. Then, the images are scaled to 640×640, and the mean and standard deviation of all image pixels in the dataset are calculated. The average pixel value is then subtracted from the mean and divided by the standard deviation to complete standardization. Finally, the images are converted from array type to tensor type, completing image augmentation and preprocessing.

[0099] The preprocessing of text data adopts one-hot vectorization. In this embodiment, the query text is "Lanzhou Beef Noodles". Assuming that the character library of the total dataset has 8 characters in total, namely "Zhong", "Guo", "Yin", "Hang", "Lan", "Zhou", "La" and "Mian", the basic feature of each character is an 8-dimensional vector 00000000. For the character "Lan", which is at the 5th position in the character library, its one-hot vector is 00001000. The remaining characters are also converted into one-hot vectors according to this rule, and finally all the vectors are spliced to obtain the one-hot vector of the entire text, completing the preprocessing of the text.

[0100] (2) The processed image data is sent to the IDM and ASM models to generate image sequence feature F I , and the text data is sent to the TEM model to generate text sequence feature F T , for joint training under the guidance of the loss function.

[0101] (2.1) The image data I is sent into the IDM model. The IDM model includes a text instance detection module and a sequence feature module. The text instance detection module is responsible for detecting all text-like instances in the image I and preliminarily extracting the features of the instances. Certainly, some of these instances may not be real text instances. The structure of this module mainly includes a residual network and a pyramid network:

[0102] S=FPN(Res(I))

[0103] where Res represents the residual network, which is composed of 5 sub-network modules all being convolutional networks; FPN represents the pyramid network, which is composed of 3 convolutional network modules; I represents the input scene text image; S represents the image feature extracted by IDM, with a feature size of 256×40×40. The feature S is then input to generate the detection boxes of all text-like instances in the image I:

[0104]

[0105] where W B and b B are the trainable weight and bias value; B={b i |i=1…N} is the set of all generated detection boxes of text-like instances, b i is the detection box of the i-th instance, and one detection box consists of 4 values, which are respectively the horizontal and vertical coordinates of the top-left corner and the horizontal and vertical coordinates of the bottom-right corner. The feature set R={R of these N instances is extracted from the feature S according to the detection boxes i |i=1…N}, R i is the feature of the i-th instance, and the final feature size is 256×4×15.

[0106] Compared to typical convolutional networks, residual networks have deeper layers, allowing the model to access deeper features while improving convergence stability. Pyramid networks can extract features of different sizes, possessing receptive fields of multiple scales, and can encompass text features of various scales, providing more comprehensive information. The detector head network uses exponential and reciprocal operations to ensure that the output value is between 0 and 1, preventing the output bounding box from being too large.

[0107] (2.2) The obtained instance features are fed into the ASM model, and the obtained scores E are used to determine the model's performance. i To filter out and remove all instances of text in R:

[0108]

[0109] in and E represents trainable weights and biases. i This represents the probability that the i-th instance output by the ASM model is a text instance. According to E... i The probability can be used to further filter out non-text instances, retaining text instances with the characteristic A. i The feature size is also 256×4×15. In this embodiment, the threshold is set to 0.5, i.e., E. i Instances with a value greater than 0.5 are retained as text instances, while instances with a value less than 0.5 are deleted.

[0110] (2.3) The filtered text instance features A i The data is then fed into the subsequent sequence feature module to further extract sequence information from the text instance features, accelerating the model's similarity learning process. The final image sequence features generated by the IDM model are:

[0111]

[0112] Where Conv Ii BiLSTM represents the i-th convolutional module in the sequence feature module. I This is a bidirectional long short-term memory network module. It generates image sequence features. The size is 128×15. Bidirectional Long Short-Term Memory (LSTM) networks are commonly used sequence information extraction models that can effectively capture forward and backward information of sequences and generate sequence features containing contextual information.

[0113] (2.4) The text data is fed into the TEM model to directly extract the text sequence features. The features are generated as follows:

[0114]

[0115] Where Conv TiBiLSTM represents the i-th convolutional module in the sequence feature module. T It is a bidirectional long short-term memory network module; W T and b T T represents the trainable weights and biases; i This represents the one-hot vector of the text corresponding to the i-th instance. The text sequence features generated by the model. The size is also 128×15.

[0116] (2.5) Directly calculate image sequence features and text sequence features Feature similarity Substitute the distribution similarity metric loss function:

[0117]

[0118] Where N is the number of data points. For the i-th IDM feature, For the j-th TEM feature, For the text corresponding to the i-th IDM feature, For the text corresponding to the j-th TEM feature, W D y represents the trainable weights; ED(x, y) is the regularized edit distance function. Edit distance is a commonly used method to measure text distance. Assuming the obtained edit distance is ed, and the longest text length in the ground truth values ​​corresponding to image features and text features is ll, then the regularized edit distance is 1-ed / ll.

[0119] By constraining the similarity calculation results of IDM and TEM features from the perspective of distribution, the similarity distribution is aligned with the text editing distance score distribution. Compared with the Eulerian distance alignment used in traditional methods, this allows the model to understand feature similarity from the data distribution, and the learned features are more generalizable.

[0120] (2.6) The ASM model is trained using a text classification loss function. The inputs are the probability output of the ASM for each instance and the corresponding true value, where the text classification loss function L... cls for:

[0121]

[0122] Among them, Y i E represents whether the i-th instance is text or non-text. i Y represents the prediction result of the automatic detection and screening model for the i-th instance; M is the number of all instances detected by the IDM model. In this embodiment, if the truth value of the i-th instance is text, then Y... i The value of Y is 1; otherwise, Y iThe value is 0.

[0123] (2.7) Image sequence features In the input multi-granularity text prediction loss function, the predicted characters, length, and frequency of each character in the text corresponding to this feature are calculated.

[0124]

[0125] Among them W V and b V To predict trainable weights and biases in characters, V i W represents the truth value of the text character corresponding to the i-th instance, and is a one-hot vector; L and b L To predict trainable weights and biases within a text length, L i W represents the text length corresponding to the i-th instance; C and b C To predict the trainable weights and biases in the frequency of text characters, C i This represents the number of text characters corresponding to the i-th instance.

[0126] Unlike traditional methods that only predict characters, this application also predicts text length and the frequency of character occurrences. Calculating a loss function based on these predictions and applying it as a guide allows the features to contain richer sequence information, improving their expressive power from multiple perspectives, enhancing the model's similarity learning performance, and accelerating the model's training process.

[0127] (2.8) For the detection boxes output in the IDM model, the following detection loss function is used for guidance:

[0128]

[0129] lb i Represents prediction box b i The corresponding truth boxes; x∩y represents the area of ​​the intersection of boxes x and y, and x∪y represents the area of ​​the union of boxes x and y.

[0130] (2.9) The total loss function of the model is the sum of the distribution similarity metric loss function, the text classification loss function, the multi-granularity text prediction loss function, and the detection loss function:

[0131]

[0132] (2.10) This model targets the total loss function and requires two training phases: pre-training and fine-tuning. During pre-training, the model's learning rate is 0.01, and it is trained for 90,000 epochs on a synthetic dataset, with the learning rate decaying to 0.001 and 0.0001 at epochs 30,000 and 60,000, respectively. During fine-tuning, the model's learning rate is 0.001, and it is trained for 80,000 epochs on a real-world text dataset, with the learning rate decaying to 0.0001 at epoch 40,000.

[0133] (3) Calculate the similarity between the image sequence features generated by IDM and the text sequence features generated by TEM after ASM screening, and select the image instance corresponding to the image feature with the highest similarity as the result returned by retrieval.

[0134] During the inference phase, based on the given query text, the text and images from the image library are preprocessed and then input into the trained retrieval model. The model outputs image sequence features and text sequence features, and calculates the similarity between features according to the above process. In this embodiment, the query text is "Lanzhou Ramen". It is assumed that all images have undergone detection and feature extraction, and the detected instances are "Lanzhou Ramen", "Huizhou Banmian", "Chaoshan Shengyan", and some background. ASM can exclude all background and retain only the three text instances mentioned above. When finally calculating feature similarity, the similarity score between the features of the query text "Lanzhou Ramen" and the features of the image instance "Lanzhou Ramen" is 0.99, and the similarity scores with the features of "Huizhou Banmian" and "Chaoshan Shengyan" are 0.75 and 0.1, respectively. Therefore, the query text "Lanzhou Ramen" and the image instance "Lanzhou Ramen" have the highest matching degree.

[0135] (4) In the scene text retrieval system, the user inputs query text, and based on the above reasoning process, retrieves images containing the text. In this embodiment, the user inputs the text "Lanzhou Ramen", then the system retrieves images and image instances containing the text from all images, and marks "Lanzhou Ramen" in the image instances.

[0136] This application was tested on the commonly used datasets SVT and STR to verify its effectiveness. The evaluation metric used in this experiment is the average accuracy across all classes (mAP), which is a commonly used metric for evaluating the overall retrieval performance.

[0137] The comparative methods chosen for the experiments were IRTC, RTNN, TSAA, YOLO-STR, and ABCNet, all classic and commonly used methods for scene-based text retrieval. The experimental results on the test dataset are shown in Table 1.

[0138] Table 1

[0139]

[0140]

[0141] It can be seen that this application is significantly better than the above comparison methods in terms of evaluation indicators, which fully demonstrates the superiority of the automatic detection and screening mechanism and improved loss function proposed in this application.

[0142] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A scene text retrieval method based on automatic detection and screening, characterized in that, include: The system receives the input query text and inputs it into the scene text retrieval model, wherein the scene text retrieval model includes an image branching model, a text branching model, and an automatic detection and screening model. The query text is fed into a trained text branching model to obtain the text sequence features of the query text; Obtain the image sequence features of each image in the target image library; Calculate the similarity between the image sequence features of each image in the target image library and the text sequence features of the query text, and return the image containing the query text based on the similarity calculation results; The training process of the scene text retrieval model includes: Step S1: Preprocess and augment the image and text datasets; Step S2: The preprocessed and augmented scene image data is fed into the image branching model to detect text-like instances in each image, and then fed into the automatic detection and screening model to remove interfering non-text instances, thus obtaining the image sequence features of each image in the image dataset; Step S3: Feed each text in the text dataset into the text branching model to obtain text sequence features; Step S4: Constrain the similarity calculation results of image sequence features and text sequence features from a distributional perspective using a distributional similarity metric loss function, aligning the similarity distribution with the text edit distance score distribution. The distributional similarity metric loss function... L dis for: in: N The number of data points. For the first Image sequence features, For the first Text sequence features, For the first Text corresponding to the features of an image sequence For the first The text corresponding to each text sequence feature W D For trainable weights, ED (·) is the regularized edit distance function. for and The similarity calculation results are as follows: in: The Manhattan distance function; The image branching model includes: The text instance detection module is used to detect all text-like instances in an image and extract instance features; The sequence feature module is used to obtain image sequence features based on the instance features of text instances in the image; The basic network structure of the text instance detection module is as follows: in: This represents a residual network, consisting of 5 sub-network modules, all of which are convolutional networks. This represents a pyramid network, which consists of three convolutional network modules. This represents the input image. This represents the extracted image features, which are used to generate all text-like instance detection boxes: in: and These are trainable weights and biases; The collection of all generated text-based instance detection boxes. For the first The detection bounding boxes for each instance, based on the detection bounding boxes from the features Extract The feature set of an instance , For the first Characteristics of an instance; The image sequence features obtained by the sequence feature module are: in: The first in the sequence feature module Each convolutional module, It is a bidirectional long short-term memory network module. Instance features for text instances; The automatic detection and screening model filters out all non-text instances in the feature set of instances and deletes them. The structure of the automatic detection and screening model is as follows: in: , , , , and These are trainable weights and biases; This represents the probability that each instance output by the automatic detection and screening model is a text instance.

2. The scene text retrieval method based on automatic detection and screening according to claim 1, characterized in that, In step S2, the automatic detection and screening model uses a text classification loss function to distinguish non-text instances detected by the image branching model. L cls Specifically: in: Y i Indicates the first Is each instance text or non-text? This indicates that the automatic detection and screening model is for the first... Prediction results for each instance; This represents the number of all instances detected by the image branching model.

3. The scene text retrieval method based on automatic detection and screening according to claim 1, characterized in that, The image branching model uses a multi-granularity text prediction loss function to predict the characters, text length, and number of characters of the ground truth text corresponding to the image sequence features, and compares them with the ground truth to enhance sequence information. The multi-granularity text prediction loss function... L txt for: in: W V and b V To predict trainable weights and biases in characters, Indicates the first The text character truth values ​​corresponding to each instance are one-hot vectors. W L and b L To predict trainable weights and biases within a text length, Indicates the first The text length corresponding to each instance W C and b C To predict the trainable weights and biases in the number of times a text character is encountered, Indicates the first The number of text characters corresponding to each instance.

4. The scene text retrieval method based on automatic detection and screening according to claim 1, characterized in that, The loss function of the text instance detection module L det for: in: Represents the prediction box The corresponding truth box; Display box and The area of ​​the intersecting part Display box and The area of ​​the union of the sets.

5. The scene text retrieval method based on automatic detection and screening according to claim 1, characterized in that, The text sequence features obtained by the text branching model are: in: The first in the sequence feature module Each convolutional module, It is a bidirectional long short-term memory network module. and For trainable weights and biases, Indicates the first The text corresponding to each instance.

6. The scene text retrieval method based on automatic detection and screening according to claim 1, characterized in that, The preprocessing of the image dataset includes image color adjustment, image size scaling, image pixel normalization, and image tensor quantization, while the preprocessing of the text dataset includes one-hot vectorization.

7. A scene text retrieval device based on automatic detection and screening, comprising a memory, a processor, and a program stored in the memory, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-6.

8. A storage medium having a program stored thereon, characterized in that, When the program is executed, it implements the method as described in any one of claims 1-6.

Images