Method and system for text recognition search in pictures

Abstract

The present invention provides a text recognition search method and system for images, which relates to the field of image search technology, and includes preprocessing an image to be searched to obtain a preprocessed image; performing text recognition on the preprocessed image using a text recognition model to obtain a text recognition result; extracting a text-image feature vector of a corresponding position in the preprocessed image according to the position of the text, and determining a text confidence content vector based on the text content and in combination with the text confidence; constructing a text-image fusion feature based on the text-image feature vector and the text confidence content vector, and constructing an image search index; performing word segmentation and semantic analysis on the search text to obtain a query fusion vector; in the image search index, determining a matching result by calculating the similarity between the query fusion vector and the text-image fusion feature, and extracting an image to be searched corresponding to the preprocessed image as an image search result according to a mapping relationship between the text-image fusion feature and the preprocessed image.

Description

Technical Field

[0001] The present invention relates to image search technology, and in particular to a text recognition search method and system for images. Background Art

[0002] With the rapid development of digital image processing and Internet technology, the amount of image data has exploded. Traditional image search methods mainly search based on the visual content of images, such as low-level features such as color, texture, and shape. However, these features are difficult to accurately express the semantic information of images, resulting in a gap between search results and user needs. In order to improve the accuracy of image search, researchers began to explore the use of text information in images to assist search. However, traditional text recognition technology mainly targets scanned document images, and has poor text recognition effects on natural scene images. In recent years, the rise of deep learning technology has brought new breakthroughs in image text recognition. Algorithms based on convolutional neural networks and recurrent neural networks can effectively locate and extract text information from images with complex backgrounds.

[0003] Although deep learning text recognition algorithms have made progress, existing algorithms mainly focus on the text content itself and do not make full use of text location, size, font and other attribute information, which results in the inability to fully utilize the visual features of text when building image search indexes. In addition, most existing image search systems use simple keyword matching methods, which makes it difficult to handle long sentence search requests entered by users, affecting the flexibility and accuracy of the search.

[0004] In summary, a new text recognition search method for images is needed; it can simultaneously utilize the content information and visual features of the text; build an efficient image search index; and flexibly process natural language search requests input by users. The present invention can solve the problems in the prior art. Summary of the invention

[0005] The embodiments of the present invention provide a method and system for text recognition and search in images, which can solve the problems in the prior art.

[0006] According to a first aspect of the embodiments of the present invention,

[0007] A method for text recognition and search in images is provided, comprising:

[0008] Receive a picture to be searched, pre-process the picture to be searched to obtain a pre-processed picture, wherein the pre-processing includes image denoising, image enhancement and image segmentation; perform text recognition on the pre-processed picture using a text recognition model to obtain a text recognition result, wherein the text recognition result includes text content, text position and text confidence;

[0009] According to the text position, extract the text image feature vector of the corresponding position in the pre-processed image, and determine the text confidence content vector based on the text content and the text confidence; construct a text image fusion feature based on the text image feature vector and the text confidence content vector, determine the mapping relationship between the text content, the text image fusion feature and the pre-processed image, and construct an image search index;

[0010] Receive search text, perform word segmentation and semantic analysis on the search text, and obtain a query fusion vector; in the image search index, determine a matching result by calculating the similarity between the query fusion vector and the text-image fusion feature; based on the matching result and according to the mapping relationship between the text-image fusion feature and the pre-processed image, extract the image to be searched corresponding to the pre-processed image as the image search result.

[0011] In an optional embodiment,

[0012] Receiving a picture to be searched, and preprocessing the picture to be searched to obtain a preprocessed picture includes:

[0013] Adaptive wavelet threshold denoising is performed on the image to be searched, a probability model of wavelet coefficients is constructed, an optimal wavelet threshold is determined using Bayesian estimation, adaptive threshold processing is performed on the wavelet coefficients, and wavelet reconstruction is performed to obtain a denoised image;

[0014] Performing adaptive histogram equalization processing on the denoised image, determining a cumulative distribution function of a local area by calculating a grayscale histogram of the denoised image, and adaptively adjusting a grayscale mapping relationship of the local area based on the cumulative distribution function to obtain an enhanced image;

[0015] The enhanced image is segmented into multiple superpixels through a superpixel segmentation algorithm, the similarity matrix between superpixels is determined as the edge weight of the undirected graph, a superpixel graph cut undirected graph is constructed, and the global optimal solution of the superpixel graph cut undirected graph is solved by the maximum flow minimum cut algorithm in combination with the pre-set probability distribution of the text area, the segmentation result of the text area is obtained, and the preprocessed image is obtained.

[0016] In an optional embodiment,

[0017] Combined with the pre-set probability distribution of the text area, the global optimal solution of the superpixel graph cut undirected graph is solved by the maximum flow minimum cut algorithm to obtain the segmentation result of the text area, including:

[0018] Introducing source points and sink points into the superpixel graph cut undirected graph, assigning initial flow to each superpixel node in combination with a pre-set probability distribution of the text area, and connecting the superpixel nodes to the source points and sink points respectively to form an initial flow network;

[0019] Based on the initial traffic network, initialize the distance label of each node, dynamically adjust the push strategy according to the excess traffic of the node, select the outgoing edge that meets the conditions to push the excess traffic, and update the residual capacity of the edge and the excess traffic of the node; dynamically adjust the relabeling strategy according to the distance label of the node, and update the distance label of the node that cannot perform the push operation;

[0020] After reaching the maximum flow, based on the distance labeling and residual network, the nodes reachable from the source are marked by a depth-first search algorithm, and the set of unmarked nodes is taken as the minimum cut set;

[0021] According to the superpixel labels corresponding to the minimum cut set, a region growing algorithm is used to merge superpixels to obtain a segmentation result of the text area.

[0022] In an optional embodiment,

[0023] The text recognition model is used to perform text recognition on the pre-processed image, and the text recognition results obtained include:

[0024] Input the preprocessed image into the enhanced attention convolutional neural network, determine the convolution feature map through the convolution layer, multiply the spatial attention map generated by the spatial attention mechanism by the convolution feature map element by element, adaptively adjust the weight of each position of the convolution feature map, highlight the primary features of the text area, and then multiply the channel attention vector generated by the channel attention mechanism by the convolution feature map channel by channel, adaptively adjust the weight of each channel of the convolution feature map, extract key features from the primary features, and determine the enhanced feature map;

[0025] Input the enhanced feature map into a bidirectional attention gated recurrent unit, encode the enhanced feature map in both the forward and reverse time directions, calculate the encoder attention weight through the encoder hidden state of the current time step and the enhanced feature map, obtain the encoder attention context vector, and merge it with the encoder current input and the hidden state of the previous time step, update the encoder hidden state, and obtain a context-related feature sequence;

[0026] Input the context-related feature sequence into a unidirectional attention gated recurrent unit, calculate the decoder attention weight through the current decoder hidden state and the context-related feature sequence at each time step, obtain the decoder attention context vector, and fuse it with the current decoder hidden state to obtain an updated decoder hidden state, and generate a character probability distribution sequence through a fully connected layer and a softmax function;

[0027] According to the preset number of candidate sequences, an empty candidate sequence set is constructed. Based on each time step of the character probability distribution sequence, according to the score and length penalty factor of the current candidate sequence, the candidate character with the highest score is selected from the current character probability distribution according to the number of candidate sequences, and a new candidate sequence is generated and put into the candidate sequence set until the maximum time step is reached. From the final candidate sequence set, the candidate sequence with the highest score is selected as the text recognition result.

[0028] In an optional embodiment,

[0029] The score of the candidate sequence is calculated as follows:

[0030] ;

[0031] in, y represents the candidate sequence number, score( y ) represents the candidate sequence y The score, len( y ) represents the total length of the time steps of the candidate sequence, t represents the time step number, P ( y t | y <t , x ) means that given an input graph x and the character sequence generated previously y <t Under the condition of y t The probability of (len( y )+1) α represents the length penalty factor, α A hyperparameter representing the length penalty factor.

[0032] In an optional embodiment,

[0033] According to the text position, extract the text image feature vector of the corresponding position in the pre-processed image, determine the text confidence content vector based on the text content and the text confidence; construct a text image fusion feature based on the text image feature vector and the text confidence content vector, determine the mapping relationship between the text content, the text image fusion feature and the pre-processed image, and construct the image search index including:

[0034] According to the text position, the text image feature vector of the corresponding position in the preprocessed image is extracted, and based on the text content and in combination with the text confidence, the text confidence content vector is determined; based on the text image feature vector and the text confidence content vector, a text image fusion feature is constructed, the mapping relationship from the text content to the text image fusion feature is determined, and an image search index is constructed.

[0035] According to the text position in the text recognition result, the text position is mapped to the scale corresponding to the preprocessed image, and the bounding box coordinates of the corresponding text in the preprocessed image are obtained. According to the bounding box coordinates, the corresponding text image area is cropped from the preprocessed image, and the scale of the text image area is normalized to adjust it to a preset fixed size. A pre-trained convolutional neural network is used to extract features from the normalized text image area to obtain a text image feature vector;

[0036] Applying one-hot vector representation to the text content in the text recognition result to construct a text content vector, and performing weighted processing on the text content vector based on the text confidence in the text recognition result to obtain a text confidence content vector;

[0037] The text confidence content vector and the corresponding text image feature vector are concatenated to generate a fused feature vector, and the fused feature vector is L2 normalized to obtain a text image fusion feature;

[0038] The text contents in all the text recognition results are traversed to extract all unique text contents, and a unique number is assigned to each unique text content, and a dictionary is constructed with the text content as the index key and the number as the index value; the preprocessed images corresponding to the text contents are arranged in reverse order according to the correlation scores with the text-image fusion features, and an inverted list is constructed, and the corresponding number is obtained in the dictionary according to the text content, and a one-to-one mapping is established between the inverted list and the number to construct an image search index.

[0039] In an optional embodiment,

[0040] Receiving search text, performing word segmentation and semantic analysis on the search text to obtain a query fusion vector; in the image search index, determining a matching result by calculating the similarity between the query fusion vector and the text-image fusion feature, and based on the matching result, extracting a to-be-searched image corresponding to the pre-processed image as an image search result according to a mapping relationship between the text-image fusion feature and the pre-processed image, including:

[0041] Segment the search text to obtain a query word list, and based on each query word in the query word list, obtain a corresponding word vector through a word vector model to obtain a query word vector list;

[0042] Performing semantic analysis on the search text to obtain a query semantic vector;

[0043] Fusing the query word vector list with the query semantic vector to obtain the query fusion vector;

[0044] Calculating the cosine similarity between the query fusion vector and each text-image fusion feature in the image search index, sorting them according to the cosine similarity, and selecting candidate matching results according to a preset similarity threshold;

[0045] According to the mapping relationship between the candidate matching results and the pre-processed images, the corresponding pre-processed images are extracted, and the images to be searched corresponding to the pre-processed images are extracted to form image search results.

[0046] A text recognition search system for images is provided, comprising:

[0047] The first unit is used to receive a picture to be searched, preprocess the picture to be searched to obtain a preprocessed picture, wherein the preprocessing includes image denoising, image enhancement and image segmentation; perform text recognition on the preprocessed picture using a text recognition model to obtain a text recognition result, wherein the text recognition result includes text content, text position and text confidence;

[0048] The second unit is used to extract the text image feature vector of the corresponding position in the pre-processed image according to the text position, determine the text confidence content vector based on the text content and the text confidence; construct a text image fusion feature based on the text image feature vector and the text confidence content vector, determine the mapping relationship between the text content, the text image fusion feature and the pre-processed image, and construct an image search index;

[0049] The third unit is used to receive search text, perform word segmentation and semantic analysis on the search text, and obtain a query fusion vector; in the image search index, determine the matching result by calculating the similarity between the query fusion vector and the text-image fusion feature, and based on the matching result and according to the mapping relationship between the text-image fusion feature and the pre-processed image, extract the image to be searched corresponding to the pre-processed image as the image search result.

[0050] According to a third aspect of the embodiments of the present invention,

[0051] An electronic device is provided, comprising:

[0052] processor;

[0053] a memory for storing processor-executable instructions;

[0054] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0055] A fourth aspect of the embodiments of the present invention is:

[0056] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0057] In the present invention, by introducing source points and sink points, combined with a pre-set probability distribution, based on the maximum flow minimum cut algorithm, the global optimal text area segmentation result is obtained in the superpixel graph cut undirected graph, and the text area and non-text area in the image are accurately distinguished; the maximum flow minimum cut algorithm is used to ensure that the obtained segmentation scheme is globally optimal, so that the text area segmentation result will not be affected by local extreme points, thereby improving the robustness and reliability of the algorithm; a dynamic adjustment method is adopted in the push strategy and the relabeling strategy, and the algorithm parameters are dynamically adjusted according to the excess flow and distance label of the node, so that the algorithm has adaptability and robustness, and can be obtained in different scenarios. Good text area segmentation effect; by enhancing the attention convolutional neural network, combined with spatial attention and channel attention mechanisms, it can adaptively adjust the weight of the convolution feature map, highlight the primary features of the text area, and improve the accuracy and robustness of feature extraction; the introduction of the length penalty factor can effectively penalize longer sequences, avoid excessive bias towards long sequences and ignore the generation quality, help improve the model's ability to generate short sequences, and balance the relationship between sequence length and quality; combine text image features and confidence content to form a multimodal fusion representation, which comprehensively utilizes the visual information and semantic information of the text and improves the accuracy and robustness of text recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A schematic diagram of a flow chart of a method for text recognition and search in an image according to an embodiment of the present invention;

[0059] Figure 2 The figure is a schematic diagram of the structure of a text recognition and search system for images according to an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0062] Figure 1 FIG. 1 is a flow chart of a method for text recognition and search in an image according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0063] S101. Receive a picture to be searched, pre-process the picture to be searched, and obtain a pre-processed picture, wherein the pre-processing includes image denoising, image enhancement, and image segmentation; perform text recognition on the pre-processed picture using a text recognition model to obtain a text recognition result, wherein the text recognition result includes text content, text position, and text confidence;

[0064] The text confidence refers to the confidence or credibility measure of the text recognition system for each recognized text result, which indicates the system's degree of certainty about the recognition result, that is, the system's confidence in the result. Generally, the confidence is a value between 0 and 1, where 0 means complete uncertainty and 1 means complete certainty.

[0065] In order to remove the noise in the image and improve the accuracy of subsequent text recognition, the search image is subjected to image denoising. In order to enhance the contrast of the image and improve the image clarity to improve the effect of text recognition, the denoised image is subjected to image enhancement. The image is divided into different areas to better identify and locate the text area, and the enhanced image is subjected to image segmentation.

[0066] Use the text recognition model to perform text recognition on the preprocessed image, identify the text content in the image, and obtain the location information and confidence of the text;

[0067] In this embodiment, by removing noise from the image, the image is made clearer, interference is reduced, and the accuracy of subsequent processing steps is improved; by enhancing the contrast, clarity and details of the image, the text is made more clearly visible and the performance of the text recognition model is improved; by segmenting the image into different areas or objects, it helps to locate and identify the text area, reduce background interference, and improve the accuracy of text recognition; the model can accurately identify the text content in the image and determine the location information of the text in the image, helping the user understand the system's credibility for the recognition results, so as to perform subsequent processing or decision-making.

[0068] In an optional embodiment, receiving a picture to be searched, and preprocessing the picture to be searched to obtain a preprocessed picture includes:

[0069] Adaptive wavelet threshold denoising is performed on the image to be searched, a probability model of wavelet coefficients is constructed, an optimal wavelet threshold is determined using Bayesian estimation, adaptive threshold processing is performed on the wavelet coefficients, and wavelet reconstruction is performed to obtain a denoised image;

[0070] Performing adaptive histogram equalization processing on the denoised image, determining a cumulative distribution function of a local area by calculating a grayscale histogram of the denoised image, and adaptively adjusting a grayscale mapping relationship of the local area based on the cumulative distribution function to obtain an enhanced image;

[0071] The enhanced image is segmented into multiple superpixels through a superpixel segmentation algorithm, the similarity matrix between superpixels is determined as the edge weight of the undirected graph, a superpixel graph cut undirected graph is constructed, and the global optimal solution of the superpixel graph cut undirected graph is solved by the maximum flow minimum cut algorithm in combination with the pre-set probability distribution of the text area, the segmentation result of the text area is obtained, and the preprocessed image is obtained.

[0072] The superpixel segmentation algorithm specifically refers to an image segmentation technology that divides an image into regions with similar features. The regions are regions with certain connectivity and closeness. Each region is regarded as a superpixel. Superpixels can reduce the amount of data in the image and are more in line with the human visual system's cognition of the image, which helps to improve the efficiency and accuracy of subsequent processing tasks.

[0073] The superpixel graph cut undirected graph specifically refers to a graph constructed on the basis of superpixel segmentation, in which superpixels are used as nodes of the graph, and the similarity between superpixels is represented by the weight of the edge. The superpixel graph cut undirected graph is used to represent the local structure and features in the image, so as to perform subsequent image segmentation and analysis tasks;

[0074] First, the search image is subjected to wavelet decomposition. By selecting appropriate wavelet basis functions, such as Haar wavelet, Daubechies wavelet, etc., the image is decomposed at multiple scales to obtain low-frequency approximate coefficients representing image overview information and high-frequency detail coefficients representing image detail texture information. Then, a probability model of wavelet coefficients is constructed. It is assumed that the wavelet coefficients obey the generalized Gaussian distribution (GGD). The statistical characteristics of the wavelet coefficients are characterized by estimating the scale parameters and shape parameters of the GGD. Then, the Bayesian estimation method is used to use the empirical wavelet coefficients obtained from the wavelet decomposition as observation data. The parameters of GGD are estimated by using the Bayesian formula; the variance of the wavelet coefficients and the noise variance are combined; the optimal wavelet threshold is determined according to the Bayesian criterion; the difference between the denoised wavelet coefficients and the original signal is minimized; the wavelet coefficients are soft-thresholded; the absolute value of the wavelet coefficients is subtracted from the threshold; the result is compared with zero to obtain the maximum value; the continuous contraction of the coefficients is achieved; finally, the processed wavelet coefficients are reconstructed by wavelet; the low-frequency approximation coefficients and the high-frequency detail coefficients are recombined by inverse wavelet transform; the denoised image is obtained; the main structure and details of the original image are retained; and the influence of noise is suppressed at the same time;

[0075] Calculate the grayscale histogram of the denoised image; count the frequency of each grayscale level in the image. Divide the image into multiple local areas; calculate the cumulative distribution function for each local area; reflect the cumulative probability distribution of the grayscale level in the area, and according to the cumulative distribution function; determine the grayscale mapping relationship of the local area; map the original grayscale value to the new grayscale range; expand the grayscale dynamic range of the local area; enhance the contrast of the image. Adaptively adjust the grayscale value of the local area; make the grayscale distribution in the local area more balanced; highlight the details of the image, and finally; merge the images processed in all local areas to obtain an overall enhanced image; improve the visual quality of the image;

[0076] Use the SLIC (Simple Linear Iterative Clustering) superpixel segmentation algorithm to segment the enhanced image into multiple superpixels. A superpixel is a set of pixels in an image that have similar color, texture and other features. By dividing an image into multiple locally coherent regions, the computational complexity of subsequent processing can be reduced, and the color, texture, position and other features of each superpixel can be extracted. The similarity matrix between superpixels is calculated to measure the similarity between superpixels, and superpixels are used as nodes for graph cut optimization. The similarity matrix is ​​used as the edge weight between nodes. A superpixel graph cut undirected graph model is constructed based on the prior probability distribution of text areas. The energy terms of the source point (representing the text area) and the sink point (representing the non-text area) are set to guide the graph cut optimization process using the maximum flow minimum cut algorithm. The global optimal solution of the superpixel graph cut undirected graph is solved. The optimal segmentation result of dividing the image into text areas and non-text areas is obtained, and the superpixels corresponding to the minimum cut are marked. Superpixels belonging to the same category are merged to obtain a preliminary text area segmentation result, and the segmentation result is subjected to morphological processing, such as corrosion operation to remove small noise areas and dilation operation to fill holes in the text area. The final text area segmentation result is obtained as a preprocessed image output.

[0077] In this embodiment, the optimal wavelet threshold is determined by Bayesian estimation, so that the noise in the image can be filtered out more accurately and useful information can be retained; the adaptive threshold processing can dynamically adjust the threshold according to the characteristics of the image and effectively process noise of different intensities; the cumulative distribution function of the local area can better adjust the contrast of the image and highlight the details of the image; the grayscale mapping relationship is adaptively adjusted to better retain the detail information of the image and improve the visual quality of the image; the SLIC superpixel segmentation algorithm can be used to effectively segment the image into regions with similar features, reducing the computational complexity of subsequent processing; the superpixel segmentation can extract the color, texture and other features of each region, which is helpful for subsequent text region segmentation and recognition tasks; the maximum flow minimum cut algorithm can solve the global optimal solution in the superpixel graph cut undirected graph to obtain accurate text region segmentation results.

[0078] In an optional embodiment, in combination with a pre-set probability distribution of the text area, the global optimal solution of the superpixel graph cut undirected graph is solved by the maximum flow minimum cut algorithm to obtain the segmentation result of the text area, including:

[0079] Introducing source points and sink points into the superpixel graph cut undirected graph, assigning initial flow to each superpixel node in combination with a pre-set probability distribution of the text area, and connecting the superpixel nodes to the source points and sink points respectively to form an initial flow network;

[0080] Based on the initial traffic network, initialize the distance label of each node, dynamically adjust the push strategy according to the excess traffic of the node, select the outgoing edge that meets the conditions to push the excess traffic, and update the residual capacity of the edge and the excess traffic of the node; dynamically adjust the relabeling strategy according to the distance label of the node, and update the distance label of the node that cannot perform the push operation;

[0081] After reaching the maximum flow, based on the distance labeling and residual network, the nodes reachable from the source are marked by a depth-first search algorithm, and the set of unmarked nodes is taken as the minimum cut set;

[0082] According to the superpixel labels corresponding to the minimum cut set, a region growing algorithm is used to merge superpixels to obtain a segmentation result of the text area.

[0083] The source point represents the starting point of the flow network in graph theory, which is usually a node flowing into the network. In the present invention, the source point specifically refers to a text area, which is the input point of information; the sink point represents the end point of the flow network in graph theory, which is usually a node flowing out of the network. In the present invention, the sink point specifically refers to a non-text area, which is the output point of information;

[0084] The distance label specifically refers to the distance between nodes on the shortest path in the maximum flow minimum cut algorithm, and is usually used to assist in searching for augmented paths in residual networks;

[0085] The excess flow specifically refers to the flow generated when the flow received by a node in the flow network exceeds the maximum flow it can accommodate. In the maximum flow minimum cut algorithm, the excess flow is used to indicate whether there is an augmenting path in the flow network;

[0086] The outgoing edge refers to an edge flowing out from a node. In the present invention, the outgoing edge specifically refers to an edge connecting a superpixel node and other nodes;

[0087] The residual capacity refers to the remaining available flow capacity on each edge in the flow network. In the maximum flow minimum cut algorithm, the residual capacity is used to indicate the additional flow that can be added to a certain edge in the residual network; the residual network specifically refers to the flow path that has not been used in the original network;

[0088] The push strategy is used to dynamically adjust the push mode of traffic according to the excess traffic of the node in the maximum flow minimum cut algorithm, usually including selecting the outgoing edge that meets the conditions to push the excess traffic, and updating the residual capacity of the edge and the excess traffic of the node; the relabeling strategy is used to dynamically adjust the label of the node according to the distance label of the node in the maximum flow minimum cut algorithm, usually including updating the distance label of the node that cannot perform the push operation, so as to find an augmented path in the residual network;

[0089] On the basis of the constructed superpixel graph cut undirected graph, two special nodes are introduced: source and sink, where the source represents the text area and the sink represents the non-text area. According to the prior probability distribution of the text area, the initial probability flow is allocated to each superpixel node, and each superpixel node is connected to the source and sink respectively. The weight of the connection edge is set according to the probability that the superpixel belongs to the text area or the non-text area, forming an initial flow network.

[0090] Based on the initial traffic network; set the distance label for each node according to the distance from the node to the sink. Initially, the distance label of the source point is the number of nodes in the graph cut undirected graph; the distance label of the sink is 0; the distance label of other nodes is 1; according to the excess flow of each node, that is, the difference between the current inflow and outflow of the node; dynamically adjust the push strategy, for nodes with excess flow greater than 0; select the outgoing edges that meet the following conditions for push operations: first, the residual capacity of the edge is greater than 0; second, the distance label of the end node of the edge is 1 less than the distance label of the current node; push the excess flow to the outgoing edge that meets the conditions; update the residual capacity of the edge and the excess flow of the end node.

[0091] For nodes that cannot perform push operations, that is, there are no outgoing edges that meet the push conditions, it is necessary to dynamically adjust their distance labels, find out the nodes connected to the edges connected to the current node with residual capacity greater than 0, select the value with the smallest distance label among the nodes, add 1 as the new distance label of the current node, if the distance label of the current node is greater than the number of nodes in the graph cut undirected graph, it means that the node cannot reach the sink, set its distance label to infinity, adjust the distance label of the node that cannot push excess traffic through the relabeling strategy, so that it has the opportunity to push excess traffic to the sink in subsequent push operations;

[0092] Repeat the push strategy and labeling strategy until the excess flow of all nodes is 0. At this time, the maximum flow is reached. Based on the final distance label and residual network, through the depth-first search algorithm, starting from the source point, all reachable nodes are marked, and the set of unmarked nodes constitutes the minimum cut set. It represents the optimal partitioning scheme for dividing the graph cut undirected graph into text areas and non-text areas.

[0093] According to the superpixel labeling corresponding to the minimum cut set; merge superpixels belonging to the same category and use the region growing algorithm; use each unmerged superpixel as a seed; expand to the surrounding area; merge adjacent superpixels with the same label into a larger area; until all superpixels are merged; finally obtain the segmentation results of text area and non-text area, the text area is formed by the superpixels belonging to the source point; the non-text area is formed by the superpixels belonging to the sink point.

[0094] In this embodiment, by introducing source points and sink points, combined with a pre-set probability distribution, and based on the maximum flow minimum cut algorithm, the globally optimal text area segmentation result is obtained in the superpixel graph cut undirected graph, and the text area and non-text area in the image are accurately distinguished; using the region growing algorithm, according to the superpixel label corresponding to the minimum cut set, the superpixels belonging to the same category are merged, which effectively eliminates the over-segmentation phenomenon caused by superpixel segmentation and improves the accuracy of text area segmentation; through the maximum flow minimum cut algorithm, it is ensured that the obtained segmentation scheme is globally optimal, so that the text area segmentation result will not be affected by local extreme points, thereby improving the robustness and reliability of the algorithm; in the push strategy and the relabeling strategy, a dynamic adjustment method is adopted, and the algorithm parameters are dynamically adjusted according to the excess flow and distance label of the node, so that the algorithm has adaptability and robustness, and can achieve good text area segmentation effects in different scenarios; superpixel segmentation and maximum flow minimum cut algorithms are adopted to transform complex image segmentation problems into efficient graph theory problem solving, thereby improving the computational efficiency and processing speed of the algorithm.

[0095] In an optional embodiment, the pre-processed image is subjected to text recognition using a text recognition model, and the text recognition result obtained includes:

[0096] Input the preprocessed image into the enhanced attention convolutional neural network, determine the convolution feature map through the convolution layer, multiply the spatial attention map generated by the spatial attention mechanism by the convolution feature map element by element, adaptively adjust the weight of each position of the convolution feature map, highlight the primary features of the text area, and then multiply the channel attention vector generated by the channel attention mechanism by the convolution feature map channel by channel, adaptively adjust the weight of each channel of the convolution feature map, extract key features from the primary features, and determine the enhanced feature map;

[0097] Input the enhanced feature map into a bidirectional attention gated recurrent unit, encode the enhanced feature map in both the forward and reverse time directions, calculate the encoder attention weight through the encoder hidden state of the current time step and the enhanced feature map, obtain the encoder attention context vector, and merge it with the encoder current input and the hidden state of the previous time step, update the encoder hidden state, and obtain a context-related feature sequence;

[0098] Input the context-related feature sequence into a unidirectional attention gated recurrent unit, calculate the decoder attention weight through the current decoder hidden state and the context-related feature sequence at each time step, obtain the decoder attention context vector, and fuse it with the current decoder hidden state to obtain an updated decoder hidden state, and generate a character probability distribution sequence through a fully connected layer and a softmax function;

[0099] According to the preset number of candidate sequences, an empty candidate sequence set is constructed. Based on each time step of the character probability distribution sequence, according to the score and length penalty factor of the current candidate sequence, the candidate character with the highest score is selected from the current character probability distribution according to the number of candidate sequences, and a new candidate sequence is generated and put into the candidate sequence set until the maximum time step is reached. From the final candidate sequence set, the candidate sequence with the highest score is selected as the text recognition result.

[0100] The length penalty factor specifically refers to balancing the score of the candidate sequence while considering the length of the candidate sequence. The length of the candidate sequence may affect the score. A candidate sequence that is too long or too short may be detrimental to recognition accuracy. The length penalty factor adjusts the score of the candidate sequence so that a longer candidate sequence is not overly penalized due to its length, while ensuring that a shorter candidate sequence can also have a chance to be considered, thereby balancing the relationship between the length of the candidate sequence and the score, and more accurately selecting the final text recognition result;

[0101] Input the preprocessed image into the enhanced attention convolutional neural network for feature extraction, perform convolution operation on the input image through the convolution layer; obtain a preliminary convolution feature map; generate a spatial attention map using the spatial attention mechanism; learn the importance of different positions of the image; assign weights to each position of the convolution feature map, and multiply the spatial attention map with the convolution feature map element by element; adaptively adjust the weights of each position of the convolution feature map; highlight the primary features of the text area; generate a channel attention vector using the channel attention mechanism; learn the importance of different feature channels; assign weights to each channel of the convolution feature map; multiply the channel attention vector with the convolution feature map channel by channel; adaptively adjust the weights of each channel of the convolution feature map; extract key features from the primary features; obtain the final enhanced feature map;

[0102] Input the enhanced feature map into the bidirectional attention gated recurrent unit for encoding, in the forward time direction; according to the encoder hidden state and enhanced feature map of the current time step; calculate the encoder attention weight through the attention mechanism; generate the encoder attention context vector, fuse the context vector with the current encoder input and the hidden state of the previous time step; update the encoder hidden state of the current time step; in the reverse time direction; also use the attention mechanism to calculate the encoder attention weight and context vector; and update the encoder hidden state; concatenate the forward and reverse encoder hidden states; obtain a feature sequence that fuses the bidirectional context information; called a context-associated feature sequence;

[0103] Input the context-related feature sequence into the unidirectional attention gated recurrent unit for decoding, at each time step; according to the current decoder hidden state and the context-related feature sequence; calculate the decoder attention weight through the attention mechanism; generate the decoder attention context vector, fuse the context vector with the current decoder hidden state; update the decoder hidden state; pass the updated decoder hidden state through the fully connected layer and the softmax function; generate the character probability distribution of the current time step, and repeat the decoding process; until the maximum decoding time step is reached; obtain the complete character probability distribution sequence;

[0104] According to the preset number of candidate sequences, that is, the beam width of the beam search algorithm; construct an empty candidate sequence set, for each time step of the character probability distribution sequence; traverse each candidate sequence in the current candidate sequence set: according to the score and length penalty factor of the current candidate sequence; select the candidate character with the highest score from the character probability distribution of the current time step; generate a new candidate sequence; put the newly generated candidate sequence into the candidate sequence set; if the size of the candidate sequence set exceeds the preset number of candidate sequences; retain the first several candidate sequences with the highest scores; eliminate the remaining candidate sequences; repeat the traversal; until the maximum time step is reached; from the final candidate sequence set; select the candidate sequence with the highest score as the result output of text recognition.

[0105] In this embodiment, by enhancing the attention convolutional neural network and combining the spatial attention and channel attention mechanisms, the weights of the convolutional feature map can be adaptively adjusted to highlight the primary features of the text area, thereby improving the accuracy and robustness of feature extraction; the feature sequence is encoded using a bidirectional attention gated recurrent unit, which can effectively integrate the contextual information in the positive and negative time directions, thereby improving the understanding and modeling capabilities of the text sequence context; the context-associated feature sequence is decoded by a unidirectional attention gated recurrent unit to generate a character probability distribution sequence, which can accurately capture the probability distribution of each character in the text sequence and improve the recognition accuracy of the text content; the beam search algorithm is used to generate a candidate sequence from the character probability distribution sequence, which can dynamically select the candidate sequence with the highest score according to the preset beam width, thereby improving the accuracy and efficiency of text recognition.

[0106] In an optional embodiment, the score of the candidate sequence is calculated as follows:

[0107] ;

[0108] in, y represents the candidate sequence number, score( y ) represents the candidate sequence y The score, len( y ) represents the total length of the time steps of the candidate sequence,t represents the time step number, P ( y t | y <t , x ) means that given an input graph x and the character sequence generated previously y <t Under the condition of y t The probability of (len( y )+1) α represents the length penalty factor, α A hyperparameter representing the length penalty factor.

[0109] The score of the candidate sequence is calculated by a formula consisting of two parts. First, the sum of the logarithmic probabilities is calculated for each character in the sequence, taking into account the probability of generating the current character given an input image and a previously generated character sequence. Then, all the logarithmic probabilities are summed, and the logarithm of the base is used to ensure that the score is within an appropriate range. Secondly, a length penalty is applied to the sum of the logarithmic probabilities. The purpose of the penalty is to penalize longer sequences by dividing the sum by a length-related penalty factor. The formula for calculating the score of the candidate sequence calculates the score of the candidate sequence by taking into account the probability of character generation and the length of the sequence.

[0110] According to the formula, the formula evaluates the candidate sequence by considering the generation probability of each character in the candidate sequence and the length of the sequence, so that the model can score according to the quality of the generated character sequence, thereby better distinguishing high-quality sequences from low-quality sequences; the introduction of the length penalty factor can effectively penalize longer sequences, avoid excessive bias towards long sequences and ignore the generation quality, help improve the model's ability to generate short sequences, and balance the relationship between the length and quality of the sequence; through this formula, the system is helped to automatically select the best candidate sequence, thereby improving the efficiency and accuracy of the text recognition system.

[0111] S102. According to the text position, extract the text image feature vector of the corresponding position in the pre-processed image, determine the text confidence content vector based on the text content and the text confidence; construct a text image fusion feature based on the text image feature vector and the text confidence content vector, determine the mapping relationship between the text content, the text image fusion feature and the pre-processed image, and construct an image search index;

[0112] The text image feature vector specifically refers to extracting features from the text image to obtain a high-dimensional vector representing the text image. The high-dimensional vector captures the visual features of the text image and can be used for subsequent similarity comparison and retrieval tasks;

[0113] The text confidence content vector specifically refers to combining text content with text confidence, and is used to measure the confidence of each text and the corresponding content. The text confidence content vector can be regarded as a weighted representation.

[0114] According to the position of the text, the text image at the corresponding position is extracted from the preprocessed image; the feature extraction of the extracted text image is performed to obtain the text image feature vector, and the confidence content vector of each text is calculated by combining the text content and the confidence content vector, and the text image feature vector and the confidence content vector are fused to construct the text image fusion feature, and the image search index is constructed based on the mapping relationship between the text content, the text image fusion feature and the preprocessed image;

[0115] In this embodiment, by extracting the feature vector of the text image, the text is effectively represented visually, which is beneficial to the subsequent similarity comparison and retrieval; combined with the text confidence, the importance of each text can be evaluated more accurately, improving the judgment and understanding of the text content; the text image features and confidence content are combined to form a multimodal fusion representation, which comprehensively utilizes the visual information and semantic information of the text and improves the accuracy and robustness of text recognition; by constructing an image search index, the text content and its corresponding image can be quickly retrieved and matched, which provides convenience for image retrieval and related applications.

[0116] In an optional embodiment, according to the text position, extracting the text image feature vector of the corresponding position in the pre-processed image, based on the text content and in combination with the text confidence, determining the text confidence content vector; based on the text image feature vector and the text confidence content vector, constructing a text image fusion feature, determining the mapping relationship between the text content, the text image fusion feature and the pre-processed image, constructing the image search index includes:

[0117] According to the text position, extract the text image feature vector of the corresponding position in the preprocessed image, and determine the text confidence content vector based on the text content and the text confidence; construct a text image fusion feature based on the text image feature vector and the text confidence content vector, determine the mapping relationship from the text content to the text image fusion feature, and construct an image search index

[0118] According to the text position in the text recognition result, the text position is mapped to the scale corresponding to the preprocessed image, and the bounding box coordinates of the corresponding text in the preprocessed image are obtained. According to the bounding box coordinates, the corresponding text image area is cropped from the preprocessed image, and the scale of the text image area is normalized to adjust it to a preset fixed size. A pre-trained convolutional neural network is used to extract features from the normalized text image area to obtain a text image feature vector;

[0119] Applying one-hot vector representation to the text content in the text recognition result to construct a text content vector, and performing weighted processing on the text content vector based on the text confidence in the text recognition result to obtain a text confidence content vector;

[0120] The text confidence content vector and the corresponding text image feature vector are concatenated to generate a fused feature vector, and the fused feature vector is L2 normalized to obtain a text image fusion feature;

[0121] The text contents in all the text recognition results are traversed to extract all unique text contents, and a unique number is assigned to each unique text content, and a dictionary is constructed with the text content as the index key and the number as the index value; the preprocessed images corresponding to the text contents are arranged in reverse order according to the correlation scores with the text-image fusion features, and an inverted list is constructed, and the corresponding number is obtained in the dictionary according to the text content, and a one-to-one mapping is established between the inverted list and the number to construct an image search index.

[0122] The one-hot vector representation specifically refers to a method of converting text information into a vector representation, in which each word is represented as a vector, the dimension of the vector is the same as the number of words in the dictionary, only one element is 1, and the other elements are 0; specifically, the length of the one-hot vector is equal to the number of words in the dictionary, each word has a unique index in the dictionary, and the corresponding one-hot vector is 1 at the corresponding index position, and 0 at other positions;

[0123] Assume that there are five words in the dictionary, namely ["apple"; "banana"; "grape"; "mango"; "watermelon"], then each word can be represented by a 5-dimensional one-hot vector:

[0124] "apple": [1; 0; 0; 0; 0], "banana": [0; 1; 0; 0; 0], "grape": [0; 0; 1; 0;0], "mango": [0; 0; 0; 1; 0], "watermelon": [0; 0; 0; 0; 1];

[0125] According to the text position information in the text recognition result; map the text position to the scale corresponding to the preprocessed image; obtain the bounding box coordinates of each character in the preprocessed image; for the bounding box coordinates of each character; crop the corresponding text image area from the preprocessed image; normalize the scale of the cropped text image area; adjust it to a preset fixed size, such as 64×64; use the pre-trained convolutional neural network to extract features from the normalized text image area; obtain a fixed-length text image feature vector;

[0126] For each text content in the text recognition result, one-hot encoding is used to vectorize it and construct a text content vector. According to the confidence of each character in the text recognition result, the corresponding text content vector is weighted and the text content vector is multiplied by the corresponding confidence to obtain the text confidence content vector, which reflects the reliability of the recognition result.

[0127] The text image feature vector and the text confidence content vector are concatenated; a fused feature vector is generated, which contains the visual information of the text image and the semantic information of the text content; the fused feature vector is L2 normalized to make its length 1; the final text image fusion feature is obtained, wherein L2 normalization can eliminate the scale difference between feature vectors, which is convenient for subsequent similarity calculation;

[0128] Traverse all text contents in the text recognition results; extract all unique text contents; build a character set; assign a unique number to each unique text content in the character set; build a dictionary with text content as the index key; the number as the index value; establish a mapping relationship between text content and number; for each text content; according to its corresponding text image fusion feature; calculate the relevance score with the pre-processed image; preferably use a metric such as cosine similarity to calculate the similarity between the fusion feature and the image feature; arrange the pre-processed images in reverse order of the relevance score; build an inverted list with text content as the key; store the sorting results of the pre-processed images associated with it; map the text content to the corresponding number according to the dictionary; establish a one-to-one mapping relationship between the inverted list and the number; build an image search index; the image search index uses the number as the key; store the inverted list of the corresponding text content; and realize fast retrieval from text content to related images.

[0129] In this embodiment, a text-image fusion feature is constructed by extracting text-image feature vectors and combining them with text confidence content vectors, comprehensively considering the visual information and semantic information of the text to improve the accuracy of the search results; an image search index is constructed to effectively model the correlation between the text content and the preprocessed image, making the search results more relevant and accurate; an inverted list is constructed and a dictionary is used to establish a fast search mechanism from text content to related images, so that users can quickly obtain related images by entering text content, thereby improving search efficiency.

[0130] S103. Receive search text, perform word segmentation and semantic analysis on the search text, and obtain a query fusion vector; in the image search index, determine a matching result by calculating the similarity between the query fusion vector and the text-image fusion feature, and based on the matching result and according to the mapping relationship between the text-image fusion feature and the pre-processed image, extract the image to be searched corresponding to the pre-processed image as the image search result.

[0131] The system receives the search text input by the user, performs word segmentation and semantic analysis on the search text to understand the user's intention, and converts it into a query fusion vector. In the image search index, the system calculates the similarity between the query fusion vector and the text-image fusion feature. Based on the result of the similarity calculation, the system determines the matching result, that is, the image most relevant to the search request. Based on the matching result and the mapping relationship between the text-image fusion feature and the pre-processed image, the system extracts the image to be searched corresponding to the pre-processed image as the final image search result.

[0132] In this embodiment, word segmentation and semantic analysis ensure accurate understanding of user search requests, improve the relevance and accuracy of search results, and combine the fusion search method of text and image features to comprehensively utilize information of different modalities and improve the comprehensiveness and accuracy of search results; the extracted image search results are intuitively displayed, providing users with a convenient and intuitive search experience, enhancing the practicality of the system and user satisfaction.

[0133] In an optional embodiment, receiving a search text, performing word segmentation and semantic analysis on the search text, and obtaining a query fusion vector; in the image search index, determining a matching result by calculating a similarity between the query fusion vector and a text-image fusion feature; and based on the matching result and according to a mapping relationship between the text-image fusion feature and the pre-processed image, extracting a to-be-searched image corresponding to the pre-processed image as an image search result includes:

[0134] Segment the search text to obtain a query word list, and based on each query word in the query word list, obtain a corresponding word vector through a word vector model to obtain a query word vector list;

[0135] Performing semantic analysis on the search text to obtain a query semantic vector;

[0136] Fusing the query word vector list with the query semantic vector to obtain the query fusion vector;

[0137] Calculating the cosine similarity between the query fusion vector and each text-image fusion feature in the image search index, sorting them according to the cosine similarity, and selecting candidate matching results according to a preset similarity threshold;

[0138] According to the mapping relationship between the candidate matching results and the pre-processed images, the corresponding pre-processed images are extracted, and the images to be searched corresponding to the pre-processed images are extracted to form image search results.

[0139] Perform word segmentation on the search text entered by the user; use existing natural language processing tools to segment the search text into a series of independent words; obtain a query word list, and word segmentation can extract keyword information from the search text; prepare for subsequent vectorization representation;

[0140] For each query word in the query word list, obtain the corresponding word vector representation through the pre-trained word vector model, which maps each word into a real-valued vector of fixed dimension, and captures the semantic relationship between words; combine the word vectors of all query words in the query word list into a vector list, and obtain the query word vector list, which represents the semantic information of the search text at the word level;

[0141] Perform semantic analysis on the search text; use the pre-trained language model to encode the search text; obtain the query semantic vector, the language model can capture the contextual information and semantic relationship of the search text; generate a comprehensive semantic representation, the query semantic vector represents the information of the search text at the overall semantic level; supplement the word-level information in the query word vector list;

[0142] The query word vector list and the query semantic vector are merged to obtain the query fusion vector, which can be merged by weighted average, concatenation and other methods; the information at the word level and the semantic level are integrated, and the query fusion vector integrates the word information and semantic information of the search text; it can more comprehensively express the user's search intention;

[0143] Calculate the cosine similarity between the query fusion vector and each text-image fusion feature in the image search index. The cosine similarity measures the cosine value of the angle between two vectors. The larger the value, the higher the similarity between the vectors. Sort the text-image fusion features in the image search index according to the calculated cosine similarity. The results with higher similarity are ranked higher. Set a preset similarity threshold based on the search accuracy and recall rate. Select the text-image fusion features with cosine similarity greater than or equal to the threshold as candidate matching results.

[0144] According to the mapping relationship between the candidate matching results and the pre-processed images; find the pre-processed images corresponding to each candidate matching result; for each pre-processed image; extract its corresponding original image to be searched, there is a one-to-one corresponding relationship between the pre-processed image and the image to be searched; through this relationship, the original image data can be restored; the extracted images to be searched are combined according to the sorting order of the candidate matching results; to form the final image search results.

[0145] In this embodiment, word segmentation and semantic analysis ensure accurate understanding of user search requests. The word vectors and semantic vectors obtained through the word vector model and language model better capture the semantic information of the search text and improve the relevance and accuracy of the search results. The fusion search method combining word vectors and semantic vectors comprehensively utilizes the word information and semantic information of the text, thereby improving the comprehensiveness and accuracy of the search results.

[0146] Figure 2 FIG. 1 is a schematic diagram of a text recognition and search system for images according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0147] The first unit is used to receive a picture to be searched, preprocess the picture to be searched to obtain a preprocessed picture, wherein the preprocessing includes image denoising, image enhancement and image segmentation; perform text recognition on the preprocessed picture using a text recognition model to obtain a text recognition result, wherein the text recognition result includes text content, text position and text confidence;

[0148] The second unit is used to extract the text image feature vector of the corresponding position in the pre-processed image according to the text position, determine the text confidence content vector based on the text content and the text confidence; construct a text image fusion feature based on the text image feature vector and the text confidence content vector, determine the mapping relationship between the text content, the text image fusion feature and the pre-processed image, and construct an image search index;

[0149] The third unit is used to receive search text, perform word segmentation and semantic analysis on the search text, and obtain a query fusion vector; in the image search index, determine the matching result by calculating the similarity between the query fusion vector and the text-image fusion feature, and based on the matching result and according to the mapping relationship between the text-image fusion feature and the pre-processed image, extract the image to be searched corresponding to the pre-processed image as the image search result.

[0150] According to a third aspect of the embodiments of the present invention,

[0151] An electronic device is provided, comprising:

[0152] processor;

[0153] a memory for storing processor-executable instructions;

[0154] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0155] A fourth aspect of the embodiments of the present invention is:

[0156] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0157] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A text recognition search method for images, characterized in that: include: Receiving a picture to be searched, and preprocessing the picture to be searched to obtain a preprocessed picture, wherein the preprocessing includes image denoising, image enhancement, and image segmentation; Performing text recognition on the preprocessed image using a text recognition model to obtain a text recognition result, wherein the text recognition result includes text content, text position, and text confidence; According to the text position, extract the text image feature vector of the corresponding position in the pre-processed image, and determine the text confidence content vector based on the text content and the text confidence; construct a text image fusion feature based on the text image feature vector and the text confidence content vector, determine the mapping relationship between the text content, the text image fusion feature and the pre-processed image, and construct an image search index; Receive search text, perform word segmentation and semantic analysis on the search text, and obtain a query fusion vector; in the image search index, determine a matching result by calculating the similarity between the query fusion vector and the text-image fusion feature, and based on the matching result and according to the mapping relationship between the text-image fusion feature and the pre-processed image, extract a to-be-searched image corresponding to the pre-processed image as an image search result; Receiving a picture to be searched, and preprocessing the picture to be searched to obtain a preprocessed picture includes: Adaptive wavelet threshold denoising is performed on the image to be searched, a probability model of wavelet coefficients is constructed, an optimal wavelet threshold is determined using Bayesian estimation, adaptive threshold processing is performed on the wavelet coefficients, and wavelet reconstruction is performed to obtain a denoised image; Performing adaptive histogram equalization processing on the denoised image, determining a cumulative distribution function of a local area by calculating a grayscale histogram of the denoised image, and adaptively adjusting a grayscale mapping relationship of the local area based on the cumulative distribution function to obtain an enhanced image; The enhanced image is segmented into multiple superpixels through a superpixel segmentation algorithm, the similarity matrix between superpixels is determined as the edge weight of the undirected graph, a superpixel graph cut undirected graph is constructed, and the global optimal solution of the superpixel graph cut undirected graph is solved by the maximum flow minimum cut algorithm in combination with the pre-set probability distribution of the text area, the segmentation result of the text area is obtained, and the preprocessed image is obtained.

2. The method according to claim 1, characterized in that Combined with the pre-set probability distribution of the text area, the global optimal solution of the superpixel graph cut undirected graph is solved by the maximum flow minimum cut algorithm to obtain the segmentation result of the text area, including: Introducing source points and sink points into the superpixel graph cut undirected graph, assigning initial flow to each superpixel node in combination with a pre-set probability distribution of the text area, and connecting the superpixel nodes to the source points and sink points respectively to form an initial flow network; Based on the initial traffic network, initialize the distance label of each node, dynamically adjust the push strategy according to the excess traffic of the node, select the outgoing edge that meets the conditions to push the excess traffic, and update the residual capacity of the edge and the excess traffic of the node; dynamically adjust the relabeling strategy according to the distance label of the node, and update the distance label of the node that cannot perform the push operation; After reaching the maximum flow, based on the distance labeling and residual network, the nodes reachable from the source are marked by a depth-first search algorithm, and the set of unmarked nodes is taken as the minimum cut set; According to the superpixel labels corresponding to the minimum cut set, a region growing algorithm is used to merge superpixels to obtain a segmentation result of the text area.

3. The method according to claim 1, characterized in that The text recognition model is used to perform text recognition on the pre-processed image, and the text recognition results obtained include: Input the preprocessed image into the enhanced attention convolutional neural network, determine the convolution feature map through the convolution layer, multiply the spatial attention map generated by the spatial attention mechanism by the convolution feature map element by element, adaptively adjust the weight of each position of the convolution feature map, highlight the primary features of the text area, and then multiply the channel attention vector generated by the channel attention mechanism by the convolution feature map channel by channel, adaptively adjust the weight of each channel of the convolution feature map, extract key features from the primary features, and determine the enhanced feature map; Input the enhanced feature map into a bidirectional attention gated recurrent unit, encode the enhanced feature map in both the forward and reverse time directions, calculate the encoder attention weight through the encoder hidden state of the current time step and the enhanced feature map, obtain the encoder attention context vector, and merge it with the encoder current input and the hidden state of the previous time step, update the encoder hidden state, and obtain a context-related feature sequence; Input the context-related feature sequence into a unidirectional attention gated recurrent unit, calculate the decoder attention weight through the current decoder hidden state and the context-related feature sequence at each time step, obtain the decoder attention context vector, and fuse it with the current decoder hidden state to obtain an updated decoder hidden state, and generate a character probability distribution sequence through a fully connected layer and a softmax function; According to the preset number of candidate sequences, an empty candidate sequence set is constructed. Based on each time step of the character probability distribution sequence, according to the score and length penalty factor of the current candidate sequence, the candidate character with the highest score is selected from the current character probability distribution according to the number of candidate sequences, and a new candidate sequence is generated and put into the candidate sequence set until the maximum time step is reached. From the final candidate sequence set, the candidate sequence with the highest score is selected as the text recognition result.

4. The method according to claim 3, characterized in that The score of the candidate sequence is calculated as follows: ； in, y represents the candidate sequence number, score( y ) represents the candidate sequence y The score, len( y ) represents the total length of the time steps of the candidate sequence, t represents the time step number, P ( y t | y <t , x ) means that given an input graph x and the character sequence generated previously y <t Under the condition of y t The probability of (len( y )+1) α represents the length penalty factor, α A hyperparameter representing the length penalty factor.

5. The method according to claim 1, characterized in that According to the text position, extract the text image feature vector of the corresponding position in the pre-processed image, determine the text confidence content vector based on the text content and the text confidence; construct a text image fusion feature based on the text image feature vector and the text confidence content vector, determine the mapping relationship between the text content, the text image fusion feature and the pre-processed image, and construct the image search index including: According to the text position in the text recognition result, the text position is mapped to the scale corresponding to the preprocessed image, and the bounding box coordinates of the corresponding text in the preprocessed image are obtained. According to the bounding box coordinates, the corresponding text image area is cropped from the preprocessed image, and the scale of the text image area is normalized to adjust it to a preset fixed size. A pre-trained convolutional neural network is used to extract features from the normalized text image area to obtain a text image feature vector; Applying one-hot vector representation to the text content in the text recognition result to construct a text content vector, and performing weighted processing on the text content vector based on the text confidence in the text recognition result to obtain a text confidence content vector; The text confidence content vector and the corresponding text image feature vector are concatenated to generate a fused feature vector, and the fused feature vector is L2 normalized to obtain a text image fusion feature; The text contents in all the text recognition results are traversed to extract all unique text contents, and a unique number is assigned to each unique text content, and a dictionary is constructed with the text content as the index key and the number as the index value; the preprocessed images corresponding to the text contents are arranged in reverse order according to the correlation scores with the text-image fusion features, and an inverted list is constructed, and the corresponding number is obtained in the dictionary according to the text content, and a one-to-one mapping is established between the inverted list and the number to construct an image search index.

6. The method according to claim 1, characterized in that Receiving search text, performing word segmentation and semantic analysis on the search text to obtain a query fusion vector; in the image search index, determining a matching result by calculating the similarity between the query fusion vector and the text-image fusion feature, and based on the matching result, extracting a to-be-searched image corresponding to the pre-processed image as an image search result according to a mapping relationship between the text-image fusion feature and the pre-processed image, including: Segment the search text to obtain a query word list, and based on each query word in the query word list, obtain a corresponding word vector through a word vector model to obtain a query word vector list; Performing semantic analysis on the search text to obtain a query semantic vector; Fusing the query word vector list with the query semantic vector to obtain the query fusion vector; Calculating the cosine similarity between the query fusion vector and each text-image fusion feature in the image search index, sorting them according to the cosine similarity, and selecting candidate matching results according to a preset similarity threshold; According to the mapping relationship between the candidate matching result and the pre-processed image, the corresponding pre-processed image is extracted, and the image to be searched corresponding to the pre-processed image is extracted to form an image search result.

7. A text recognition and search system for images, used to implement the method according to any one of claims 1 to 6, characterized in that: include: The first unit is used to receive a picture to be searched, and preprocess the picture to be searched to obtain a preprocessed picture, wherein the preprocessing includes image denoising, image enhancement and image segmentation; Performing text recognition on the preprocessed image using a text recognition model to obtain a text recognition result, wherein the text recognition result includes text content, text position, and text confidence; The second unit is used to extract the text image feature vector of the corresponding position in the pre-processed image according to the text position, determine the text confidence content vector based on the text content and the text confidence; construct a text image fusion feature based on the text image feature vector and the text confidence content vector, determine the mapping relationship between the text content, the text image fusion feature and the pre-processed image, and construct an image search index; The third unit is used to receive search text, perform word segmentation and semantic analysis on the search text, and obtain a query fusion vector; in the image search index, determine the matching result by calculating the similarity between the query fusion vector and the text-image fusion feature, and based on the matching result and according to the mapping relationship between the text-image fusion feature and the pre-processed image, extract the image to be searched corresponding to the pre-processed image as the image search result.

8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images