Method and device for retrieving characters in picture, electronic equipment and storage medium

Abstract

The invention provides a method and device for retrieving characters in a picture, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence and the field of financial science and technology. The method comprises the following steps: acquiring a target query picture and a target query character; performing character detection on the target query picture through the target single character detection model to obtain a retrieved single character picture sequence which comprises at least two candidate retrieved single character pictures; searching a preset picture database according to the target query character to obtain a first query single character picture and a second query single character picture; screening the candidate retrieval individual character pictures according to the first query individual character picture to obtain a first retrieval individual character picture; screening candidate retrieval individual character pictures after the first retrieval individual character picture according to the second query individual character picture to obtain a second retrieval individual character picture; and according to the first retrieval character picture and the second retrieval character picture, determining target retrieval character information. The accuracy of searching the characters in the picture can be improved.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence technology and financial technology, and in particular to a method and device, electronic device, and storage medium for retrieving text from an image. Background Art

[0002] The method of searching for text in an image refers to searching for text or words in a query image based on the query text. For example, in the field of financial technology, a text search is performed on an insurance policy image based on the query text entered by the user (such as health insurance, pet insurance) to determine whether the text in the insurance policy image contains the query text.

[0003] The common practice of current image text retrieval technology is to use a text recognition model (such as an OCR recognition model) to perform text recognition on the entire image, and then match the recognized text according to the query text. There are usually two problems. The first problem is that if the text is recognized as an approximate character, the retrieval cannot be completed. The second problem is that if the character library of the text recognition model is limited, and the query text is not in the text recognition character library, the retrieval cannot be completed. For example, if the query text is a traditional or rare character in an ancient book, the retrieval may not be completed because the character is not in the character library.

[0004] It can be seen that the existing method for retrieving text in images has the technical problem of low retrieval accuracy. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to provide a method and device, electronic device, and storage medium for retrieving text in images, which can improve the accuracy of retrieving text in images.

[0006] To achieve the above purpose, a first aspect of an embodiment of the present application provides a method for retrieving text in a picture, the method comprising:

[0007] Obtain target query images and target query text;

[0008] Performing text detection on the target query image through a preset target single-word detection model to obtain a retrieval single-word image sequence; wherein the retrieval single-word image sequence includes at least two candidate retrieval single-word images;

[0009] Searching a preset picture database according to the target query text to obtain a query single-word picture sequence; wherein the query single-word picture sequence includes a first query single-word picture and a second query single-word picture;

[0010] Screening each of the candidate search word images according to the first query word image to obtain a first search word image;

[0011] The candidate search single-word pictures after the first search single-word picture are screened according to the second query single-word picture to obtain a second search single-word picture;

[0012] According to the first search single word image and the second search single word image, target search text information is determined.

[0013] In some embodiments, the step of screening the candidate retrieval word pictures after the first retrieval word picture according to the second query word picture to obtain the second retrieval word picture includes:

[0014] Extracting pictures from the retrieval word picture sequence according to the first retrieval word picture to obtain a candidate retrieval word picture; wherein the candidate retrieval word picture is a candidate retrieval word picture after the first retrieval word picture;

[0015] Performing typesetting detection on the first search single-word image and the to-be-selected search single-word image to obtain a typesetting detection result;

[0016] If the typesetting detection result indicates that the first search word picture and the to-be-selected search word picture are in the same row or the same column, then performing picture matching on the second query word picture and the to-be-selected search word picture to obtain a picture matching result;

[0017] If the image matching result indicates that the second query single-word image matches the to-be-selected retrieval single-word image, the to-be-selected retrieval single-word image is determined as the second retrieval single-word image.

[0018] In some embodiments, after performing typesetting detection on the first search single-word image and the to-be-selected search single-word image to obtain a typesetting detection result, the method further includes:

[0019] The first search word image is updated, including:

[0020] If the typesetting detection result indicates that the first search word picture and the candidate search word picture are not in the same row or the same column, the candidate search word pictures after the first search word picture are screened according to the first query word picture to obtain an updated first search word picture;

[0021] If the picture matching result indicates that the second query single-word picture does not match the candidate retrieval single-word picture, the candidate retrieval single-word pictures after the first retrieval single-word picture are screened according to the first query single-word picture to obtain an updated first retrieval single-word picture.

[0022] In some embodiments, the step of screening each of the candidate search word images according to the first query word image to obtain the first search word image includes:

[0023] Performing feature extraction on the first query single word image through a feature layer of a preset target single word classification model to obtain a first query single word vector;

[0024] Perform feature extraction on the candidate retrieval word image through the feature layer of the target word classification model to obtain a candidate retrieval word vector;

[0025] Perform vector matching based on the first query word vector and the candidate search word vector to obtain the vector matching degree corresponding to the candidate search word image;

[0026] Select each of the candidate retrieval single-word pictures according to the vector matching degree to obtain the first retrieval single-word picture.

[0027] In some embodiments, determining target search text information according to the first search word image and the second search word image includes:

[0028] Determining first search word position information according to the position of the first search word image in the target query image;

[0029] Determining second search word position information according to the position of the second search word image in the target query image;

[0030] Information fusion is performed based on the first search word position information and the second search word position information to obtain the target search text information.

[0031] In some embodiments, before extracting features from the first query word image through the feature layer of the preset target text classification model to obtain the first query word vector, the method further includes:

[0032] The target word detection model and the target word classification model are obtained by training, including:

[0033] Get sample query images;

[0034] Performing text detection on the sample query image through a preset initial single-word detection model to obtain a sample retrieval single-word image;

[0035] Performing text classification on the sample search word image through a preset initial word classification model to obtain a sample word category;

[0036] The loss is calculated according to the sample single word category and the preset label single word category to obtain target loss data; wherein the label single word font category is used to characterize the word to which the sample retrieval single word image belongs;

[0037] According to the target loss data, the parameters of the initial single word detection model are adjusted to obtain a target single word detection model, and the parameters of the initial single word classification model are adjusted to obtain a target single word classification model.

[0038] In some embodiments, the text classification of the sample search word image is performed using a preset initial word classification model to obtain a sample word category, including:

[0039] Performing feature extraction on the sample retrieval single word image through the feature layer of the initial single word classification model to obtain a sample retrieval single word feature vector;

[0040] Performing single word classification on the sample search single word feature vector through the classification layer of the initial single word classification model to obtain the sample single word category;

[0041] The target single-word classification model is obtained by adjusting parameters of the feature layer of the initial single-word classification model.

[0042] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a device for retrieving text in a picture, the device comprising:

[0043] A data acquisition module, used to acquire a target query image and a target query text;

[0044] A text detection module is used to perform text detection on the target query image through a preset target single-word detection model to obtain a search single-word image sequence; wherein the search single-word image sequence includes at least two candidate search single-word images;

[0045] An image search module is used to search a preset image database according to the target query text to obtain a query single-word image sequence; wherein the query single-word image sequence includes a first query single-word image and a second query single-word image;

[0046] A first picture screening module, used for screening each of the candidate search single-word pictures according to the first query single-word picture to obtain a first search single-word picture;

[0047] A second picture screening module, used for screening the candidate search single word pictures after the first search single word picture according to the second query single word picture, to obtain a second search single word picture;

[0048] The search information determination module is used to determine the target search text information according to the first search word image and the second search word image.

[0049] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the method of retrieving text in an image described in the first aspect when executing the computer program.

[0050] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium, and the storage medium stores a computer program. When the computer program is executed by a processor, the method for retrieving text in an image described in the first aspect above is implemented.

[0051] The method for retrieving text in an image, the device for retrieving text in an image, the electronic device and the computer-readable storage medium proposed in the present application do not need to recognize text from the target query image when performing text retrieval on the target query image according to the target query text. Instead, on the one hand, a retrieval single-word picture sequence is detected from the target query image through a target single-word detection model. On the other hand, a query single-word picture sequence is found from the image database according to the target query text. Then, the retrieval single-word picture sequence is screened according to the single-word pictures in the query single-word picture sequence to obtain a first retrieval single-word picture and a second retrieval single-word picture. In this way, the possibility of retrieval failure caused by errors in text recognition or limitations of the text recognition character library is reduced. In summary, the present application can improve the accuracy of retrieving text in an image.

[0052] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is a flow chart of a method for retrieving text in a picture provided by an embodiment of the present application;

[0054] Figure 2 yes Figure 1 Flowchart of step 104 in;

[0055] Figure 3 is a flow chart of a method for retrieving text in a picture provided by another embodiment of the present application;

[0056] Figure 4 yes Figure 3 Flow chart of step 303 in;

[0057] Figure 5 yes Figure 1 Flowchart of step 105 in;

[0058] Figure 6 It is a module structure block diagram of a device for retrieving text in a picture provided by an embodiment of the present application;

[0059] Figure 7 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0061] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0063] First, some nouns involved in this application are analyzed:

[0064] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0065] Natural language processing (NLP): NLP uses computers to process, understand and apply human languages ​​(such as Chinese, English, etc.). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text comprehension. Natural language processing is commonly used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information image processing, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0066] The embodiments of the present application propose a method for retrieving text in a picture, a device for retrieving text in a picture, an electronic device and a computer-readable storage medium, which can effectively solve the current text search problem in pictures, support all Chinese characters, and solve the retrieval problem of various types of pictures such as horizontal / vertical.

[0067] The method for retrieving text in an image provided by the embodiment of the present application can be applied to a terminal and a server, and can also be software running on the server. The server can be configured as an independent physical server, or as a server cluster or distributed system consisting of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the method for retrieving text in an image, etc., but is not limited to the above forms.

[0068] The present application can be used in many general or special computer system environments or configurations. For example: server computers, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0069] The embodiments of the present application provide a method for retrieving text in a picture and a device, electronic device, and storage medium for retrieving text in a picture, which are specifically explained through the following embodiments. First, the method for retrieving text in a picture in the embodiments of the present application is described.

[0070] It should be noted that in each specific implementation method of the present application, when it comes to processing data related to the user's identity or characteristics such as the user's behavioral data, the user's permission or consent will be obtained first, and the collection, use and processing of such data will comply with relevant laws, regulations and standards.

[0071] Reference Figure 1 , Figure 1 It is an optional flowchart of the method for retrieving text in an image provided in an embodiment of the present application, which may include but is not limited to steps 101 to 106.

[0072] Step 101, obtaining a target query image and a target query text;

[0073] Step 102, performing text detection on the target query image through a preset target single word detection model to obtain a retrieval single word image sequence; wherein the retrieval single word image sequence includes at least two candidate retrieval single word images;

[0074] Step 103, searching a preset image database according to the target query text to obtain a query single-word image sequence; wherein the query single-word image sequence includes a first query single-word image and a second query single-word image;

[0075] Step 104, screening each candidate search word picture according to the first query word picture to obtain a first search word picture;

[0076] Step 105, screening the candidate search single word pictures after the first search single word picture according to the second query single word picture to obtain a second search single word picture;

[0077] Step 106, determining target search text information based on the first search word image and the second search word image.

[0078] In the steps 101 to 106 shown in the embodiment of the present application, when facing a text search for a target query image according to the target query text, it is not necessary to identify the text from the target query image, but on the one hand, a search single word image sequence is detected from the target query image through a target single word detection model. On the other hand, a query single word image sequence is found from an image database according to the target query text. Then, the search single word image sequence is screened according to the single word images in the query single word image sequence to obtain a first search single word image and a second search single word image. In this way, the possibility of retrieval failure caused by errors in text recognition or limitations of the text recognition character library is reduced. In addition, when performing image screening, after the first search single word image is screened, it is only necessary to screen the candidate search single word images after the first search single word image according to the second query single word image, thereby reducing the number of image comparisons and ensuring the efficiency of image screening. In summary, the present application can improve the accuracy of text in the search image and ensure the efficiency of image screening.

[0079] In step 101 of some embodiments, a target query image and a target query text are obtained. The target query image refers to an image containing certain text content and is used for retrieval. The target query image includes a medical examination report image, a car insurance claim form image, a document image, etc. The target query text refers to the text to be queried. For example, the user's handwritten "compulsory traffic insurance" is used as the target query text, and the user's corresponding car insurance claim form image is used as the target query image.

[0080] In step 102 of some embodiments, a target query image is subjected to text detection by a preset target single word detection model to obtain a retrieval single word image sequence. The retrieval single word image sequence refers to a sequence composed of images. The retrieval single word image sequence includes at least two candidate retrieval single word images. The candidate retrieval single word image refers to the image area where the single word in the target query image is located.

[0081] The target single word detection model is a neural network model used for text detection. For example, the target single word detection model is a resnet18 model or a resnet50 model.

[0082] In one embodiment, step 102 may include:

[0083] Performing text coordinate detection on the target query image through a preset target single word detection model to obtain a target single word coordinate sequence, wherein the target single word coordinate sequence includes at least two single word coordinates, and each single word coordinate has a detection confidence;

[0084] Cut out the target query image according to each word coordinate in the target coordinate sequence to obtain a candidate retrieval word image;

[0085] Summarize according to each candidate retrieved character picture to obtain a retrieved character picture sequence.

[0086] The benefit of the above embodiment is that by cropping pictures based on character coordinates to obtain candidate retrieved character pictures, it is possible to avoid extracting candidate retrieved character pictures that are not characters, thereby reducing the number of pictures, and further contributing to improving the subsequent picture screening efficiency.

[0087] In step 103 of some embodiments, search the preset picture database according to the target query text to obtain a query character picture sequence. The query character picture sequence refers to a sequence composed of pictures.

[0088] If the target query text includes a first query character and a second query character, the query character picture sequence includes a first query character picture and a second query character picture. The first query character picture is ranked before the second query character picture. For example, if the target query text is "health", the first query character picture is the picture corresponding to "jian", and the second query character picture is the picture corresponding to "kang".

[0089] If the target query text includes a first query character, a second query character, and a third query character, the query character picture sequence includes a first query character picture, a second query character picture, and a third query character picture. The first query character picture, the second query character picture, and the third query character picture are arranged in order. For example, if the target query text is "health insurance", the first query character picture is the picture corresponding to "jian", the second query character picture is the picture corresponding to "kang", and the third query character picture is the picture corresponding to "xian".

[0090] It can be understood that the number of characters in the target query text is equal to the number of pictures in the query character picture sequence.

[0091] In one embodiment, step 103 may include: segment the target query text to obtain a first query character and a second query character; search the candidate character pictures in the picture database according to the first query character to obtain a first query character picture; search the candidate character pictures in the picture database according to the second query character to obtain a second query character picture; summarize the first query character picture and the second query character picture to obtain a query character picture sequence. In this way, on the basis of being able to obtain the query character picture sequence, the acquisition method is simplified and the flexibility is high.

[0092] In one embodiment, the above-mentioned step of searching for candidate single-word images in the image database according to the first query single word to obtain the first query single-word image may include: searching the candidate single-word sub-library of the image database according to the first query single word to obtain the target single-word sub-library; classifying the first query single-word image by the target single-word classification model to obtain the first target font category; randomly selecting a picture from the target picture sub-library according to the first target font category to obtain the first query single-word image. Each candidate picture font library includes at least two different font images of the same single word. In this way, by first finding the sub-library and then finding the first query single-word image from the sub-library according to the font category, the number of comparisons between the first query single word and other irrelevant pictures in the image database is reduced, thereby improving the search efficiency.

[0093] It is understandable that the above step of searching the candidate single word pictures in the picture database according to the second query single word to obtain the second query single word picture is basically the same as the above embodiment, and will not be repeated here. Similarly, if there is a third query single word picture, the search process is also basically the same as the above embodiment.

[0094] In step 104 of some embodiments, each candidate search word picture is screened according to the first query word picture to obtain a first search word picture. The first search word picture is a candidate search word picture that matches the first query word picture.

[0095] In one embodiment, referring to Figure 2 , step 104 may include:

[0096] Step 201, extracting features from a first query word image through a feature layer of a preset target word classification model to obtain a first query word vector;

[0097] Step 202, extracting features from the candidate retrieval word image through the feature layer of the target word classification model to obtain a candidate retrieval word vector;

[0098] Step 203, performing vector matching based on the first query word vector and the candidate search word vector to obtain a vector matching degree corresponding to the candidate search word image;

[0099] Step 204 , selecting each candidate retrieval single-word picture according to the vector matching degree to obtain a first retrieval single-word picture.

[0100] In one embodiment, before step 201, refer to Figure 3 The method for retrieving text in an image provided by this embodiment may further include: training a target single word detection model and a target single word classification model, including:

[0101] Step 301, obtaining a sample query image;

[0102] Step 302, performing text detection on the sample query image through a preset initial single word detection model to obtain a sample retrieval single word image;

[0103] Step 303, classifying the sample search word image by using a preset initial word classification model to obtain a sample word category;

[0104] Step 304, performing loss calculation according to the sample single word category and the preset label single word category to obtain target loss data; wherein the label single word font category is used to characterize the word to which the sample retrieval single word image belongs;

[0105] Step 305, adjusting parameters of the initial single-word detection model according to the target loss data to obtain a target single-word detection model, and adjusting parameters of the initial single-word classification model to obtain a target single-word classification model.

[0106] In step 301, the sample query image refers to an image containing certain text content and is used for retrieval. The sample query image includes a medical examination report image, a car insurance claim form image, a document image, etc.

[0107] In one example, a picture dataset supporting retrieval of all Chinese characters is constructed to solve the problem that the current handwriting dataset cannot cover all Chinese characters. After the picture dataset is constructed, a sample query picture can be obtained from the picture dataset.

[0108] The specific steps to construct the image dataset are as follows: (1) For all the current 20,000+ Chinese characters, select different printed font libraries and handwritten font libraries, and generate Chinese character images of different sizes and backgrounds for each Chinese character. Each single character has multiple font images. (2) Randomly generate multiple Chinese characters for classification model training: For the multiple images above, treat multiple images of the same character, including handwritten and multiple printed fonts, as one category and put the same character in one image sub-library. (3) Randomly generate multiple Chinese character detection training data: Randomly select some images from (1) and (2), paste a picture corresponding to a character in each row of a new image, and record the coordinate position of the character in the image (x, y, w, h), where x is the value of the horizontal coordinate of the upper left corner of the image, y is the value of the vertical coordinate of the upper left corner, w is the width of the Chinese character, and h is the height of the image. For each image, there is a list of text that records the coordinates of each character as the label of the training set.

[0109] In step 302, the initial single word detection model is a neural network model for text detection. For example, the initial single word detection model is a resnet18 model or a resnet50 model. The initial single word detection model has the same structure as the target single word detection model mentioned above, but has different parameters.

[0110] In one embodiment, step 302 may include:

[0111] Performing text coordinate detection on the target query image through an initial single-word detection model to obtain a sample initial single-word coordinate sequence, wherein the sample initial single-word coordinate sequence includes at least two single-word coordinates, and each single-word coordinate has a detection confidence;

[0112] The single-word coordinates whose detection confidence is less than or equal to the confidence threshold in the sample initial single-word coordinate sequence are deleted to obtain the sample target coordinate sequence;

[0113] The sample query image is cut out according to each single word coordinate in the sample target coordinate sequence to obtain a sample retrieval single word image.

[0114] The benefit of the above embodiment is that by detecting the confidence to filter out unreasonable single-word coordinates and using the filtered single-word coordinates for cutting out images, it is possible to avoid extracting candidate retrieval single-word images that are not words, thereby reducing the number of images, and further helping to improve the accuracy and efficiency of model training.

[0115] In step 303, the initial single word classification model is a neural network model for text classification, for example, the initial single word classification model is a resnet18 model or a resnet50 model.

[0116] In one embodiment, referring to Figure 4 , step 303 may include:

[0117] Step 401, extracting features from the sample retrieval single word image through the feature layer of the initial single word classification model to obtain a sample retrieval single word feature vector;

[0118] Step 402, single-word classification is performed on the sample retrieved single-word feature vector through the classification layer of the initial single-word classification model to obtain the sample single-word category.

[0119] The initial single-word classification model generally consists of multiple layers, where the penultimate layer is the feature layer and the last layer is the classification layer (such as the softmax layer). The vector dimension of the feature layer output is usually 128 or 256 dimensions, representing the feature information of the image. The target single-word classification model is obtained by adjusting the parameters of the feature layer of the initial single-word classification model.

[0120] The benefit of the above embodiment is that text classification is performed with the help of a part of the existing classification model, and only a part of the structure needs to be trained, which can reduce the training cost.

[0121] In step 304, an error function may be used to calculate the loss of the sample word category and the preset label word category to obtain target loss data. The error function may include a mean square error function, a Euclidean distance function, a cosine similarity function, and the like.

[0122] In step 305, during the parameter adjustment process, the target loss data is compared with a preset loss threshold. If the target loss data is greater than the loss threshold, the parameters of the initial single-word detection model and the initial single-word classification model are continuously adjusted until the target loss data is less than or equal to the loss threshold. The parameter adjustment is then stopped to obtain the target single-word detection model and the target single-word classification model.

[0123] The benefit of the embodiment of step 301 to step 305 is that a target single-word detection model and a target single-word classification model can be trained, and the training effect is good.

[0124] In step 201, the feature layer of the target word classification model maps the first query word image to a high-dimensional vector space to obtain a first query word vector.

[0125] In step 202, the feature layer of the target word classification model maps the candidate retrieval word image to a high-dimensional vector space to obtain a candidate retrieval word vector.

[0126] It should be noted that there is no particular order in which step 201 and step 202 are performed.

[0127] In step 203, the higher the vector matching degree, the more likely it is that the candidate search word image is approximately the same as / similar to the first query word image.

[0128] In one embodiment, step 203 may include: normalizing the first query word vector to obtain a first word normalized vector; normalizing the candidate search word vector to obtain a candidate word normalized vector; performing a vector inner product operation on the first word normalized vector and the candidate word normalized vector to obtain a vector matching degree corresponding to the candidate search word image. In this way, the calculation efficiency is improved, which helps to improve the search efficiency.

[0129] In step 204, the candidate retrieval word images may be sorted in descending order according to the vector matching degree, so as to determine the first ranked candidate retrieval word image as the first retrieval word image. Other selection methods may also be adopted, such as randomly selecting the first retrieval word image from the candidate retrieval word images whose vector matching degree is greater than a threshold.

[0130] The benefit of the above-mentioned embodiment of step 201 to step 204 is that feature extraction is performed through the feature layer of the target single-word classification model without separately calling or setting a specific feature extractor, which simplifies the complexity of feature extraction and thus improves screening efficiency.

[0131] In step 105 of some embodiments, the candidate retrieval single word picture after the first retrieval single word picture is screened according to the second query single word picture to obtain the second retrieval single word picture. For example, the retrieval single word picture sequence includes P1, P2, P3, P4, a total of 4 candidate retrieval single word pictures. If the first retrieval single word picture is P1, then only P2, P3, P4 need to be screened according to the second query single word picture. If the first retrieval single word picture is P2, then only P3 and P4 need to be screened according to the second query single word picture. If the first retrieval single word picture is P3, then only P4 needs to be screened according to the second query single word picture.

[0132] In one embodiment, referring to Figure 5 , step 105 may include:

[0133] Step 501, extracting images from a search word image sequence according to a first search word image to obtain a search word image to be selected; wherein the search word image to be selected is a candidate search word image after the first search word image;

[0134] Step 502, performing layout detection on the first search single word image and the search single word image to be selected to obtain a layout detection result;

[0135] Step 503, if the typesetting detection result indicates that the first search word picture and the to-be-selected search word picture are in the same row or the same column, then image matching is performed on the second query word picture and the to-be-selected search word picture to obtain an image matching result;

[0136] Step 504: If the image matching result indicates that the second query single-word image matches the to-be-selected retrieval single-word image, the to-be-selected retrieval single-word image is determined as the second retrieval single-word image.

[0137] In step 501, the first search single word picture is a candidate search single word picture corresponding to the first query single word picture, and the position of the candidate search single word picture in the search single word picture sequence is fixed, so the position of the first search single word picture in the search single word picture sequence can be determined. According to the position of the first search single word picture in the search single word picture sequence, the next candidate search single word picture can be extracted from the search single word picture sequence to obtain the search single word picture to be selected. For example, the search single word picture sequence includes P1, P2, P3, and P4, a total of 4 candidate search single word pictures. If the first search single word picture is P1, the search single word picture to be selected is P2. If the first search single word picture is P2, the search single word picture to be selected is P3. If the first search single word picture is P3, the search single word picture to be selected is P4.

[0138] In step 502, the layout detection result indicates that the first search single-word image and the to-be-selected search single-word image are in the same row or column, or indicates that the first search single-word image and the to-be-selected search single-word image are not in the same row or column.

[0139] In one embodiment, step 502 may include:

[0140] Obtaining the position of the first search word image in the target query image to obtain first coordinate information;

[0141] Obtain the position of the single-word image to be selected for retrieval in the target query image to obtain the coordinate information to be selected;

[0142] If the difference between the ordinate of the first coordinate information and the ordinate of the to-be-selected coordinate information is less than or equal to a predetermined first threshold, a typesetting detection result is generated for characterizing that the first search single-word image and the to-be-selected search single-word image are in the same row;

[0143] If the difference between the abscissa of the first coordinate information and the abscissa of the coordinate information to be selected is less than or equal to a predetermined second threshold, a typesetting detection result is generated to characterize that the first search single-word image and the search single-word image to be selected are in the same column;

[0144] If the difference between the ordinate of the first coordinate information and the ordinate of the selected coordinate information is greater than a predetermined first threshold, and the difference between the abscissa of the first coordinate information and the abscissa of the selected coordinate information is greater than or equal to a predetermined second threshold, a typesetting detection result is generated to characterize that the first retrieved single-word image and the selected retrieved single-word image are not in the same column or the same column.

[0145] In step 503, if the layout detection result indicates that the first search word image and the search word image to be selected are in the same row or column, it means that the word corresponding to the first search word image and the word corresponding to the search word image to be selected are in the same row or column, reflecting the actual layout situation of whether the text is horizontal or vertical.

[0146] The image matching result indicates that the second query single-word image matches the to-be-selected retrieval single-word image, or indicates that the second query single-word image does not match the to-be-selected retrieval single-word image.

[0147] In one embodiment, performing image matching on the second query single word image and the to-be-selected search single word image in step 504 to obtain an image matching result may include:

[0148] Extract features of the second query word image through the feature layer of the target word classification model to obtain a second query word vector;

[0149] Through the feature layer of the target word classification model, feature extraction is performed on the image of the word to be retrieved, and a vector of the word to be retrieved is obtained;

[0150] Perform vector matching based on the second query word vector and the to-be-selected search word vector to obtain a vector matching degree corresponding to the to-be-selected search word image;

[0151] If the vector matching degree is greater than a predetermined matching degree threshold, a picture matching result is generated for representing the matching between the second query single word picture and the to-be-selected retrieval single word picture;

[0152] If the vector matching degree meets the predetermined matching degree threshold, a picture matching result is generated to indicate that the second query single-word picture does not match the to-be-selected retrieval single-word picture.

[0153] The benefit of the above embodiment is that feature extraction is performed through the feature layer of the target single-word classification model without separately calling or setting a specific feature extractor, which simplifies the complexity of feature extraction and thus improves screening efficiency.

[0154] In one embodiment, the above-mentioned step of performing vector matching based on the second query single word vector and the single word vector to be selected for retrieval to obtain the vector matching degree corresponding to the single word image to be selected for retrieval may include: normalizing the second query single word vector to obtain the second single word normalized vector; normalizing the single word vector to be selected for retrieval to obtain the single word normalized vector to be selected; performing vector inner product operation on the second single word normalized vector and the single word normalized vector to be selected to obtain the vector matching degree corresponding to the single word image to be selected for retrieval. In this way, the calculation efficiency is improved, which helps to improve the retrieval efficiency.

[0155] In one embodiment, after step 502, the method for retrieving characters in a picture provided by this embodiment may further include: updating the first search single-character picture, including:

[0156] If the typesetting detection result indicates that the first search single-word picture and the candidate search single-word picture are not in the same row or the same column, then the candidate search single-word pictures after the first search single-word picture are screened according to the first query single-word picture to obtain an updated first search single-word picture;

[0157] If the image matching result indicates that the second query single-word image does not match the candidate retrieval single-word image, the candidate retrieval single-word images after the first retrieval single-word image are screened according to the first query single-word image to obtain an updated first retrieval single-word image.

[0158] It should be noted that after obtaining the updated first search word image, step 105 will be re-entered to obtain the second search word image.

[0159] The benefit of the above embodiment is that, when the second search single-word picture cannot be found, the first search single-word picture can be effectively updated to re-search the second search single-word picture, thereby improving the search success rate.

[0160] In step 106 of some embodiments, target search text information is determined based on the first search word image and the second search word image. The target search text information includes information corresponding to the first search word image and information corresponding to the second search word image.

[0161] In one embodiment, step 106 may include: determining the first search word position information according to the position of the first search word image in the target query image; determining the second search word position information according to the position of the second search word image in the target query image; and performing information fusion based on the first search word position information and the second search word position information to obtain the target search text information. In this way, the position information of the target query text in the target query image can be obtained based on the target search text information.

[0162] In another embodiment, step 106 may include: obtaining a first search word based on the words included in the first search word image; obtaining a second search word based on the words included in the second search word image; and obtaining target search text information by summarizing the first search word and the second search word.

[0163] In summary of the above embodiments, the present application can at least achieve the following technical effects: 1. Solve the problem of OCR text retrieval in horizontal or vertical pictures, and can solve the problem of OCR not recognizing words or similar words. For similar words that OCR recognizes incorrectly, approximate matching can also be performed. 2. A large number of training samples can be quickly and randomly generated, and a large number of single-word detection data and text feature extraction training can be generated without manual annotation. 3. It can support all 2w Chinese characters, and can also well support search and recognition for handwriting or printed characters that are not in training. 4. Change the entire search problem into single-word detection and single-word vector approximate comparison, and solve the problem of approximate retrieval of horizontal or vertical text by combining the horizontal and vertical coordinate positions. 5. By converting the query text into simplified and traditional characters, it can also support the input of simplified characters, convert the simplified characters into the corresponding traditional characters, and then search, thereby solving the problem of inputting simplified Chinese characters but searching for traditional characters.

[0164] See also Figure 6 The embodiment of the present application also provides a device for retrieving text in a picture, which can implement the above method for retrieving text in a picture. Figure 6The module structure block diagram of the device for retrieving text in images provided in an embodiment of the present application includes: a data acquisition module 601, a text detection module 602, an image search module 603, a first image screening module 604, a second image screening module 605, and a retrieval information determination module 606. Among them, the data acquisition module 601 is used to obtain the target query image and the target query text; the text detection module 602 is used to perform text detection on the target query image through a preset target single-word detection model to obtain a retrieval single-word image sequence; wherein the retrieval single-word image sequence includes at least two candidate retrieval single-word images; the image search module 603 is used to search a preset image database according to the target query text to obtain a query single-word image sequence; wherein the query single-word image sequence includes a first query single-word image and a second query single-word image; the first image screening module 604 is used to screen each candidate retrieval single-word image according to the first query single-word image to obtain a first retrieval single-word image; the second image screening module 605 is used to screen the candidate retrieval single-word images after the first retrieval single-word image according to the second query single-word image to obtain a second retrieval single-word image; the retrieval information determination module 606 is used to determine the target retrieval text information according to the first retrieval single-word image and the second retrieval single-word image.

[0165] In one embodiment, the apparatus for retrieving characters in a picture further includes: a picture updating module, which is used to update the first search single-character picture.

[0166] In one embodiment, the apparatus for retrieving text in a picture further includes: a model training module, which is used to train a target single-word detection model and a target single-word classification model.

[0167] It should be noted that the specific implementation of the device for retrieving text in an image is basically the same as the specific implementation of the method for retrieving text in an image mentioned above, and will not be repeated here.

[0168] The embodiment of the present application also provides an electronic device, which includes: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the above-mentioned method of retrieving text in a picture. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0169] See also Figure 7 , Figure 7 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0170] The processor 701 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0171] The memory 702 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 702 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 702, and the processor 701 calls and executes the method for retrieving text in a picture in the embodiment of this application;

[0172] Input / output interface 703, used to implement information input and output;

[0173] Communication interface 704, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0174] A bus 705 that transmits information between the various components of the device (e.g., the processor 701, the memory 702, the input / output interface 703, and the communication interface 704);

[0175] The processor 701 , the memory 702 , the input / output interface 703 and the communication interface 704 are connected to each other in communication within the device via a bus 705 .

[0176] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned method of retrieving text in an image.

[0177] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0178] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0179] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0180] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0181] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0182] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0183] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0184] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0185] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0186] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0187] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0188] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for retrieving text in a picture, characterized in that: The method comprises: Obtain target query images and target query text; Performing text detection on the target query image through a preset target single-word detection model to obtain a retrieval single-word image sequence; wherein the retrieval single-word image sequence includes at least two candidate retrieval single-word images; Searching a preset picture database according to the target query text to obtain a query single-word picture sequence; wherein the query single-word picture sequence includes a first query single-word picture and a second query single-word picture; Screening each of the candidate search word images according to the first query word image to obtain a first search word image; The candidate search single word pictures after the first search single word picture are screened according to the second query single word picture to obtain a second search single word picture; According to the first search single word image and the second search single word image, target search text information is determined.

2. The method according to claim 1, characterized in that The step of screening the candidate search word pictures after the first search word picture according to the second query word picture to obtain a second search word picture includes: Extracting pictures from the retrieval word picture sequence according to the first retrieval word picture to obtain a candidate retrieval word picture; wherein the candidate retrieval word picture is a candidate retrieval word picture after the first retrieval word picture; Performing typesetting detection on the first search single-word image and the to-be-selected search single-word image to obtain a typesetting detection result; If the typesetting detection result indicates that the first search word picture and the to-be-selected search word picture are in the same row or the same column, then performing picture matching on the second query word picture and the to-be-selected search word picture to obtain a picture matching result; If the image matching result indicates that the second query single-word image matches the to-be-selected retrieval single-word image, the to-be-selected retrieval single-word image is determined as the second retrieval single-word image.

3. The method according to claim 2, characterized in that After performing typesetting detection on the first search single-word image and the to-be-selected search single-word image to obtain a typesetting detection result, the method further includes: The first search word image is updated, including: If the typesetting detection result indicates that the first search word picture and the candidate search word picture are not in the same row or the same column, the candidate search word pictures after the first search word picture are screened according to the first query word picture to obtain an updated first search word picture; If the picture matching result indicates that the second query single-word picture does not match the candidate retrieval single-word picture, the candidate retrieval single-word pictures after the first retrieval single-word picture are screened according to the first query single-word picture to obtain an updated first retrieval single-word picture.

4. The method according to any one of claims 1 to 3, characterized in that: The step of screening each of the candidate search word images according to the first query word image to obtain a first search word image includes: Performing feature extraction on the first query single word image through a feature layer of a preset target single word classification model to obtain a first query single word vector; Perform feature extraction on the candidate retrieval word image through the feature layer of the target word classification model to obtain a candidate retrieval word vector; Perform vector matching based on the first query word vector and the candidate search word vector to obtain the vector matching degree corresponding to the candidate search word image; Select each of the candidate retrieval single-word pictures according to the vector matching degree to obtain the first retrieval single-word picture.

5. The method according to any one of claims 1 to 3, characterized in that: The step of determining target search text information according to the first search word image and the second search word image comprises: Determining first search word position information according to the position of the first search word image in the target query image; Determining second search word position information according to the position of the second search word image in the target query image; Information fusion is performed based on the first search word position information and the second search word position information to obtain the target search text information.

6. The method according to claim 4, characterized in that Before extracting features of the first query word image through the feature layer of the preset target text classification model to obtain the first query word vector, the method further includes: The target word detection model and the target word classification model are obtained by training, including: Get sample query images; Performing text detection on the sample query image through a preset initial single-word detection model to obtain a sample retrieval single-word image; Performing text classification on the sample search word image through a preset initial word classification model to obtain a sample word category; The loss is calculated according to the sample single word category and the preset label single word category to obtain target loss data; wherein the label single word font category is used to characterize the word to which the sample retrieval single word image belongs; According to the target loss data, the parameters of the initial single word detection model are adjusted to obtain a target single word detection model, and the parameters of the initial single word classification model are adjusted to obtain a target single word classification model.

7. The method according to claim 6, characterized in that The method of performing text classification on the sample search word image by using a preset initial word classification model to obtain a sample word category includes: Performing feature extraction on the sample retrieval single word image through the feature layer of the initial single word classification model to obtain a sample retrieval single word feature vector; Performing single word classification on the sample search single word feature vector through the classification layer of the initial single word classification model to obtain the sample single word category; The target single-word classification model is obtained by adjusting parameters of the feature layer of the initial single-word classification model.

8. A device for retrieving text in a picture, characterized in that: The device comprises: A data acquisition module, used to acquire a target query image and a target query text; A text detection module is used to perform text detection on the target query image through a preset target single-word detection model to obtain a search single-word image sequence; wherein the search single-word image sequence includes at least two candidate search single-word images; An image search module is used to search a preset image database according to the target query text to obtain a query single-word image sequence; wherein the query single-word image sequence includes a first query single-word image and a second query single-word image; A first picture screening module, used for screening each of the candidate search single-word pictures according to the first query single-word picture to obtain a first search single-word picture; A second picture screening module, used for screening the candidate search single word pictures after the first search single word picture according to the second query single word picture to obtain a second search single word picture; The search information determination module is used to determine the target search text information according to the first search word image and the second search word image.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images