Text recognition methods, text recognition model training methods, devices and storage media
By acquiring target image features and reference feature sequences, and combining feature extraction and multi-task parallel prediction models, the problem of text recognition accuracy in cases of large character spacing and difficulty in distinguishing Chinese lines and columns is solved, achieving efficient and accurate text recognition results.
Patent Information
- Application Number
- CN202211356777.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-11-01
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2042-11-01
AI Technical Summary
Existing technologies have poor accuracy in text detection and recognition in scenarios with large character spacing and difficulty in distinguishing Chinese lines and columns.
A text recognition method is adopted, which determines the character position, category and connection relationship by acquiring the features and reference feature sequences of the target image. The text recognition is performed using a pre-trained text recognition model, which includes feature extraction, position encoding, and a combination of search and decoding models. A multi-task parallel prediction model is used to improve the recognition accuracy.
It improves the accuracy of text detection and recognition, especially in scenarios with text of arbitrary shapes, significantly enhancing the recognition effect and saving the data resources consumed in recognition.
Smart Images

Figure CN115909376B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of artificial intelligence technology, specifically deep learning, image processing, and computer vision technology, and can be applied to scenarios such as optical character recognition. Background Technology
[0002] Text detection and recognition technology can be widely used in various sectors of society, such as education, healthcare, and finance. Most related technologies employ a text granularity-based approach for text detection and recognition. However, in text scenarios with large character spacing and difficulty in distinguishing Chinese lines and columns, it is challenging to determine the text granularity based on visual information, leading to poor accuracy in detection and recognition. Summary of the Invention
[0003] This disclosure provides a text recognition method, a text recognition model training method, an apparatus, and a storage medium.
[0004] According to a first aspect of this disclosure, a text recognition method is provided, comprising:
[0005] Obtain the target features of the target image;
[0006] Obtain N reference feature sequences, where N is a positive integer;
[0007] Based on the target features and N reference feature sequences, N D-dimensional feature sequences are obtained, where D is a positive integer;
[0008] Based on N D-dimensional feature sequences, determine the character positions, character categories, and connections between characters in the target image;
[0009] Based on character position, character category, and the connection relationship between characters, the text recognition result of the target image is obtained.
[0010] According to a second aspect of this disclosure, a method for training a text recognition model is provided, comprising:
[0011] Obtain the training images;
[0012] Obtain N reference feature sequences, where N is a positive integer;
[0013] Input the image to be trained and N reference feature sequences into the model to be trained, and obtain the predicted values of the character positions, categories, and inter-character connections in the image to be trained from the output of the model.
[0014] By using the predicted values of character position, category, and inter-character connectivity, as well as the ground truth values of character position, category, and inter-character connectivity in the image to be trained, the text recognition model is trained to obtain the text recognition model.
[0015] According to a third aspect of this disclosure, a text recognition device is provided, comprising:
[0016] The first acquisition module is used to acquire the target features of the target image;
[0017] The second acquisition module is used to acquire N reference feature sequences, where N is a positive integer;
[0018] The first determining module is used to obtain N D-dimensional feature sequences based on the target features and N reference feature sequences, where D is a positive integer.
[0019] The second determining module is used to determine the character positions, character categories, and connections between characters in a target image based on N D-dimensional feature sequences.
[0020] The recognition module is used to obtain the text recognition results of the target image based on the character position, character category, and the connection relationship between characters.
[0021] According to a fourth aspect of this disclosure, a text recognition model training apparatus is provided, comprising:
[0022] The third acquisition module is used to acquire the images to be trained;
[0023] The fourth acquisition module is used to acquire N reference feature sequences, where N is a positive integer;
[0024] The third determination module is used to input the image to be trained and N reference feature sequences into the model to be trained, and obtain the predicted values of the position, category and inter-character connection relationship of the characters in the image to be trained output by the model to be trained.
[0025] The training module is used to train the training model by using the predicted values of character position, category, and inter-character connectivity, as well as the ground truth values of character position, category, and inter-character connectivity in the image to be trained, to obtain the text recognition model.
[0026] According to a fifth aspect of this disclosure, an electronic device is provided, comprising:
[0027] At least one processor;
[0028] Memory that is communicatively connected to at least one processor;
[0029] The memory stores instructions that can be executed by at least one processor, which enables the at least one processor to perform the text recognition method provided in the first aspect and / or the text recognition model training method provided in the second aspect.
[0030] According to a sixth aspect of this disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to perform the text recognition method provided in the first aspect and / or the text recognition model training method provided in the second aspect.
[0031] According to the seventh aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the text recognition method provided in the first aspect and / or the text recognition model training method provided in the second aspect.
[0032] According to the technical solution disclosed herein, the accuracy of text detection and recognition can be improved.
[0033] The above overview is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features of this application will become readily apparent from the accompanying drawings and the following detailed description. Attached Figure Description
[0034] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments disclosed in this application and should not be construed as limiting the scope of this application.
[0035] Figure 1 This is a schematic flowchart of a text recognition method according to an embodiment of the present disclosure;
[0036] Figure 2 This is a schematic diagram of the decoding model in the text recognition model according to an embodiment of the present disclosure;
[0037] Figure 3 This is a schematic diagram illustrating the processing of the prediction model in a text recognition model according to an embodiment of the present disclosure;
[0038] Figure 4 This is an illustration of text images of any shape according to embodiments of this disclosure;
[0039] Figure 5 This is a schematic diagram of a text recognition processing flow according to an embodiment of the present disclosure;
[0040] Figure 6 This is a schematic diagram illustrating the text recognition effect according to an embodiment of the present disclosure;
[0041] Figure 7 This is a flowchart illustrating a text recognition model training method according to an embodiment of the present disclosure;
[0042] Figure 8This is a schematic diagram of the structure of a text recognition device according to an embodiment of the present disclosure;
[0043] Figure 9 This is a schematic diagram of the structure of a text recognition model training device according to an embodiment of the present disclosure;
[0044] Figure 10 This is a schematic diagram of a text recognition scenario according to an embodiment of the present disclosure;
[0045] Figure 11 This is a schematic diagram illustrating a scenario of training a text recognition model according to an embodiment of the present disclosure;
[0046] Figure 12 This is a schematic diagram of the structure of an electronic device used to implement the text recognition model training method and / or text recognition method of the embodiments of this disclosure. Detailed Implementation
[0047] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.
[0048] The terms "first," "second," and "third," etc., used in the embodiments, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, such as including a series of steps or units. A method, system, product, or apparatus is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.
[0049] In related technologies, to address text recognition scenarios with arbitrary shapes, text detection and recognition schemes mainly employ a two-level approach based on candidate boxes: first, text candidate boxes are obtained, and then fine-grained detection and recognition are performed on the text within the boxes. However, detection and recognition based on text granularity often struggles to determine the granularity of text based on visual information in scenarios with large character spacing. Furthermore, it is difficult to obtain accurate results in scenarios where Chinese lines and columns are difficult to distinguish.
[0050] In order to at least partially solve one or more of the above-mentioned problems and other potential problems, this disclosure proposes a text recognition method that can improve the accuracy of text detection and recognition.
[0051] This disclosure provides a text recognition method. Figure 1 This is a flowchart illustrating a text recognition method according to an embodiment of the present disclosure. This text recognition method can be applied to a text recognition device. The text recognition device is located in an electronic device. The electronic device includes, but is not limited to, fixed devices and / or mobile devices. For example, fixed devices include, but are not limited to, servers, which can be cloud servers or ordinary servers. For example, mobile devices include, but are not limited to, mobile phones, tablets, and vehicle-mounted terminals. In some possible implementations, the text recognition method can also be implemented by a processor calling computer-readable instructions stored in memory. Figure 1 As shown, the text recognition method includes:
[0052] S101: Obtain the target features of the target image;
[0053] S102: Obtain N reference feature sequences, where N is a positive integer;
[0054] S103: Based on the target features and N reference feature sequences, obtain N D-dimensional feature sequences, where D is a positive integer;
[0055] S104: Based on N D-dimensional feature sequences, determine the character positions, character categories, and connections between characters in the target image;
[0056] S105: Based on character position, character category, and the connection relationship between characters, obtain the text recognition result of the target image.
[0057] In this embodiment of the disclosure, the target image is the image to be identified. The target image includes text content.
[0058] In this embodiment of the disclosure, any feasible implementation method can be used to acquire the target image. For example, the target image can be obtained by taking a picture. Another example is that the target image can be acquired locally. Yet another example is that the target image can be received from another device. The above are merely illustrative examples and are not intended to limit all possible methods of acquiring the target image; they are simply not exhaustive.
[0059] In this embodiment of the disclosure, the reference feature sequence is the feature sequence referenced during text recognition. The reference feature sequence can be obtained through self-learning. For example, given initial learned features, N reference feature sequences are obtained through a self-learning model. These N reference feature sequences can be continuously updated as the self-learning progresses. Taking English text recognition as an example, 37 reference feature sequences can be obtained based on the 26 English letters, the ten digits 0-9, and one non-character. Taking Chinese text recognition as an example, hundreds or thousands of reference feature sequences can be obtained based on Chinese characters in a dictionary, the ten digits 0-9, and one non-character.
[0060] In this embodiment, D refers to the number of dimensions of the feature sequence. For example, D = 2; or D = 3; or D = 5, etc. The value of D can be set or adjusted according to the recognition accuracy, or according to the recognition speed. Taking D = 3 as an example, the 3-dimensional feature sequence includes the feature sequence in the character position dimension, the feature sequence in the character category dimension, and the feature sequence in the relationship dimension between the character and other characters. The above is only an illustrative example and is not intended to limit all possible ways of D-dimensional feature sequences; it is simply not exhaustive.
[0061] In this embodiment, the character position can be the location of the character within the target image, which can be understood as the character frame position. This embodiment does not limit the shape of the character frame. For example, the character frame can be rectangular, circular, elliptical, etc.
[0062] In this embodiment of the disclosure, the character category can be the category to which the character belongs. For example, whether the character belongs to the alphabet, the number, the Chinese character, or a non-character category, etc.
[0063] In this embodiment of the disclosure, the connection relationship between characters can be the connection relationship between the current character and other characters. Here, the connection relationship can be represented by the positional relationship between characters. For example, if there are M characters in the target image, the connection relationship between any character and the other M-1 characters can be represented by the distance between the positions of any character and the other M-1 characters.
[0064] In this embodiment of the disclosure, determining the character positions, character categories, and connections between characters in a target image based on N D-dimensional feature sequences includes: determining the character positions, character categories, and connections between characters in the target image based on the N D-dimensional feature sequences. Here, the character positions, character categories, and connections between characters in the target image can be determined simultaneously. This helps improve the speed of text recognition.
[0065] The technical solution of this disclosure includes: acquiring target features of a target image; acquiring N reference feature sequences; obtaining N D-dimensional feature sequences based on the target features and the N reference feature sequences, which can provide unified data support for text recognition and improve the accuracy of the feature sequences on which text recognition is based; determining the character positions, character categories, and connections between characters in the target image based on the N D-dimensional feature sequences, which can simultaneously determine the character positions, character categories, and connections between characters, thereby helping to improve the speed of text recognition; and obtaining the text recognition result of the target image based on the character positions, character categories, and connections between characters. By considering the connections between characters during text recognition, the accuracy of text recognition can be improved, especially the text recognition effect in arbitrary shape text scenarios.
[0066] In some embodiments, S101 may include:
[0067] S101a: Obtain the initial features of the target image through the feature extraction model in the text recognition model;
[0068] S101b: Add positional encoding to the initial features to obtain the target features.
[0069] In this embodiment, the text recognition model is a pre-trained model, and the feature extraction model is a component of the text recognition model. How the text recognition model is trained will be described in detail later and will not be repeated here.
[0070] Here, the feature extraction model can use Convolutional Neural Networks (CNNs), and any CNN structure can be employed. Models such as Visual GeometryGroup Network (VGG), Renset (a backbone feature extraction network), DenseNet, and MobileNet (a lightweight convolutional neural network), as well as operators that can improve network performance, such as deformable convolution, dilated convolution, and inception (dilated) variants, can all be used in this approach. To simultaneously acquire local features and deeper semantic features, feature pyramid networks (FPNs) and path aggregation networks (PANs) can also be used to extract image features.
[0071] In some implementations, the target image is first input into the feature extraction model of the text recognition model to obtain the initial features of the target image output by the feature extraction model; then, positional encoding is added to the initial features to obtain the target features. Any feasible positional encoding method can be used, and will not be listed here. This improves the recognizability of the determined target features, thereby helping to improve the accuracy of the determined M-dimensional feature sequence, and ultimately improving the text recognition performance.
[0072] In some implementations, the target image is directly input into the feature extraction model of the text recognition model to obtain the initial features of the target image output by the feature extraction model; these initial features are then determined as the target features of the target image. This improves the speed of determining the target features, which in turn helps improve the speed of determining the M-dimensional feature sequence, thereby improving the text recognition speed.
[0073] This can improve the recognizability of the identified target features, thereby helping to improve the accuracy of identifying the M-dimensional feature sequence, and thus helping to improve the text recognition effect.
[0074] In some embodiments, S102 may include:
[0075] S102a: Input randomly initialized features into the text recognition model to search the model;
[0076] S102b: Obtain the N reference feature sequences output by the search model.
[0077] In this embodiment of the disclosure, the text recognition model is a pre-trained model, and the search model (query model) is a component of the text recognition model. How the text recognition model is trained will be described in detail later and will not be repeated here.
[0078] Here, the randomly initialized features can be adjusted according to the pre-identified content to make them more suitable for the target image, thereby helping to improve the accuracy of text recognition results. These randomly initialized features can be extracted from a known database.
[0079] In some implementations, N reference feature sequences generated by the query model are obtained by inputting randomly initialized features into the query model. This improves the speed of obtaining N reference feature sequences and helps to enhance the uniformity of the text recognition process.
[0080] Here, during the automatic learning process, the query model continuously updates N reference feature sequences. These N reference feature sequences can be learned through embedding or obtained through a predefined function, which will not be listed here.
[0081] In some implementations, obtaining N reference feature sequences includes: retrieving N reference feature sequences corresponding to a pre-identified target from a database. Here, the pre-identified target can be Chinese or a foreign language. This improves the accuracy of obtaining the N reference feature sequences.
[0082] This can improve the speed of acquiring N reference feature sequences, which helps to improve the uniformity of the text recognition process.
[0083] In some embodiments, S103 may include:
[0084] S103a: Input the target features and N reference feature sequences into the text recognition model to decode the model;
[0085] S103b: Obtain the N D-dimensional feature sequences output by the decoding model.
[0086] In this embodiment, the text recognition model is a pre-trained model, and the decoding model (transformer-decoder model) is a component of the text recognition model. How the text recognition model is trained will be described in detail later and will not be repeated here.
[0087] In some implementations, the target features and N reference feature sequences are input into the decoding model of the text recognition model to obtain N D-dimensional feature sequences of the target image output by the decoding model. The value of D can be set or adjusted according to the recognition accuracy or the recognition speed, and will not be listed here.
[0088] Figure 2 The diagram illustrates the processing of the decoding model in a text recognition model, as shown below. Figure 2 As shown, the image features of the target image and N reference feature sequences are input into the decoding model (transformer decoder model), and the decoding model outputs N D-dimensional feature sequences.
[0089] In this way, based on N reference feature sequences and combined with the target features of the target image, N D-dimensional feature sequences can be determined, which can provide data support for text recognition and improve the accuracy of the feature sequences on which text recognition is based.
[0090] In some embodiments, S104 may include:
[0091] S104a: Input the N D-dimensional feature sequences into the first prediction model, the second prediction model, and the third prediction model of the text recognition model, respectively;
[0092] S104b: Obtain the character positions of the target image output by the first prediction model, the character categories of the target image output by the second prediction model, and the connection relationships between the characters of the target image output by the third prediction model.
[0093] In this embodiment of the disclosure, the first prediction model is used to predict the position of the character, the second prediction model is used to predict the category of the character, and the third prediction model is used to predict the connection relationship between characters.
[0094] In this embodiment of the disclosure, the prediction model includes a first prediction model, a second prediction model, and a third prediction model.
[0095] In this embodiment of the disclosure, the text recognition model is a pre-trained model, and the prediction model is a component of the text recognition model. How the text recognition model is trained will be described in detail later and will not be repeated here.
[0096] In some implementations, each D-dimensional feature sequence is input into a first prediction model, a second prediction model, and a third prediction model of a text recognition model to obtain the character position of the D-dimensional feature sequence output by the first prediction model, the character category of the D-dimensional feature sequence output by the second prediction model, and the connection relationship between the characters of the D-dimensional feature sequence output by the third prediction model.
[0097] Figure 3 The diagram illustrates the processing of the prediction model in a text recognition model, such as... Figure 3 As shown, each D-dimensional feature sequence of the target image is input into the first, second, and third prediction models, respectively. The first nonlinear module outputs the character's position, the second nonlinear module outputs the character's category, and the third nonlinear module outputs the connection relationship between the character and other characters. The first, second, and third prediction models all employ a nonlinear feedforward network (FFN) structure, and their parameters are not shared. Thus, by using a multi-task parallel approach to simultaneously predict character position, character category, and the connection relationship between characters, text detection and recognition results can be obtained efficiently.
[0098] In this way, based on N D-dimensional feature sequences, the character positions, character categories, and connections between characters in a target image can be predicted simultaneously, which helps to improve the speed of text recognition.
[0099] In some embodiments, the text recognition result includes a bounding box and a string of text. S105 may include:
[0100] S105a: Determine candidate characters based on character category;
[0101] S105b: Determine the position of candidate characters based on their positions;
[0102] S105c: Based on the connection relationship between characters and the position of candidate characters, obtain the bounding box and string of the text included in the target image.
[0103] Here, candidate characters are characters that match the recognition target. For example, if the target image contains M characters, and two of these characters are classified as non-characters, then the target image contains M-2 candidate characters.
[0104] Here, determining the position of candidate characters based on character position includes: determining the position of each candidate character based on the position of the character corresponding to each character.
[0105] In some implementations, the bounding box and string of the text included in the target image are obtained based on the connection relationship between characters and the position of the candidate characters, including: determining the bounding box of the text based on the position of each candidate character; determining the adjacent characters of each candidate character based on the connection relationship between characters, and then obtaining the string.
[0106] Figure 4 This is an illustration of an arbitrary-shaped text image according to embodiments of this disclosure, such as... Figure 4 As shown, the image contains the strings "OYSTER" and "COMPANY", which are not on the same horizontal line and belong to arbitrary shape text; the "TOMALESBAY", "Since 1909", "Quality Oysters", and "The best…you depend on it!" in the image belong to normal shape text.
[0107] Figure 5 This is a schematic diagram of text recognition processing according to an embodiment of the present disclosure. Figure 4 The image shown is input into a text recognition model. The text recognition model determines the character positions, character categories, and connections between characters in the target image. Based on the character positions (which can be denoted as boxes), character categories (which can be denoted as cls), and connections between characters (which can be denoted as linking), the text recognition result of the target image is obtained, that is, the bounding box of the text instance and the corresponding string.
[0108] Will Figure 4 The image shown is input into a text recognition model, which can yield the following results: Figure 6 The text recognition results are shown. For example... Figure 6 As shown, both normal text and text of arbitrary shape can be correctly recognized. It can handle long text and text of arbitrary shape that are difficult for existing models to handle, thus improving the accuracy of text detection and recognition.
[0109] Therefore, when performing text recognition, considering character category, character position, and the connection relationship between characters simultaneously can improve the accuracy of text recognition, especially in text recognition scenarios with arbitrary shapes.
[0110] It should be understood that Figure 2 , Figure 3 , Figure 4 , Figure 5 and Figure 6 The schematic diagrams shown are merely illustrative and not limiting, and are scalable; those skilled in the art can use them as a basis. Figure 2 , Figure 3 , Figure 4 , Figure 5 and Figure 6 Even with various obvious changes and / or substitutions to the examples, the resulting technical solutions still fall within the scope of this disclosure.
[0111] This disclosure provides a text recognition model training method that can be applied to electronic devices. The following will be combined with... Figure 7 The flowchart shown illustrates a text recognition model training method provided in this disclosure. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order.
[0112] S701: Obtain the training image;
[0113] S702: Obtain N reference feature sequences, where N is a positive integer;
[0114] S703: Input the image to be trained and N reference feature sequences into the model to be trained, and obtain the predicted values of the character positions, categories, and inter-character connections of the image to be trained output by the model.
[0115] S704: Train the model to be trained by using the predicted values of character position, category, and inter-character connection relationship, as well as the ground truth values of character position, category, and inter-character connection relationship in the image to be trained, to obtain a text recognition model.
[0116] This disclosure does not restrict the execution order of S701 and S702. S701 and S702 may be executed simultaneously. Alternatively, S701 may be executed before S702. Alternatively, S702 may be executed before S701.
[0117] In this embodiment of the disclosure, the ground truth values of character position, category, and inter-character connection relationships in the image to be trained can be manually labeled ground truth values or reliable ground truth values obtained from a data source. It should be noted that this disclosure does not limit the source of the ground truth values.
[0118] In this embodiment of the disclosure, the text recognition model is used to output text recognition results based on the input image. The text recognition results may include text content and bounding boxes of the text.
[0119] The model to be trained is used to predict the position, category, and inter-character connectivity of characters in the image to be trained, based on the image to be trained and N reference feature sequences, and outputs text recognition results, including bounding boxes and strings. Based on this, it can be understood that in this embodiment, the model to be trained may include at least one feature extraction model for extracting target features from the target image; at least one search model for obtaining N reference feature sequences; and at least one prediction model for predicting the position, category, and inter-character connectivity of characters in the image to be trained. This prediction model includes at least a first prediction model, a second prediction model, and a third prediction model, wherein the first prediction model predicts the position of characters, the second prediction model predicts the category of characters, and the third prediction model predicts the inter-character connectivity. The text recognition model is obtained by training the model to be trained using the image to be trained; therefore, it has the same model structure as the model to be trained, the difference being that the model parameters are updated after training.
[0120] In this way, compared to the processing scheme of first obtaining text candidate boxes and then performing fine-grained detection and recognition on the text within the boxes, this method can improve the speed of text detection and recognition, as well as the accuracy of text detection and recognition.
[0121] In some embodiments, the model to be trained includes a first prediction model, a second prediction model, and a third prediction model. The first prediction model is used to predict the position of a character; the second prediction model is used to predict the category of a character; and the third prediction model is used to predict the connection relationships between characters.
[0122] In some embodiments, S703 may include:
[0123] S703a: Determine the first loss function based on the predicted position value and the true position value of the character;
[0124] S703b: Determine the second loss function based on the character's predicted class value and the character's true class value;
[0125] S703c: Determine the third loss function based on the predicted values and true values of inter-character connectivity;
[0126] S703d: The first prediction model is trained using the first loss function, the second prediction model is trained using the second loss function, and the third prediction model is trained using the third loss function.
[0127] Here, the first loss function can be the cross-entropy loss function (CE). It can be understood that the first loss function can also be other types of loss functions that can be used for multi-class classification tasks.
[0128] Here, the second loss function can be SmoothL1. It's understandable that the second loss function can also be other types of loss functions that can be used for regression tasks.
[0129] Here, the third loss function can be the binary cross-entropy loss function (BCE). It can be understood that the third loss function can also be other types of loss functions applicable to binary classification tasks.
[0130] This disclosure does not limit the execution order of S703a, S703b, and S703c. S703a, S703b, and S703c may be executed simultaneously. Alternatively, S703a may be executed before S703b and S703c, and S703b and S703c may be executed simultaneously. Alternatively, S703b may be executed before S703a and S703c, and S703a and S703c may be executed simultaneously. Alternatively, S703c may be executed before S703a and S703b, and S703a and S703b may be executed simultaneously. Alternatively, they may be executed sequentially in the order of S703a, S703b, and S703c. Alternatively, they may be executed sequentially in the order of S703a, S703c, and S703b. Alternatively, they may be executed sequentially in the order of S703b, S703a, and S703c. Alternatively, execute S703b, S703c, and S703a in that order.
[0131] This allows for a better determination of the loss function, improves the training speed of the text recognition model, and further enhances the accuracy of text detection and recognition.
[0132] In some embodiments, S704 may include:
[0133] S704a: Determine the intersection over union (IOU) ratio between the predicted position value and the true position value of a character;
[0134] S704b: Based on the intersection-union ratio, an M*N matrix is obtained, where N is the number of reference feature sequences and M is the number of characters included in the image to be trained;
[0135] S704c: Based on the M*N matrix, obtain the matching relationship between the predicted position value and the true position value of the character;
[0136] S704d: Based on the matching relationship, the category corresponding to the reference feature sequence that matches the ground truth value is set as the category of the character;
[0137] S704e: Based on the reference feature sequence that matches the position truth value, the truth value of the connection relationship belonging to the same text instance is set to 1, and the truth value of the connection relationship not belonging to the same text instance is set to 0, where 1 indicates that there is a connection relationship and 0 indicates that there is no connection relationship.
[0138] Based on this, training of the model to be trained is performed.
[0139] For training data S = {I, {L1, L2, ..., L...} n}}, where I is the image, L i ={L i1 ,L i2 …L im} represents a text instance, L ij Let I be the j-th character box corresponding to the i-th text instance and its category. Inputting I into the network, the network will output the prediction result P = {p1, p2, p3}, where p... i ={cls i ,loc i ,link i} These correspond to the character category prediction, position prediction, and connection relationship prediction for the i-th query, respectively. In this embodiment, the Hungarian algorithm is used to match the character category prediction, position prediction, and connection relationship prediction with their respective ground truth values (labels), dynamically assigning the ground truth value to each prediction result P.
[0140] For example, the specific training process is as follows:
[0141] Step 1: Calculate the IOU between each pair of predicted character boxes (referred to as prediction boxes) and character box labels (referred to as truth boxes) to obtain an M*N matrix;
[0142] Step 2: Based on the matrix obtained in Step 1, the matching relationship between the prediction box (referring to the character box) and the truth box is obtained through the Hungarian algorithm.
[0143] Step 3: Based on the matching relationships obtained in Step 2, set the label of the classification task corresponding to the query that matches the truth box to the corresponding character category, and set it to 0 for queries that do not match (0 indicates non-character). At the same time, based on the matched queries, set the label of the connection relationship belonging to the same text instance to 1 (1 indicates a connection relationship), and set the label of the connection relationship not belonging to the same text instance to 0 (0 indicates no connection relationship).
[0144] Step 4: Optimize the classification task, localization task (character bounding box position prediction), and character connection task using the three loss functions CE, SmoothL1, and BCE respectively, until convergence reaches a preset cutoff condition. Here, the preset cutoff condition can be that the accuracy reaches a preset threshold, or that the accuracy no longer improves within a preset time.
[0145] This can improve the training and optimization effect of the text recognition model, and further improve the accuracy of the text recognition model in text detection and recognition.
[0146] This disclosure proposes a high-efficiency and high-precision arbitrary-shape text detection and recognition scheme, which can further improve the detection and recognition accuracy of text, especially long text and arbitrary-shape text that are difficult for existing models to handle. This brings higher accuracy to other text recognition and understanding applications and improves text recognition performance. In addition, it can also save the bandwidth consumed by text recognition and save bandwidth resources.
[0147] This disclosure provides a text recognition device, such as... Figure 8 As shown, the text recognition device may include: a first acquisition module 801, used to acquire target features of a target image; a second acquisition module 802, used to acquire N reference feature sequences, where N is a positive integer; a first determination module 803, used to obtain N D-dimensional feature sequences based on the target features and the N reference feature sequences, where D is a positive integer; a second determination module 804, used to determine the character positions, character categories, and connections between characters in the target image based on the N D-dimensional feature sequences; and a recognition module 805, used to obtain the text recognition result of the target image based on the character positions, character categories, and connections between characters.
[0148] In some embodiments, the first acquisition module 801 is configured to: acquire initial features of the target image through a feature extraction model in a text recognition model; and add positional encoding to the initial features to obtain target features.
[0149] In some embodiments, the second acquisition module 802 is configured to: input randomly initialized features into the text recognition model to search the model; and acquire the N reference feature sequences output by the search model.
[0150] In some embodiments, the first determining module 803 is configured to: input the target feature and the N reference feature sequences into the decoding model of the text recognition model; and obtain the N D-dimensional feature sequences output by the decoding model.
[0151] In some embodiments, the second determining module 804 is configured to: input the N D-dimensional feature sequences into the first prediction model, the second prediction model, and the third prediction model of the text recognition model respectively; and obtain the character positions of the target image output by the first prediction model, the character categories of the target image output by the second prediction model, and the connection relationships between the characters of the target image output by the third prediction model.
[0152] In some embodiments, the recognition module 805 is configured to: determine candidate characters based on the character category; determine the position of the candidate characters based on the character position; and obtain the bounding box and string of the text included in the target image based on the connection relationship between the characters and the position of the candidate characters, wherein the text recognition result includes the bounding box and string of the text.
[0153] Those skilled in the art should understand that the functions of each processing module in the text recognition device of this disclosure embodiment can be understood with reference to the relevant description of the foregoing text recognition method. Each processing module in the text recognition device of this disclosure embodiment can be implemented by an analog circuit that implements the functions of this disclosure embodiment, or by running software that executes the functions of this disclosure embodiment on an electronic device.
[0154] The text recognition device of this disclosure can improve the accuracy of text detection and recognition.
[0155] This disclosure provides a text recognition model training device, such as... Figure 9 As shown, the device may include: a third acquisition module 901 for acquiring a training image; a fourth acquisition module 902 for acquiring N reference feature sequences, where N is a positive integer; a third determination module 903 for inputting the training image and the N reference feature sequences into a training model to obtain the predicted values of character positions, categories, and inter-character connections of the training image output by the training model; and a training module 904 for training the training model using the predicted values of character positions, categories, and inter-character connections, as well as the ground truth values of character positions, categories, and inter-character connections of the training image, to obtain a text recognition model.
[0156] In some embodiments, the model to be trained includes a first prediction model, a second prediction model, and a third prediction model. The training module 904 is configured to: determine a first loss function based on the predicted position value of the character and the true position value of the character; determine a second loss function based on the predicted category value of the character and the true category value of the character; and determine a third loss function based on the predicted connection relationship between the characters and the true connection relationship between the characters.
[0157] The first prediction model is trained using the first loss function, the second prediction model is trained using the second loss function, and the third prediction model is trained using the third loss function. The first prediction model is used to predict the position of the character, the second prediction model is used to predict the category of the character, and the third prediction model is used to predict the connection relationship between characters.
[0158] In some embodiments, the training module 904 is further configured to: determine the intersection-union ratio (IUR) of the predicted position value and the true position value of the character; obtain an M*N matrix based on the IUR, where N is the number of reference feature sequences and M is the number of characters included in the image to be trained; obtain the matching relationship between the predicted position value and the true position value of the character based on the M*N matrix; set the category corresponding to the reference feature sequence that matches the true position value as the category of the character based on the matching relationship; and set the true value of the connection relationship belonging to the same text instance to 1 and the true value of the connection relationship not belonging to the same text instance to 0 based on the reference feature sequence that matches the true position value, where 1 indicates a connection relationship and 0 indicates no connection relationship.
[0159] Those skilled in the art should understand that the functions of each processing module in the text recognition model training device of this disclosure can be understood with reference to the relevant description of the text recognition model training method described above. Each processing module in the text recognition model training device of this disclosure can be implemented by an analog circuit that implements the functions of this disclosure embodiment, or by running software that executes the functions of this disclosure embodiment on an electronic device.
[0160] The text recognition model training device of this disclosure can improve the accuracy of text detection and recognition.
[0161] This disclosure provides a schematic diagram of a text recognition scenario, such as... Figure 10 As shown.
[0162] As previously described, the text recognition method provided in this disclosure is applied to electronic devices. Electronic devices are intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices.
[0163] Obtain the target features of the target image;
[0164] Obtain N reference feature sequences, where N is a positive integer;
[0165] Based on the target feature and the N reference feature sequences, N D-dimensional feature sequences are obtained, where D is a positive integer;
[0166] Based on the N D-dimensional feature sequences, the character positions, character categories, and connections between characters in the target image are determined.
[0167] Based on the character's position, category, and connections between characters, the text recognition result of the target image is obtained.
[0168] The target features and reference feature sequences of the target image can be obtained from an image data source. The image data source can be various forms of data storage devices, such as laptops, desktop computers, workstations, personal digital assistants (PDAs), servers, blade servers, mainframes, and other suitable computers. The image data source can also represent various forms of mobile devices, such as PDAs, cellular phones, smartphones, wearable devices, and other similar computing devices. Furthermore, the image data source and the user terminal can be the same device.
[0169] It should be understood that Figure 10 The scene diagrams shown are merely illustrative and not restrictive; those skilled in the art can interpret them based on... Figure 10 Even with various obvious changes and / or substitutions to the examples, the resulting technical solutions still fall within the scope of this disclosure.
[0170] This disclosure also provides a text recognition model training device, such as... Figure 11 As shown.
[0171] As described above, the text recognition model training method provided in this disclosure is applied to electronic devices. Electronic devices are intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices.
[0172] Obtain the training images;
[0173] Obtain N reference feature sequences, where N is a positive integer;
[0174] Input the image to be trained and N reference feature sequences into the model to be trained, and obtain the predicted values of the character positions, categories, and inter-character connections in the image to be trained from the output of the model.
[0175] By using the predicted values of character position, category, and inter-character connectivity, as well as the ground truth values of character position, category, and inter-character connectivity in the image to be trained, the text recognition model is trained to obtain the text recognition model.
[0176] The training images and reference feature sequences can be obtained from an image data source. The image data source can be various forms of data storage devices, such as laptops, desktop computers, workstations, personal digital assistants (PDAs), servers, blade servers, mainframes, and other suitable computing devices. The image data source can also represent various forms of mobile devices, such as PDAs, cellular phones, smartphones, wearable devices, and other similar computing devices. Furthermore, the image data source and the user terminal can be the same device.
[0177] It should be understood that Figure 11 The scene diagrams shown are merely illustrative and not restrictive; those skilled in the art can interpret them based on... Figure 11 Even with various obvious changes and / or substitutions to the examples, the resulting technical solutions still fall within the scope of this disclosure.
[0178] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0179] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.
[0180] Figure 12A schematic block diagram of an example electronic device 1200 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.
[0181] like Figure 12 As shown, device 1200 includes a computing unit 1201, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 1202 or a computer program loaded from storage unit 1208 into random access memory (RAM) 1203. RAM 1203 may also store various programs and data required for the operation of device 1200. The computing unit 1201, ROM 1202, and RAM 1203 are interconnected via bus 1204. Input / output (I / O) interface 1205 is also connected to bus 1204.
[0182] Multiple components in device 1200 are connected to I / O interface 1205, including: input unit 1206, such as keyboard, mouse, etc.; output unit 1207, such as various types of monitors, speakers, etc.; storage unit 1208, such as disk, optical disk, etc.; and communication unit 1209, such as network card, modem, wireless transceiver, etc. Communication unit 1209 allows device 1200 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0183] The computing unit 1201 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 1201 performs the various methods and processes described above, such as text recognition methods / text recognition model training methods. For example, in some embodiments, the text recognition method / text recognition model training method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1208. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1200 via ROM 1202 and / or communication unit 1209. When the computer program is loaded into RAM 1203 and executed by computing unit 1201, one or more steps of the text recognition method described above can be performed. Alternatively, in other embodiments, computing unit 1201 can be configured to perform the text recognition method / text recognition model training method by any other suitable means (e.g., by means of firmware).
[0184] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0185] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0186] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM), flash memory, optical fiber, compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0187] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0188] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.
[0189] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.
[0190] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.
[0191] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.
[0192] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.
[0193] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.
Claims
1. A text recognition method, comprising: Obtain the target features of the target image; Obtain N reference feature sequences, which are obtained through a self-learning model, where N is a positive integer; Based on the target features and the N reference feature sequences, N D-dimensional feature sequences are obtained, where D is a positive integer; Based on the N D-dimensional feature sequences, the character positions, character categories, and connections between characters in the target image are determined. Based on the character position, the character category, and the connection relationship between the characters, the text recognition result of the target image is obtained; wherein, the connection relationship between the characters is the connection relationship between the current character and other characters; The acquisition of N reference feature sequences includes: Based on the pre-identified target, retrieve the N reference feature sequences corresponding to the pre-identified target from the database; The process of obtaining N D-dimensional feature sequences based on the target features and the N reference feature sequences includes: The target feature and the N reference feature sequences are input into the text recognition model and the decoding model. Obtain the N D-dimensional feature sequences output by the decoding model. The D-dimensional feature sequences include feature sequences in the character position dimension, feature sequences in the character category dimension, and feature sequences in the relationship dimension between the character and other characters.
2. The method according to claim 1, wherein, The acquisition of target features of the target image includes: The initial features of the target image are obtained through the feature extraction model in the text recognition model; The initial features are encoded with location codes to obtain the target features.
3. The method according to claim 1, wherein, The acquisition of N reference feature sequences includes: Randomly initialized features are input into the text recognition model to search the model; Obtain the N reference feature sequences output by the search model.
4. The method according to claim 1, wherein, The step of determining the character positions, character categories, and inter-character connections of the target image based on the N D-dimensional feature sequences includes: The N D-dimensional feature sequences are respectively input into the first prediction model, the second prediction model, and the third prediction model of the text recognition model; The character positions of the target image output by the first prediction model, the character categories of the target image output by the second prediction model, and the connection relationships between the characters of the target image output by the third prediction model are obtained.
5. The method according to claim 1, wherein, The process of obtaining the text recognition result of the target image based on the character position, the character category, and the connection relationship between the characters includes: Candidate characters are determined based on the character category; The position of the candidate character is determined based on the character position; Based on the connection relationship between the characters and the position of the candidate characters, the bounding box and string of the text included in the target image are obtained, and the text recognition result includes the bounding box and string of the text.
6. A method for training a text recognition model, comprising: Obtain the training images; Obtain N reference feature sequences, which are obtained through a self-learning model, where N is a positive integer; The image to be trained and the N reference feature sequences are input into the model to be trained to obtain the predicted values of the character positions, categories, and inter-character connections of the image to be trained, which are output by the model to be trained. The text recognition model is trained using the predicted values of character position, category, and inter-character connection relationships, as well as the ground truth values of character position, category, and inter-character connection relationships in the image to be trained, to obtain a text recognition model. The inter-character connection relationship refers to the connection relationship between the current character and other characters. The text recognition model includes a decoding model. The target features of the target image and the N reference feature sequences are input into the decoding model to obtain N D-dimensional feature sequences of the target image output by the decoding model. The text recognition model obtains the text recognition result of the target image based on the N D-dimensional feature sequences. The D-dimensional feature sequences include feature sequences in the character position dimension, character category dimension, and inter-character relationship dimension of the target image. The acquisition of N reference feature sequences includes: Based on the pre-identified target, the N reference feature sequences corresponding to the pre-identified target are obtained from the database.
7. The method according to claim 6, wherein, The model to be trained includes a first prediction model, a second prediction model, and a third prediction model. Training the model using the predicted position values, category values, and inter-character connection relationship values of the characters, as well as the ground truth values of the position values, category values, and inter-character connection relationship values of the characters included in the image to be trained, includes: A first loss function is determined based on the predicted position value of the character and the true position value of the character; A second loss function is determined based on the predicted class value of the character and the true class value of the character; A third loss function is determined based on the predicted values of the inter-character connection relationships and the true values of the inter-character connection relationships; The first prediction model is trained using the first loss function, the second prediction model is trained using the second loss function, and the third prediction model is trained using the third loss function. The first prediction model is used to predict the position of the character, the second prediction model is used to predict the category of the character, and the third prediction model is used to predict the connection relationship between characters.
8. The method according to claim 6 or 7, wherein, The step of training the model to be trained using the predicted position values, category values, and inter-character connection relationship values of the characters, as well as the ground truth values of the position values, category values, and inter-character connection relationship values of the characters in the image to be trained, includes: Determine the intersection-union ratio (IoU) of the predicted position value and the true position value of the character; Based on the intersection-union ratio, an M*N matrix is obtained, where N is the number of reference feature sequences and M is the number of characters included in the image to be trained. Based on the M*N matrix, the matching relationship between the predicted position value of the character and the true position value of the character is obtained; Based on the matching relationship, the category corresponding to the reference feature sequence that matches the true value at the position is set as the category of the character; Based on the reference feature sequence that matches the position truth value, the truth value of the connection relationship belonging to the same text instance is set to 1, and the truth value of the connection relationship not belonging to the same text instance is set to 0, where 1 indicates that there is a connection relationship and 0 indicates that there is no connection relationship.
9. A text recognition device, comprising: The first acquisition module is used to acquire the target features of the target image; The second acquisition module is used to acquire N reference feature sequences, which are obtained through a self-learning model, where N is a positive integer; The first determining module is used to obtain N D-dimensional feature sequences based on the target features and the N reference feature sequences, where D is a positive integer; The second determining module is used to determine the character positions, character categories, and connection relationships between characters in the target image based on the N D-dimensional feature sequences. The recognition module is used to obtain the text recognition result of the target image based on the character position, the character category, and the connection relationship between the characters; wherein, the connection relationship between the characters is the connection relationship between the current character and other characters; The second acquisition module is used for: Based on the pre-identified target, retrieve the N reference feature sequences corresponding to the pre-identified target from the database; The first determining module is used for: The target feature and the N reference feature sequences are input into the text recognition model and the decoding model. Obtain the N D-dimensional feature sequences output by the decoding model. The D-dimensional feature sequences include feature sequences in the character position dimension, feature sequences in the character category dimension, and feature sequences in the relationship dimension between the character and other characters.
10. The apparatus according to claim 9, wherein, The first acquisition module is used for: The initial features of the target image are obtained through the feature extraction model in the text recognition model; The initial features are encoded with location codes to obtain the target features.
11. The apparatus according to claim 9, wherein, The second acquisition module is used for: Randomly initialized features are input into the text recognition model to search the model; Obtain the N reference feature sequences output by the search model.
12. The apparatus according to claim 9, wherein, The second determining module is used for: The N D-dimensional feature sequences are respectively input into the first prediction model, the second prediction model, and the third prediction model of the text recognition model; The character positions of the target image output by the first prediction model, the character categories of the target image output by the second prediction model, and the connection relationships between the characters of the target image output by the third prediction model are obtained.
13. The apparatus according to claim 9, wherein, The identification module is used for: Candidate characters are determined based on the character category; The position of the candidate character is determined based on the character position; Based on the connection relationship between the characters and the position of the candidate characters, the bounding box and string of the text included in the target image are obtained, and the text recognition result includes the bounding box and string of the text.
14. A text recognition model training device, comprising: The third acquisition module is used to acquire the images to be trained; The fourth acquisition module is used to acquire N reference feature sequences, which are obtained through a self-learning model, where N is a positive integer; The third determining module is used to input the image to be trained and the N reference feature sequences into the model to be trained, and to obtain the predicted values of the position, category and inter-character connection relationships of the characters in the image to be trained output by the model to be trained. A training module is used to train the model to be trained using the predicted values of the character's position, category, and inter-character connection relationships, as well as the ground truth values of the character's position, category, and inter-character connection relationships in the image to be trained, to obtain a text recognition model; wherein, the inter-character connection relationship is the connection relationship between the current character and other characters; wherein, the text recognition model includes a decoding model, which inputs the target features of the target image and the N reference feature sequences into the decoding model, obtains the N D-dimensional feature sequences of the target image output by the decoding model, and the text recognition model obtains the text recognition result of the target image based on the N D-dimensional feature sequences; wherein, the D-dimensional feature sequences include feature sequences in the character position dimension of the target image, feature sequences in the character category dimension of the target image, and feature sequences in the relationship dimension between the character and other characters in the target image; The fourth acquisition module is used for: Based on the pre-identified target, the N reference feature sequences corresponding to the pre-identified target are obtained from the database.
15. The apparatus according to claim 14, wherein, The model to be trained includes a first prediction model, a second prediction model, and a third prediction model. The training module is used for: A first loss function is determined based on the predicted position value of the character and the true position value of the character; A second loss function is determined based on the predicted class value of the character and the true class value of the character; A third loss function is determined based on the predicted values of the inter-character connection relationships and the true values of the inter-character connection relationships; The first prediction model is trained using the first loss function, the second prediction model is trained using the second loss function, and the third prediction model is trained using the third loss function. The first prediction model is used to predict the position of the character, the second prediction model is used to predict the category of the character, and the third prediction model is used to predict the connection relationship between characters.
16. The apparatus according to claim 14 or 15, wherein, The training module is also used for: Determine the intersection-union ratio (IoU) of the predicted position value and the true position value of the character; Based on the intersection-union ratio, an M*N matrix is obtained, where N is the number of reference feature sequences and M is the number of characters included in the image to be trained. Based on the M*N matrix, the matching relationship between the predicted position value of the character and the true position value of the character is obtained; Based on the matching relationship, the category corresponding to the reference feature sequence that matches the true value at the position is set as the category of the character; Based on the reference feature sequence that matches the position truth value, the truth value of the connection relationship belonging to the same text instance is set to 1, and the truth value of the connection relationship not belonging to the same text instance is set to 0, where 1 indicates that there is a connection relationship and 0 indicates that there is no connection relationship.
17. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.
18. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.
19. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-8.
Citation Information
Patent Citations
Text recognition method and training method and device of text recognition network
CN114529891A
