Image text recognition method and feature decoding model training method

Abstract

Embodiments of the present specification provide an image text recognition method and a feature decoding model training method, wherein the image text recognition method comprises: acquiring an image to be recognized, wherein the image to be recognized comprises text information; performing feature extraction on the image to be recognized to obtain a target feature map corresponding to the image to be recognized; analyzing character position information in the text information according to the target feature map to obtain a target character feature corresponding to the image to be recognized; and determining a text recognition result in the image to be recognized according to the target character feature. By analyzing the position relationship between text characters in the target feature map, the positions of each character in the text information, including the characters at the back of long text, can be accurately located, thereby improving the robustness of the image text recognition in the length of the text.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to an image text recognition method. One or more embodiments of this specification also relate to a feature decoding model training method. The specification includes an image text recognition device, a feature decoding model training device, a computing device, a computer-readable storage medium, and a computer program. Background Technology

[0002] With the development of computer technology, text recognition technology has gradually become an important foundational technology in the field of optical character recognition (OCR), and it is often used as an important atomic capability in tasks such as document intelligence and natural scene understanding.

[0003] Currently, parallel attention mechanisms are commonly used for image text recognition. However, these methods require pre-setting a limit on the maximum length of the decoded text, making them unsuitable for images containing long text. This limits image text recognition to text length, thus creating an urgent need for an image text recognition scheme that is robust to text length limitations. Summary of the Invention

[0004] In view of this, embodiments of this specification provide an image text recognition method. One or more embodiments of this specification also relate to a feature decoding model training method. An image text recognition device, a feature decoding model training device, a computing device, a computer-readable storage medium, and a computer program are provided to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, an image text recognition method is provided, comprising:

[0006] Obtain the image to be recognized, which includes text information;

[0007] Feature extraction is performed on the image to be recognized to obtain the target feature map corresponding to the image to be recognized;

[0008] The target feature map is used to analyze the character position information in the text information to obtain the target character features corresponding to the image to be recognized;

[0009] Based on the characteristics of the target character, the text recognition result in the image to be recognized is determined.

[0010] According to a second aspect of the embodiments of this specification, a feature decoding model training method is provided, applied to a cloud-side device, comprising:

[0011] Obtain a sample set, which includes multiple sample images carrying sample text labels, and the sample images include sample text information;

[0012] Extract the first sample image from the sample set, wherein the first sample image is any sample image in the sample set;

[0013] Feature extraction is performed on the first sample image to obtain the first sample feature map corresponding to the first sample image;

[0014] Input the first sample feature map into the feature decoding model to obtain the first sample character features corresponding to the first sample image;

[0015] Based on the character features of the first sample, determine the first predicted text corresponding to the first sample image;

[0016] Calculate the training loss value based on the first predicted text and the first sample text label carried by the first sample image;

[0017] Based on the training loss value, adjust the model parameters of the feature decoding model, and return to the step of extracting the first sample image from the sample set until the second preset stopping condition is reached, and obtain the model parameters of the trained feature decoding model.

[0018] Send the model parameters of the trained feature decoding model to the edge device.

[0019] According to a third aspect of the embodiments of this specification, an image text recognition device is provided, comprising:

[0020] The first acquisition module is configured to acquire an image to be recognized, wherein the image to be recognized includes text information;

[0021] The first extraction module is configured to extract features from the image to be recognized and obtain the target feature map corresponding to the image to be recognized.

[0022] The analysis module is configured to analyze the character position information in the text information based on the target feature map to obtain the target character features corresponding to the image to be recognized;

[0023] The first determining module is configured to determine the text recognition result in the image to be recognized based on the characteristics of the target character.

[0024] According to a fourth aspect of the embodiments of this specification, a feature decoding model training apparatus is provided, applied to a cloud-side device, comprising:

[0025] The second acquisition module is configured to acquire a sample set, wherein the sample set includes multiple sample images carrying sample text labels, and the sample images include sample text information;

[0026] The second extraction module is configured to extract a first sample image from the sample set, wherein the first sample image is any sample image in the sample set;

[0027] The third extraction module is configured to extract features from the first sample image to obtain the first sample feature map corresponding to the first sample image.

[0028] The input module is configured to input the first sample feature map into the feature decoding model to obtain the first sample character features corresponding to the first sample image;

[0029] The second determining module is configured to determine the first predicted text corresponding to the first sample image based on the character features of the first sample;

[0030] The calculation module is configured to calculate the training loss value based on the first predicted text and the first sample text label carried by the first sample image;

[0031] The adjustment module is configured to adjust the model parameters of the feature decoding model based on the training loss value, and return to the step of extracting the first sample image from the sample set until the second preset stopping condition is reached, so as to obtain the model parameters of the trained feature decoding model.

[0032] The sending module is configured to send the model parameters of the trained feature decoding model to the end device.

[0033] According to a fifth aspect of the embodiments of this specification, a computing device is provided, comprising:

[0034] Memory and processor;

[0035] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the method provided in the first or second aspect above.

[0036] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the methods provided in the first or second aspect described above.

[0037] According to a seventh aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the method provided in the first or second aspect described above.

[0038] This specification provides an image-text recognition method according to one embodiment, which involves acquiring an image to be recognized, wherein the image to be recognized includes text information; extracting features from the image to be recognized to obtain a target feature map corresponding to the image to be recognized; analyzing the character position information in the text information based on the target feature map to obtain target character features corresponding to the image to be recognized; and determining the text recognition result in the image to be recognized based on the target character features. By analyzing the positional relationships between text characters in the target feature map, the position of each character in the text information can be accurately located, including characters at the end of long texts, thereby improving the robustness of image-text recognition across text lengths. Attached Figure Description

[0039] Figure 1 This is an architecture diagram of an image text recognition system provided in one embodiment of this specification;

[0040] Figure 2 This is an architecture diagram of another image text recognition system provided in one embodiment of this specification;

[0041] Figure 3 This is a flowchart illustrating an image text recognition method provided in one embodiment of this specification;

[0042] Figure 4 This is a schematic flowchart illustrating the feature map enhancement process in an image text recognition method provided in one embodiment of this specification;

[0043] Figure 5 This is a flowchart of a feature decoding model training method provided in one embodiment of this specification;

[0044] Figure 6 This is a flowchart of another image text recognition method provided in one embodiment of this specification;

[0045] Figure 7 This is a flowchart illustrating the processing procedure of an image text recognition method provided in one embodiment of this specification;

[0046] Figure 8 This is a schematic diagram of an image text recognition interface provided in one embodiment of this specification;

[0047] Figure 9 This is a schematic diagram of the structure of an image text recognition device provided in one embodiment of this specification;

[0048] Figure 10 This is a schematic diagram of the structure of a feature decoding model training device provided in one embodiment of this specification;

[0049] Figure 11 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0050] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0051] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0052] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0053] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0054] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0055] Optical Character Recognition (OCR) refers to the process by which electronic devices (such as scanners or digital cameras) examine printed characters on paper, determine their shapes by detecting dark and light patterns, and then translate the shapes into computer text using character recognition methods. In other words, for printed characters, optical methods are used to convert the text in paper documents into black and white dot matrix image files, and recognition software converts the text in the image into text format for further editing and processing by word processing software.

[0056] Optical character recognition (OCR) is a fundamental technology in the field of optical character recognition, often serving as a crucial core capability in tasks such as document intelligence and natural scene understanding. Text lengths are diverse, with long texts frequently appearing in contexts such as compound words, text lines, URLs, multilingual texts, and code, making them an indispensable aspect of OCR.

[0057] Currently, text recognition is typically performed using decoders based on parallel attention mechanisms (PAT). However, the above approach requires setting a limit on the maximum length of the decoded text in advance, which cannot adapt to images containing long text, thus limiting image text recognition to text length.

[0058] To address the aforementioned issues, this specification proposes a novel decoding paradigm. By analyzing the character position information in the text information based on the target feature map, the position of each character can be quickly located, thereby achieving robust string decoding capability. Specifically, the process involves: acquiring an image to be recognized, which includes text information; extracting features from the image to obtain a target feature map corresponding to the image; analyzing the character position information in the text information based on the target feature map to obtain target character features corresponding to the image; and determining the text recognition result in the image based on the target character features.

[0059] This specification provides an image text recognition method and also relates to a feature decoding model training method. An image text recognition device, a feature decoding model training device, a computing device, a computer-readable storage medium, and a computer program are described in detail in the following embodiments.

[0060] See Figure 1 , Figure 1 This specification illustrates an architecture diagram of an image text recognition system according to one embodiment of the present specification. The image text recognition system may include a client 100 and a server 200.

[0061] Client 100 is used to send an image to be recognized to server 200, wherein the image to be recognized includes text information;

[0062] Server 200 is used to extract features from the image to be recognized to obtain the target feature map corresponding to the image to be recognized; analyze the character position information in the text information based on the target feature map to obtain the target character features corresponding to the image to be recognized; determine the text recognition result in the image to be recognized based on the target character features; and send the text recognition result to client 100.

[0063] Client 100 is also used to receive text recognition results sent by server 200.

[0064] By applying the solutions in the embodiments of this specification, the position of each character in the text information can be accurately located by analyzing the adjacent positional relationship between text characters in the target feature map, including the later characters in long texts, thereby improving the robustness of image text recognition in terms of text length.

[0065] See Figure 2 , Figure 2 This specification illustrates an architecture diagram of another image text recognition system provided in one embodiment. The image text recognition system may include multiple clients 100 and a server 200. The clients 100 may be referred to as edge devices, and the server 200 may be referred to as cloud devices. Multiple clients 100 can establish communication connections through the server 200. In the image text recognition scenario, the server 200 is used to provide image text recognition services between the multiple clients 100. Each client 100 can act as a sender or receiver, communicating through the server 200.

[0066] Users can interact with server 200 through client 100 to receive data sent by other clients 100, or send data to other clients 100, etc. In the image text recognition scenario, users can publish data streams to server 200 through client 100, server 200 can generate text recognition results based on the data stream, and push the text recognition results to other clients that have established communication.

[0067] In this system, client 100 and server 200 establish a connection via a network. The network provides the medium for communication between client 100 and server 200. The network can include various connection types, such as wired or wireless communication links or fiber optic cables. Data transmitted by client 100 may need to undergo encoding, transcoding, compression, or other processing before being published to server 200.

[0068] Client 100 can be a browser, an app (application), a web application such as an H5 (HyperText Markup Language 5) application, a lightweight application (also known as a mini-program), or a cloud application. Client 100 can be developed based on the software development kit (SDK) of the corresponding service provided by server 200, such as a real-time communication (RTC) SDK. Client 100 can be deployed on electronic devices and depends on the device or certain apps on the device to run. Electronic devices may have displays and support information browsing, such as personal mobile terminals like mobile phones, tablets, and personal computers. Various other types of applications can also be configured on electronic devices, such as human-computer interaction applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, and social media platform software.

[0069] Server 200 may include servers providing various services, such as servers providing communication services to multiple clients, servers supporting backend training of models used on clients, and servers processing data sent by clients. It should be noted that server 200 can be implemented as a distributed server cluster composed of multiple servers, or as a single server. The server can also be a server in a distributed system, or a server integrated with blockchain. The server can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0070] It is worth noting that the image text recognition method provided in the embodiments of this specification is generally executed by the server. However, in other embodiments of this specification, the client may also have similar functions to the server, thereby executing the image text recognition method provided in the embodiments of this specification. In other embodiments, the image text recognition method provided in the embodiments of this specification may also be executed jointly by the client and the server.

[0071] See Figure 3 , Figure 3 This specification shows a flowchart of an image text recognition method according to an embodiment, which specifically includes the following steps:

[0072] Step 302: Obtain the image to be recognized, wherein the image to be recognized includes text information.

[0073] In one or more embodiments of this specification, an image text recognition request can be received, the image text recognition request carrying an image text recognition object, i.e., an image to be recognized, and the image to be recognized is further recognized to obtain a text recognition result.

[0074] Specifically, the image to be recognized refers to an image that includes text information. The text information can be in different languages ​​and styles, such as Chinese, English, etc. Chinese text information can be in KaiTi, SongTi, handwritten fonts, etc. The image to be recognized can be an image from different fields and scenes. The scene corresponding to the image to be recognized can be a natural scene, such as a street shop advertisement image; the scene can also be a handwritten scene, such as a handwritten letter image. The field corresponding to the image to be recognized includes, but is not limited to, the financial field, the educational field, etc., and the specific selection depends on the actual situation. This specification does not impose any limitations on this aspect in the embodiments.

[0075] In practical applications, there are various methods for acquiring the image to be recognized, and the specific method chosen depends on the actual situation. This specification does not impose any limitations on these methods in its embodiments. In one possible implementation of this specification, the image to be recognized can be received from a user-uploaded image. In another possible implementation of this specification, the image to be recognized can be read from other data acquisition devices or a database.

[0076] Step 304: Extract features from the image to be recognized to obtain the target feature map corresponding to the image to be recognized.

[0077] In one or more embodiments of this specification, after obtaining the image to be identified, further feature extraction can be performed on the image to be identified to obtain the target feature map corresponding to the image to be identified.

[0078] Specifically, the target feature map refers to the features of the previous image or feature map that are concatenated in the image after convolution by the convolution kernel.

[0079] In practical applications, there are various methods for feature extraction from images to be recognized. The specific method should be selected according to the actual situation. This specification does not limit the specific methods used in this embodiment.

[0080] In one possible implementation of this specification, a convolutional neural network (CNN) can be used to extract features from the image to be recognized, thereby obtaining the target feature map corresponding to the image to be recognized.

[0081] In another possible implementation of this specification, a visual feature extraction model can be used to extract features from the image to be recognized to obtain the target feature map corresponding to the image to be recognized. The visual feature extraction model refers to a Transformer with local perception capability, such as FocalNet, Swin Transformer, etc. The specific model is selected according to the actual situation, and the embodiments in this specification do not limit it in any way.

[0082] Step 306: Analyze the character position information in the text information based on the target feature map to obtain the target character features corresponding to the image to be recognized.

[0083] In one or more embodiments of this specification, after obtaining an image to be recognized, extracting features from the image to be recognized, and obtaining a target feature map corresponding to the image to be recognized, the character position information in the text information can be analyzed based on the target feature map to obtain the target character features corresponding to the image to be recognized.

[0084] Specifically, character position information refers to the information of each character in the text information. Character position information includes, but is not limited to, the positional relationships between characters and character location information. The positional relationships between characters refer to the adjacency relationships between character features in the text information included in the image to be recognized; these relationships are used to generate a character nearest neighbor matrix. The character nearest neighbor matrix describes the probability of positional adjacency between character features in the text information. Character location information refers to the location information of character features in the text information included in the image to be recognized. Character location information is used to generate the first character attention map corresponding to the first character of the string. Target character features are obtained based on the character attention maps and target feature maps of each character in the text information and are used for character classification.

[0085] In practical applications, there are various ways to analyze the character position information in text information based on the target feature map to obtain the target character features corresponding to the image to be recognized. The specific method to be selected depends on the actual situation, and the embodiments in this specification do not limit this in any way.

[0086] In one possible implementation of this specification, an open-source machine learning library can be used to analyze the character position information in the text information based on the target feature map to obtain the target character features corresponding to the image to be recognized.

[0087] In another possible implementation of this specification, the target character features corresponding to the image to be recognized can be generated by a pre-trained feature decoding model. That is, the above-mentioned analysis of the character position information in the text information based on the target feature map to obtain the target character features corresponding to the image to be recognized may include the following steps:

[0088] The target feature map is input into the feature decoding model, which analyzes the character position information in the text information to obtain the target character features corresponding to the image to be recognized.

[0089] Specifically, the feature decoding model is a machine learning model, which can also be understood as an algorithmic expression that searches for patterns or makes predictions by processing massive amounts of data. The feature decoding model can be called a nearest neighbor decoder (ND), used to convert the visual feature map of text into a string of arbitrary length. The feature decoding model is trained on multiple sample images carrying sample text labels and can include a positional relationship analysis unit, a location information analysis unit, and a character alignment unit.

[0090] It should be noted that before inputting the target feature map into the feature decoding model, the two-dimensional (h×w) target feature map can be reshaped into a one-dimensional feature sequence by the following formula (1), which facilitates matrix multiplication calculation in the process of analyzing the positional relationship between characters and character positioning information.

[0091]

[0092] in, The target feature map before reshaping. Let be the reshaped target feature map, c be the number of channels in the target feature map, h be the height of the feature map, w be the width of the feature map, R be a one-dimensional feature sequence with feature dimension c, and i be the i-th round of analysis operation.

[0093] By applying the scheme of the embodiments in this specification, the target feature map is input into the feature decoding model, and the feature decoding model analyzes the character position information in the text information to obtain the target character features corresponding to the image to be recognized, thereby realizing the efficient and accurate acquisition of target character features using the feature decoding model.

[0094] In one optional embodiment of this specification, the feature decoding model includes a positional relationship analysis unit and a positioning information analysis unit. The character position information includes the positional relationship between characters and character positioning information. The above-mentioned inputting the target feature map into the feature decoding model, and then analyzing the character position information in the text information to obtain the target character features corresponding to the image to be recognized, may include the following steps:

[0095] The target feature map is input into the positional relationship analysis unit, which analyzes the positional relationships between characters to obtain the character nearest neighbor matrix.

[0096] The target feature map is input into the localization information analysis unit, which analyzes the character localization information to obtain the first character attention map.

[0097] The target character features are determined based on the target feature map, the character nearest neighbor matrix, and the first character attention map.

[0098] Specifically, the positional relationship analysis unit, which can be called the Neighboring Navigator, is used to determine the character nearest neighbor matrix based on the neighbor feature points of each feature point in the target feature map. The goal of the positional relationship analysis unit is to ensure that the next neighbor feature point of the feature point at the current character's location is a feature point involved in the next character. The positioning information analysis unit, which can be called the First Character Locator, is used to generate an attention map of the first character of the string. The first character attention map reflects the positional information of the first character of the string.

[0099] It should be noted that during the analysis process of the positional relationship analysis unit, the target feature map can be linearly mapped to determine the target linear value (key). Further processing of the target linear value using bilinear layer transport and the application of a softmax function yields the character nearest neighbor matrix. Furthermore, in this embodiment, the character [EOS] is introduced to represent the character following the last character of the text information.

[0100] In practical applications, the target linear value can be determined using the following formula (2), and the character nearest neighbor matrix can be determined using the following formula (3):

[0101]

[0102] Where [·] represents a concatenation operation, H (i) For the target feature value, For the target feature map, e EOS For a learnable embedding of [EOS], S = hw + 1, This indicates that the target feature map and the learnable embedding of [EOS] are concatenated.

[0103]

[0104] Where, N (i) Let N be the nearest neighbor matrix of characters. jk Let σ be the probability that the k-th feature is the next neighbor of the i-th feature, σ be the softmax function, c be the number of channels, and H be the probability that the k-th feature is the next neighbor of the i-th feature. (i) For the target eigenvalue, W q W r W k Let b be the learnable weights of c×c. r ∈R, T denotes the matrix transpose operation.

[0105] During the analysis process of the location information analysis unit, global average pooling can be performed on the target feature map to obtain global features as the query. Then, the global features are multiplied by the target linear value, and the softmax function is applied to determine the attention map of the first character.

[0106] In practical applications, the global features can be determined using the following formula (4), and the first character attention map can be determined using the following formula (5):

[0107]

[0108] in, For global features, GAP(·) is the global average pooling operation, F (i) For the target feature map, W q Let c be the learnable weights.

[0109]

[0110] in, This is the attention map for the first character, where σ is the softmax function and c is the number of channels. For global features, H (i) For the target eigenvalue, W k Let c be the learnable weights.

[0111] The scheme implemented in this specification involves inputting the target feature map into a positional relationship analysis unit, which analyzes the positional relationships between characters to obtain a character nearest neighbor matrix. The target feature map is then input into a positioning information analysis unit, which analyzes the character positioning information to obtain a first-character attention map. Based on the target feature map, the character nearest neighbor matrix, and the first-character attention map, the target character features are determined. By calculating the character nearest neighbor matrix in a single step, the position of each character can be quickly located, thereby achieving length-robust string decoding capabilities.

[0112] In practical applications, there are multiple ways to determine the features of a target character based on the target feature map, the character nearest neighbor matrix, and the first character attention map. The specific method to be selected depends on the actual situation, and the embodiments in this specification do not impose any limitations on this.

[0113] In one possible implementation of this specification, the character attention map of each character can be determined in the same way as the character attention map of the first character, and the target character features can be determined based on the target feature map and the character attention maps of each character.

[0114] In another possible implementation of this specification, the character alignment unit in the feature decoding model can be used to determine the target character features; that is, the feature decoding model also includes a character alignment unit. The determination of the target character features based on the target feature map, the character nearest neighbor matrix, and the first character attention map can include the following steps:

[0115] Input the character nearest neighbor matrix and the first character attention map into the character alignment unit to obtain the character attention map of each character in the text information;

[0116] Based on the target feature map and the character attention map of each character, the target character features are determined.

[0117] Specifically, the character alignment unit is used to generate attention maps for each character in the text information.

[0118] It should be noted that by inputting the character nearest neighbor matrix and the first character attention map into the character alignment unit, the character attention map of the character to be determined can be determined using the character nearest neighbor matrix and the character attention map of the preceding character. That is, the above-mentioned method of inputting the character nearest neighbor matrix and the first character attention map into the character alignment unit to obtain the character attention map of each character in the text information can include the following steps:

[0119] Input the character nearest neighbor matrix and the first character attention map into the character alignment unit. In the character alignment unit, the second character attention map of the second character is determined based on the character nearest neighbor matrix and the first character attention map of the first character. The first character is the character whose character attention map has been determined, and the second character is the character after the first character.

[0120] In practical applications, within the character alignment unit, the character nearest neighbor matrix and the first character attention map of the first character can be multiplied to determine the second character attention map of the second character. Specifically, the character attention map of each character can be determined using the following formula (6):

[0121] A j =A j-1 N (6)

[0122] Among them, A j For the second character attention map, A j-1 Let A be the attention map for the first character, N be the character nearest neighbor matrix, and the termination condition for this process be A. j The probability value on [EOS] is greater than a preset probability threshold, such as 0.6.

[0123] By applying the scheme of the embodiments of this specification, the character nearest neighbor matrix and the first character attention map are input into the character alignment unit. In the character alignment unit, the character attention map of each character can be quickly determined by formula (6), which improves the efficiency of determining the character attention map of each character.

[0124] Furthermore, when determining the target character features based on the target feature map and the character attention maps of each character, the target feature map can be multiplied by the character attention maps of each character to obtain the target character features. Assuming L-1 recursion is performed, the resulting character attention map includes L-1 real feature attention maps and one [EOS] attention map.

[0125] In practical applications, the target character features can be determined using the following formula (7):

[0126] g (i) =A (i) H (i) ∈R L×c (7)

[0127] Among them, g (i) For the target character features, A (i) For character attention graphs, H (i) The target feature value is the target feature map corresponding to the target feature map.

[0128] By applying the scheme of the embodiments of this specification, the character nearest neighbor matrix and the first character attention map are input into the character alignment unit to obtain the character attention map of each character in the text information; based on the target feature map and the character attention map of each character, the target character features are determined, thereby accurately locating the position of each character in the text information, including the later characters in long texts, thus adapting to images including text of arbitrary length and improving the robustness of image text recognition in terms of text length.

[0129] In one optional embodiment of this specification, in order to enhance the length extrapolation capability of the feature decoding model during the inference stage, this embodiment proposes an attention sharpening strategy (AS) to address the error accumulation effect when the character alignment unit sequentially calculates the character attention map. That is, before determining the second character attention map of the second character based on the character nearest neighbor matrix and the first character attention map of the first character, the following steps may be included:

[0130] Sharpen the elements in the first character attention map to obtain the updated first character attention map;

[0131] Based on the character nearest neighbor matrix and the first character attention map of the first character, determine the second character attention map of the second character, including:

[0132] Based on the character nearest neighbor matrix and the updated first character attention map, determine the second character attention map.

[0133] It should be noted that while an imprecise feature map may not directly cause prediction errors for its corresponding features, it can affect the latter through the accumulation of attention errors. In practical applications, considering that imprecise attention maps typically have high entropy, the character attention map for each character can be determined using the following formula (8) in the embodiments of this specification:

[0134]

[0135]

[0136] α j =min(1+λ(j-1),μ) (10)

[0137] in, For character attention graphs, N (i) For character nearest neighbor matrix, For acute angle distribution, α j Here, is the temperature coefficient, s and t are indices of matrix A, and λ and μ are hyperparameters, such as λ = 2 and μ = 16, used to mitigate accumulated attention errors. This is the updated first attention character image after sharpening. Let matrix A (i) The value in row j-1, column t.

[0138] By applying the scheme of the embodiments in this specification, the elements in the first character attention map are sharpened to obtain an updated first character attention map; based on the character nearest neighbor matrix and the updated first character attention map, the second character attention map of the second character is determined, which solves the error accumulation effect when the character alignment unit calculates the character attention map sequentially and enhances the length extrapolation capability of the feature decoding model.

[0139] Step 308: Determine the text recognition result in the image to be recognized based on the target character features.

[0140] In one or more embodiments of this specification, an image to be recognized is acquired, features are extracted from the image to be recognized to obtain a target feature map corresponding to the image to be recognized, and character position information in the text information is analyzed based on the target feature map to obtain target character features corresponding to the image to be recognized. Further, the text recognition result in the image to be recognized can be determined based on the target character features.

[0141] In practical applications, when determining the text recognition result in an image based on the target character features, the target character features can be input into the fully connected layer (FC) in the feature decoding model to obtain the text recognition result.

[0142] By applying the scheme of the embodiments of this specification, the position of each character in the text information can be accurately located by analyzing the positional relationship between characters in the text information based on the target feature map, including the later characters in long texts. This adapts to images including texts of arbitrary length, improving the robustness of image text recognition in terms of text length. At the same time, since the feature decoding model has good feature learning ability, the accuracy of image text recognition is improved.

[0143] In practical applications, when using Connectionist Temporal Classification (CTC) for image text recognition, the poor feature learning ability leads to poor image text recognition accuracy. Projects often require cumbersome post-processing for error correction, affecting code maintenance efficiency, model generalization, and final recognition results. When using an autoregressive text recognition decoder (SAT) for image text recognition, attention and feature computation are deeply coupled at each step, resulting in a slow decoding process and poor length extrapolation performance.

[0144] In one optional embodiment of this specification, a lightweight feature enhancement module (FEM) is proposed to enhance context modeling of long dependencies, thereby improving the accuracy of image text recognition. That is, after analyzing the character position information in the text information based on the target feature map to obtain the target character features corresponding to the image to be recognized, the following steps may be included:

[0145] Based on the target character features, the target feature map is enhanced to obtain the enhanced target feature map. Then, the process returns to the step of analyzing the character position information in the text information based on the target feature map to obtain the target character features corresponding to the image to be recognized, until the first preset stopping condition is reached and the updated target character features are obtained.

[0146] Determining the text recognition result in the image to be recognized based on the features of the target character can include the following steps:

[0147] Based on the updated target character features, the text recognition result in the image to be recognized is determined.

[0148] Specifically, the first preset stopping condition may be that the number of iterations reaches a first preset number of iterations. The first preset number of iterations may be selected according to the actual situation, and the embodiments in this specification do not impose any limitations on it.

[0149] For example, suppose the first preset iteration count is 2, and the first preset stopping condition is that the iteration count reaches 2. Feature extraction is performed on the image to be recognized to obtain the target feature map corresponding to the image. Based on the target feature map, the character position information in the text information is analyzed to obtain the target character features corresponding to the image to be recognized. At this point, the iteration count is 0, and the first preset stopping condition is not met. Then, based on the target character features, the target feature map is enhanced to obtain the enhanced target feature map. The process returns to execute the analysis of the character position information in the text information based on the target feature map to obtain the target character features corresponding to the image to be recognized. At this point, the iteration count is 1, and the first preset stopping condition is not met. The process returns to execute the analysis of the character position information in the text information based on the target feature map to obtain the target character features corresponding to the image to be recognized. At this point, the iteration count is 2, and the first preset stopping condition is met. Based on the updated target character features, the text recognition result in the image to be recognized is determined.

[0150] The scheme implemented in this specification enhances the target feature map based on the target character features, obtaining an enhanced target feature map. It then returns to the step of analyzing the character position information in the text information based on the target feature map to obtain the target character features corresponding to the image to be recognized, until a first preset stopping condition is met, resulting in updated target character features. Based on the updated target character features, the text recognition result in the image to be recognized is determined. This achieves efficient capture of long-dependent feature relationships, thereby enhancing the feature representation of the feature map and improving the accuracy of image text recognition.

[0151] In practical applications, the above-mentioned enhancement process for the target feature map based on the target character features to obtain the enhanced target feature map may include the following steps:

[0152] Analyze the inter-character feature relationships of the target character features to obtain global character features;

[0153] Contextual information analysis is performed on global character features and target feature maps to obtain the target feature map after feature enhancement.

[0154] It should be noted that the target feature map can be enhanced using a sliding window Transformer layer. By inputting the target character features into the Transformer layer, the relationships between the target character features are captured, resulting in enhanced global character features with global information. The inter-character feature relationship refers to the positional relationship between character features. Global character features are used to describe the character features in text information from a global perspective.

[0155] Further, see Figure 4 , Figure 4 This diagram illustrates a flowchart of the feature map enhancement process in an image text recognition method according to an embodiment of this specification. Global character features can be input into a feature map enhancement unit, where the global character features are placed back onto the original target feature map. After passing through a simple convolutional block, global information is diffused, enabling the entire feature map to perceive global contextual information and obtain the enhanced target feature map.

[0156] In practical applications, the enhanced target feature map can be determined using the following formulas (11) and (12):

[0157]

[0158]

[0159] Among them, F (i+1) Here, C represents the target feature map after feature enhancement, and C is the convolutional processing. To recreate the global character features onto the original target feature map, H... (i) For the target feature value, A (i) For character attention maps, This is a global character feature.

[0160] The scheme described in this specification involves analyzing the inter-character feature relationships of the target character features to obtain global character features; then, contextual information analysis is performed on the global character features and the target feature map to obtain the enhanced target feature map. Since the input target character features do not contain redundant non-character features, the amount of data processing is reduced. Furthermore, the use of a sliding window attention mechanism improves both the efficiency of feature enhancement and the accuracy of image-text recognition.

[0161] In one optional embodiment of this specification, the training method of the above-described feature decoding model may include the following steps:

[0162] Obtain a sample set, which includes multiple sample images carrying sample text labels, and the sample images include sample text information;

[0163] Extract the first sample image from the sample set, wherein the first sample image is any sample image in the sample set;

[0164] Feature extraction is performed on the first sample image to obtain the first sample feature map corresponding to the first sample image;

[0165] Input the first sample feature map into the feature decoding model to obtain the first sample character features corresponding to the first sample image;

[0166] Based on the character features of the first sample, determine the first predicted text corresponding to the first sample image;

[0167] Calculate the training loss value based on the first predicted text and the first sample text label carried by the first sample image;

[0168] Based on the training loss value, adjust the model parameters of the feature decoding model, and return to the step of extracting the first sample image from the sample set until the second preset stopping condition is reached, thus obtaining the trained feature decoding model.

[0169] Specifically, the feature decoding model is trained using supervised training, meaning that each sample image in the sample set carries a real sample text label, which serves as the training objective and guides the training process of the feature decoding model. The sample set can be obtained by reading a large number of sample images carrying sample text labels from other data acquisition devices or databases. Alternatively, it can be obtained by receiving a large number of sample images carrying sample text labels input by the user. The specific method of obtaining the sample set is chosen based on the actual situation, and this specification does not impose any limitations on this method in the embodiments.

[0170] It is worth noting that the implementation method of "extracting features from the first sample image to obtain the first sample feature map corresponding to the first sample image" is the same as the implementation method of "extracting features from the image to be recognized to obtain the target feature map corresponding to the image to be recognized"; the implementation method of "inputting the first sample feature map into the feature decoding model to obtain the first sample character feature corresponding to the first sample image" is the same as the implementation method of "inputting the target feature map into the feature decoding model, analyzing the character position information in the text information through the feature decoding model, and obtaining the target character feature corresponding to the image to be recognized"; the implementation method of "determining the first predicted text corresponding to the first sample image based on the first sample character feature" is the same as the implementation method of "determining the text recognition result in the image to be recognized based on the target character feature". Therefore, the embodiments in this specification will not be described again.

[0171] In one possible implementation of this specification, the second preset stopping condition includes a training loss value that is less than or equal to a second preset threshold. After calculating the training loss value based on the first predicted text and the first sample text label carried by the first sample image, the training loss value is compared with the second preset threshold.

[0172] Specifically, if the training loss value is greater than the second preset threshold, it indicates that there is a large difference between the first predicted text and the first sample text label carried by the first sample image, and the feature decoding model has a poor prediction ability for the first sample feature map. At this time, the model parameters of the feature decoding model can be adjusted, and the step of extracting the first sample image from the sample set can be returned to continue training the feature decoding model until the training loss value is less than or equal to the second preset threshold, indicating that there is a small difference between the first predicted text and the first sample text label carried by the first sample image, and the second preset stopping condition is reached, thus obtaining the feature decoding model that has been trained.

[0173] In another possible implementation of this specification, in addition to comparing the relationship between the training loss value and the second preset threshold, a preset number of iterations can also be used to determine whether the current feature decoding model has been trained.

[0174] Specifically, if the training loss value is greater than the second preset threshold, the model parameters of the feature decoding model are adjusted, and the step of extracting the first sample image from the sample set is returned to continue training the feature decoding model until the second preset number of iterations is reached, then the iteration is stopped, and the trained feature decoding model is obtained. The second preset threshold and the second preset number of iterations are selected according to the actual situation, and the embodiments in this specification do not limit them in any way.

[0175] In practical applications, there are many functions for calculating training loss values, such as the character cross-entropy loss function. The specific function to be selected depends on the actual situation, and the embodiments in this specification do not impose any limitations on this.

[0176] The scheme implemented in this specification involves calculating a training loss value based on the first predicted text and the first sample text label carried by the first sample image. This training loss value is then compared with a second preset stopping condition. If the second preset stopping condition is not met, training of the feature decoding model continues until it is reached, thus completing the training and obtaining the feature decoding model. By continuously adjusting the model parameters of the feature decoding model, the final feature decoding model can be made more accurate.

[0177] In one optional embodiment of this specification, the feature decoding model includes a positional relationship analysis unit, a positioning information analysis unit, and a character alignment unit; the above-mentioned inputting the first sample feature map into the feature decoding model to obtain the first sample character features corresponding to the first sample image may include the following steps:

[0178] The first sample feature map is input into the positional relationship analysis unit, which analyzes the positional relationships between characters in the first sample text information to obtain the first sample character nearest neighbor matrix.

[0179] The first sample feature map is input into the positioning information analysis unit. The positioning information analysis unit analyzes the character positioning information in the first sample text information to obtain the first character attention map of the first sample.

[0180] Input the first sample character nearest neighbor matrix and the first sample first character attention map into the character alignment unit to obtain the first sample character attention map of each character in the first sample text information;

[0181] The features of the first sample character are determined based on the first sample feature map and the first sample character attention map of each character.

[0182] It should be noted that the implementation method of "inputting the first sample feature map into the positional relationship analysis unit, and analyzing the positional relationships between characters in the first sample text information to obtain the first sample character nearest neighbor matrix" is the same as the implementation method of "inputting the target feature map into the positional relationship analysis unit, and analyzing the positional relationships between characters to obtain the character nearest neighbor matrix". Similarly, the implementation method of "inputting the first sample feature map into the positioning information analysis unit, and analyzing the character positioning information in the first sample text information to obtain the first sample first character attention map" is the same as the implementation method of "inputting the target feature map into the positioning information analysis unit, and analyzing the positional relationships between characters to obtain the first sample character nearest neighbor matrix". The implementation method of "analyzing bit information to obtain the first character attention map" is the same as that of "inputting the first sample character nearest neighbor matrix and the first sample first character attention map into the character alignment unit to obtain the first sample character attention map of each character in the first sample text information". The implementation method of "inputting the character nearest neighbor matrix and the first character attention map into the character alignment unit to obtain the character attention map of each character in the text information" is the same as that of "inputting the character nearest neighbor matrix and the first character attention map into the character alignment unit to obtain the character attention map of each character in the text information". The implementation method of "determining the first sample character feature based on the first sample feature map and the first sample character attention map of each character" is the same as that of "determining the target character feature based on the target feature map and the character attention map of each character". Therefore, the embodiments in this specification will not be described again.

[0183] By applying the scheme of the embodiments of this specification, the first sample character features are determined by using the positional relationship analysis unit, the positioning information analysis unit and the character alignment unit. The adjacent positional relationship between text characters in the sample feature map is analyzed, which can accurately locate the position of each character in the sample text information, including the later characters in long texts. This improves the robustness of image text recognition in terms of text length. At the same time, since the feature decoding model has good feature learning ability, the accuracy of image text recognition is improved.

[0184] In one optional embodiment of this specification, before adjusting the model parameters of the feature decoding model based on the training loss value, the following steps may be included:

[0185] Calculate the attention map loss value based on the first sample character attention map of each character;

[0186] Based on the training loss value, adjust the model parameters of the feature decoding model, including:

[0187] Adjust the model parameters of the feature decoding model based on the training loss value and the attention map loss value.

[0188] It should be noted that, to avoid multi-activation, the embodiments in this specification may employ minimum entropy regularization to process the first sample character attention map of each character, obtaining the first attention map loss value. This allows each attention map to focus more on the core feature regions. For a model with a limited number of iterations, the final goal is to take the average loss. Simultaneously, in the character alignment unit, to enable the character alignment unit to learn to stop the iteration of feature map calculation, the second attention map loss value can be calculated based on the first sample character attention map of each character and the attention mapping of [EOS]. Furthermore, the model parameters of the feature decoding model are adjusted jointly based on the training loss value, the first attention map loss value, and the second attention map loss value.

[0189] In practical applications, the first attention map loss value can be determined using the following formula (13), and the second attention map loss value can be determined using the following formula (14):

[0190]

[0191]

[0192] in, The loss value is the first attention map value. The loss value is the second attention map value. For the first sample character attention map of each character, Attention mapping for [EOS].

[0193] Furthermore, the total loss value for adjusting the model parameters of the feature decoding model can be determined using the following formula (15):

[0194]

[0195] Where λ1 and λ2 are the weights for loss calculation, which can be set to 0.01 and 0.001 respectively, L (i) This is the total loss value. The loss value is the first attention map value. The loss value is the second attention map value. This represents the training loss value.

[0196] In one possible implementation of this specification, during the training of the feature decoding model, after obtaining the first sample character features corresponding to the first sample image, if feature enhancement processing is performed through the feature enhancement model, then when adjusting the model parameters using the total loss value, the model parameters of both the feature decoding model and the feature enhancement model can be adjusted simultaneously.

[0197] The scheme implemented in this specification improves the accuracy and training efficiency of the feature decoding model by adjusting the model parameters through the training loss value, the second attention map loss value determined by the additional supervision of the attention map based on [EOS], and the first attention map loss value determined by the minimum entropy regularization term of the attention map.

[0198] See Figure 5 , Figure 5 This specification illustrates a flowchart of a feature decoding model training method according to an embodiment. The feature decoding model training method is applied to a cloud-side device and specifically includes the following steps:

[0199] Step 502: Obtain a sample set, wherein the sample set includes multiple sample images carrying sample text labels, and the sample images include sample text information.

[0200] Step 504: Extract the first sample image from the sample set, wherein the first sample image is any sample image in the sample set.

[0201] Step 506: Extract features from the first sample image to obtain the first sample feature map corresponding to the first sample image.

[0202] Step 508: Input the first sample feature map into the feature decoding model to obtain the first sample character features corresponding to the first sample image.

[0203] Step 510: Determine the first predicted text corresponding to the first sample image based on the character features of the first sample.

[0204] Step 512: Calculate the training loss value based on the first predicted text and the first sample text label carried by the first sample image.

[0205] Step 514: Adjust the model parameters of the feature decoding model according to the training loss value, and return to the step of extracting the first sample image from the sample set until the second preset stopping condition is reached, and obtain the model parameters of the trained feature decoding model.

[0206] Step 516: Send the model parameters of the trained feature decoding model to the edge device.

[0207] It should be noted that the specific implementation methods of steps 502 to 514 are the same as those of steps 302 to 308 above, and will not be described again in the embodiments of this specification.

[0208] In practical applications, after the cloud-side device sends the model parameters of the trained feature decoding model to the edge device, the edge device can build a feature decoding model locally based on the model parameters, and then use the feature decoding model locally to perform image text recognition.

[0209] The scheme implemented in this specification involves calculating a training loss value based on the first predicted text and the first sample text label carried by the first sample image. This training loss value is then compared with a second preset stopping condition. If the second preset stopping condition is not met, training of the feature decoding model continues until it is reached, thus completing the training and obtaining the feature decoding model. By continuously adjusting the model parameters of the feature decoding model, the final feature decoding model can be made more accurate.

[0210] See Figure 6 , Figure 6 This specification shows a flowchart of another image text recognition method provided in one embodiment, which specifically includes the following steps:

[0211] Step 602: Obtain the image to be recognized, wherein the image to be recognized includes text information.

[0212] Step 604: Extract features from the image to be recognized to obtain the target feature map corresponding to the image to be recognized.

[0213] Step 606: Perform feature transformation on the target feature map to obtain the target character features corresponding to the image to be recognized.

[0214] Step 608: Based on the target character features, enhance the target feature map to obtain the enhanced target feature map, and return to execute the step of performing feature transformation on the target feature map to obtain the target character features corresponding to the image to be recognized, until the first preset stop condition is reached to obtain the updated target character features.

[0215] Step 610: Determine the text recognition result in the image to be recognized based on the updated target character features.

[0216] It should be noted that the specific implementation methods of steps 602 to 610 are the same as those of steps 302 to 308 above, and will not be described again in the embodiments of this specification.

[0217] The scheme described in this specification involves acquiring an image to be recognized, which includes text information. Feature extraction is performed on the image to obtain a target feature map. Feature transformation is then performed on the target feature map to obtain target character features corresponding to the image. Based on the target character features, the target feature map is enhanced to obtain an enhanced target feature map. The process continues until a first preset stopping condition is met, resulting in updated target character features. Based on the updated target character features, the text recognition result in the image is determined. This method efficiently captures global context information, enhances feature maps, and further improves feature representation capabilities and the final recognition effect.

[0218] See Figure 7 , Figure 7 This specification illustrates a flowchart of the processing procedure for an image text recognition method according to an embodiment, specifically including:

[0219] Feature extraction model: Input the image to be identified into the feature extraction model to obtain the target feature map corresponding to the image to be identified;

[0220] Feature Decoding Model: The target feature map is input into the positional relationship analysis unit of the feature decoding model. The positional relationship analysis unit analyzes the positional relationships between characters to obtain the character nearest neighbor matrix. The target feature map is input into the positioning information analysis unit of the feature decoding model. The positioning information analysis unit analyzes the character positioning information to obtain the first character attention map. The character nearest neighbor matrix and the first character attention map are input into the character alignment unit of the feature decoding model to obtain the character attention maps of each character in the text information. Based on the target feature map and the character attention maps of each character, the target character features are determined.

[0221] Feature enhancement model: The target character features are input into the Transformer layer to analyze the feature relationships between characters and obtain global character features; the global character features are input into the feature map enhancement unit to analyze the context information of the global character features and the target feature map and obtain the feature-enhanced target feature map; the feature-enhanced target feature map is input into the feature decoding model;

[0222] Feature Decoding Model: The enhanced target feature map is input into the positional relationship analysis unit of the feature decoding model. The positional relationship analysis unit analyzes the positional relationships between characters to obtain a character nearest neighbor matrix. The enhanced target feature map is then input into the positioning information analysis unit of the feature decoding model. The positioning information analysis unit analyzes the character positioning information to obtain a first character attention map. The character nearest neighbor matrix and the first character attention map are input into the character alignment unit of the feature decoding model to obtain character attention maps for each character in the text information. Based on the enhanced target feature map and the character attention maps of each character, the target character features are determined until a first preset stopping condition is reached to obtain updated target character features. The updated target character features are then input into the fully connected layer of the feature decoding model to determine the text recognition result in the image to be recognized.

[0223] For example, an image to be recognized, including the character "public", is input into a feature extraction model to begin the first round of image-text recognition. After the feature decoding model determines the target character features in the first round, these features are input into a fully connected layer in the feature decoding model, and the text recognition result in the image to be recognized is "public". At this point, it can be determined that the accuracy of the current text recognition result is poor. The target character features determined in the first round can then be input into a feature enhancement model. In the feature enhancement model, the target feature map is enhanced based on the target character features determined in the first round, resulting in a feature-enhanced target feature map. The feature decoding model then performs a second round of image-text recognition on the feature-enhanced target feature map. After the feature decoding model determines the target character features in the second round, these features are input into a fully connected layer in the feature decoding model, and the text recognition result in the image to be recognized is "public". At this point, it can be determined that the current text recognition result is accurate.

[0224] Applying the scheme of the embodiments in this specification, a length-insensitive scene text recognition scheme (LISTER) is proposed. The process involves extracting a target feature map from the image to be recognized using a feature extraction model. This target feature map is then fed into a feature decoding model to obtain character-level features and string prediction results. Both the character-level features and the target feature map are then fed into a feature enhancement model to obtain an enhanced target feature map. This enhanced target feature map can then be further decoded in the feature decoding model to obtain even better recognition results. This process can be iterated multiple times. Since the feature decoding model does not require a decoding length limit, the character nearest neighbor matrix is ​​calculated solely from the target feature map, requiring no additional sequence vectors and being uncoupled from character features; only the receptive field of the target feature map for local characters is needed. Furthermore, the sequential calculation process of the character attention map is very simple, involving only vector and matrix multiplication, making it very fast. In practical applications, attention sharpening strategies can be used to enhance the model's length extrapolation capability. In addition, the feature enhancement model can efficiently capture global contextual information, enhance the target feature map, and further improve feature representation capabilities and the final recognition effect.

[0225] In practical applications, the image-text recognition method proposed in this specification has significant advantages in recognizing images including long text, and also in recognizing commonly used images including short text. See Table 1 below, which is a comparison table of image-text recognition accuracy:

[0226] Table 1 Comparison of Image-Text Recognition Accuracy

[0227] plan Short text Long and short texts Connection timing classification method 90.2 72.9 Decoder based on parallel attention mechanism 91.4 61.6 Autoregressive text recognition decoder 91.3 66.5 The nearest neighbor decoder in this scheme 91.7 76.8 The nearest neighbor decoder and feature enhancement model in this scheme 92.7 77.8

[0228] As shown in the table above, the feature decoding model proposed in this specification can achieve high-precision recognition results when used alone or in conjunction with the feature enhancement model. Furthermore, the image text recognition scheme provided by the embodiments in this specification avoids the problem of missing characters, making image text recognition more complete and accurate.

[0229] See Figure 8 , Figure 8 This diagram illustrates an image-to-text recognition interface according to an embodiment of this specification. The image-to-text recognition interface is divided into an image input interface and a text recognition result display interface. The image input interface includes an image input box, an "OK" control, and a "Cancel" control. The text recognition result display interface includes a text recognition result display box.

[0230] The user inputs an image to be recognized, which includes text information, through an image input box displayed on the client. After clicking the "OK" button, the server retrieves the image. It then performs feature extraction on the image to obtain a target feature map. Based on the target feature map, it analyzes the character position information in the text information to obtain the target character features corresponding to the image. Based on the target character features, it determines the text recognition result in the image and sends the result to the client. The client displays the text recognition result in a text recognition result display box.

[0231] In practical applications, users can interact with controls in various ways, including clicking, double-clicking, touching, hovering, swiping, long-pressing, voice control, or shaking. The specific method chosen depends on the actual situation, and this specification does not impose any limitations on this.

[0232] Corresponding to the above-described image text recognition method embodiments, this specification also provides embodiments of image text recognition devices. Figure 9 A schematic diagram of the structure of an image text recognition device according to one embodiment of this specification is shown. Figure 9 As shown, the device includes:

[0233] The first acquisition module 902 is configured to acquire an image to be recognized, wherein the image to be recognized includes text information;

[0234] The first extraction module 904 is configured to extract features from the image to be recognized and obtain the target feature map corresponding to the image to be recognized.

[0235] The analysis module 906 is configured to analyze the character position information in the text information based on the target feature map to obtain the target character features corresponding to the image to be recognized;

[0236] The first determining module 908 is configured to determine the text recognition result in the image to be recognized based on the characteristics of the target character.

[0237] Optionally, the analysis module 906 is further configured to input the target feature map into the feature decoding model, and the feature decoding model analyzes the character position information in the text information to obtain the target character features corresponding to the image to be recognized.

[0238] Optionally, the feature decoding model includes a positional relationship analysis unit and a positioning information analysis unit. The character position information includes the positional relationship between characters and the character positioning information. The analysis module 906 is further configured to input the target feature map into the positional relationship analysis unit, analyze the positional relationship between characters through the positional relationship analysis unit, and obtain a character nearest neighbor matrix; input the target feature map into the positioning information analysis unit, analyze the character positioning information through the positioning information analysis unit, and obtain a first character attention map; and determine the target character features based on the target feature map, the character nearest neighbor matrix, and the first character attention map.

[0239] Optionally, the feature decoding model also includes a character alignment unit; the analysis module 906 is further configured to input the character nearest neighbor matrix and the first character attention map into the character alignment unit to obtain the character attention map of each character in the text information; and to determine the target character features based on the target feature map and the character attention map of each character.

[0240] Optionally, the analysis module 906 is further configured to input the character nearest neighbor matrix and the first character attention map into the character alignment unit, and in the character alignment unit, determine the second character attention map of the second character based on the character nearest neighbor matrix and the first character attention map of the first character, wherein the first character is the character whose character attention map has been determined, and the second character is the character following the first character.

[0241] Optionally, the apparatus further includes: a first processing module configured to sharpen elements in the first character attention map to obtain an updated first character attention map; and an analysis module 906 further configured to determine a second character attention map of the second character based on the character nearest neighbor matrix and the updated first character attention map.

[0242] Optionally, the device further includes: a second processing module configured to enhance the target feature map based on the target character features to obtain an enhanced target feature map, and return to execute the step of analyzing the character position information in the text information based on the target feature map to obtain the target character features corresponding to the image to be recognized, until a first preset stop condition is reached to obtain the updated target character features; and a first determining module 908 further configured to determine the text recognition result in the image to be recognized based on the updated target character features.

[0243] Optionally, the second processing module is further configured to perform inter-character feature relationship analysis on the target character features to obtain global character features; and to perform context information analysis on the global character features and the target feature map to obtain the target feature map after feature enhancement.

[0244] Optionally, the apparatus further includes: a training module configured to acquire a sample set, wherein the sample set includes multiple sample images carrying sample text labels, and the sample images include sample text information; extract a first sample image from the sample set, wherein the first sample image is any sample image in the sample set; perform feature extraction on the first sample image to obtain a first sample feature map corresponding to the first sample image; input the first sample feature map into a feature decoding model to obtain a first sample character feature corresponding to the first sample image; determine a first predicted text corresponding to the first sample image based on the first sample character feature; calculate a training loss value based on the first predicted text and the first sample text label carried by the first sample image; adjust the model parameters of the feature decoding model based on the training loss value, and return to execute the step of extracting the first sample image from the sample set until a second preset stopping condition is reached to obtain a trained feature decoding model.

[0245] Optionally, the feature decoding model includes a positional relationship analysis unit, a location information analysis unit, and a character alignment unit; the training module is further configured to input the first sample feature map into the positional relationship analysis unit, which analyzes the positional relationships between characters in the first sample text information to obtain a first sample character nearest neighbor matrix; input the first sample feature map into the location information analysis unit, which analyzes the character location information in the first sample text information to obtain a first sample first character attention map; input the first sample character nearest neighbor matrix and the first sample first character attention map into the character alignment unit to obtain a first sample character attention map for each character in the first sample text information; and determine the first sample character features based on the first sample feature map and the first sample character attention maps for each character.

[0246] Optionally, the training module is further configured to calculate the attention map loss value based on the first sample character attention map of each character; and to adjust the model parameters of the feature decoding model based on the training loss value and the attention map loss value.

[0247] By applying the scheme of the embodiments of this specification, the position of each character in the text information can be accurately located by analyzing the adjacent position relationship between text characters in the target feature map, including the later characters in long texts, thereby improving the robustness of image text recognition in terms of text length. At the same time, since the feature decoding model has good feature learning ability, the accuracy of image text recognition is improved.

[0248] The above is an illustrative scheme of an image text recognition device according to this embodiment. It should be noted that the technical solution of this image text recognition device and the technical solution of the image text recognition method described above belong to the same concept. For details not described in detail in the technical solution of the image text recognition device, please refer to the description of the technical solution of the image text recognition method described above.

[0249] Corresponding to the above-described embodiments of the feature decoding model training method, this specification also provides embodiments of the feature decoding model training apparatus. Figure 10 A schematic diagram of a feature decoding model training apparatus according to one embodiment of this specification is shown. Figure 10 As shown, this device is applied to cloud-side equipment and includes:

[0250] The second acquisition module 1002 is configured to acquire a sample set, wherein the sample set includes multiple sample images carrying sample text labels, and the sample images include sample text information.

[0251] The second extraction module 1004 is configured to extract a first sample image from a sample set, wherein the first sample image is any sample image in the sample set;

[0252] The third extraction module 1006 is configured to extract features from the first sample image to obtain the first sample feature map corresponding to the first sample image.

[0253] The input module 1008 is configured to input the first sample feature map into the feature decoding model to obtain the first sample character features corresponding to the first sample image;

[0254] The second determining module 1010 is configured to determine the first predicted text corresponding to the first sample image based on the character features of the first sample;

[0255] The calculation module 1012 is configured to calculate the training loss value based on the first predicted text and the first sample text label carried by the first sample image;

[0256] The adjustment module 1014 is configured to adjust the model parameters of the feature decoding model according to the training loss value, and return to the step of extracting the first sample image from the sample set until the second preset stopping condition is reached, so as to obtain the model parameters of the trained feature decoding model.

[0257] The sending module 1016 is configured to send the model parameters of the trained feature decoding model to the end device.

[0258] The scheme implemented in this specification involves calculating a training loss value based on the first predicted text and the first sample text label carried by the first sample image. This training loss value is then compared with a second preset stopping condition. If the second preset stopping condition is not met, training of the feature decoding model continues until it is reached, thus completing the training and obtaining the feature decoding model. By continuously adjusting the model parameters of the feature decoding model, the final feature decoding model can be made more accurate.

[0259] The above is an illustrative scheme of a feature decoding model training device according to this embodiment. It should be noted that the technical solution of this feature decoding model training device and the technical solution of the feature decoding model training method described above belong to the same concept. For details not described in detail in the technical solution of the feature decoding model training device, please refer to the description of the technical solution of the feature decoding model training method described above.

[0260] Figure 11 A structural block diagram of a computing device according to one embodiment of this specification is shown. The components of the computing device 1100 include, but are not limited to, a memory 1110 and a processor 1120. The processor 1120 is connected to the memory 1110 via a bus 1130, and a database 1150 is used to store data.

[0261] The computing device 1100 also includes an access device 1140, which enables the computing device 1100 to communicate via one or more networks 1160. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 1140 may include one or more of any type of wired or wireless network interface (e.g., Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Networks (WLAN) interface, a Wi-MAX (World Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0262] In one embodiment of this specification, the aforementioned components of the computing device 1100 and Figure 11 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 11 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0263] The computing device 1100 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 1100 can also be a mobile or stationary server.

[0264] The processor 1120 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described image text recognition method or feature decoding model training method.

[0265] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device belongs to the same concept as the technical solutions of the image text recognition method and the feature decoding model training method described above. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solutions of the image text recognition method or the feature decoding model training method described above.

[0266] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described image text recognition method or feature decoding model training method.

[0267] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solutions of the image text recognition method and the feature decoding model training method described above. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solutions of the image text recognition method or the feature decoding model training method described above.

[0268] An embodiment of this specification also provides a computer program, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described image text recognition method or feature decoding model training method.

[0269] The above is an illustrative scheme of a computer program according to this embodiment. It should be noted that the technical solution of this computer program belongs to the same concept as the technical solutions of the image text recognition method and the feature decoding model training method described above. For details not described in detail in the technical solution of the computer program, please refer to the description of the technical solutions of the image text recognition method or the feature decoding model training method described above.

[0270] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0271] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0272] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0273] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0274] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. An image text recognition method, comprising: Obtain an image to be recognized, wherein the image to be recognized includes text information, the text information includes character position information, the character position information includes the positional relationship between characters and character positioning information, the positional relationship between characters includes the positional adjacency relationship between each character feature in the text information, and the character positioning information includes the positional information of the character feature in the text information; Feature extraction is performed on the image to be identified to obtain the target feature map corresponding to the image to be identified; The character position information in the text information is analyzed based on the target feature map to obtain the target character features corresponding to the image to be identified; Based on the target character features, the text recognition result in the image to be recognized is determined.

2. The method according to claim 1, wherein analyzing the character position information in the text information based on the target feature map to obtain the target character features corresponding to the image to be recognized includes: The target feature map is input into the feature decoding model, which analyzes the character position information in the text information to obtain the target character features corresponding to the image to be recognized.

3. The method according to claim 2, wherein the feature decoding model includes a location relationship analysis unit and a positioning information analysis unit; The step of inputting the target feature map into a feature decoding model, and then analyzing the character position information in the text information through the feature decoding model to obtain the target character features corresponding to the image to be recognized, includes: The target feature map is input into the positional relationship analysis unit, which analyzes the positional relationships between the characters to obtain a character nearest neighbor matrix. The target feature map is input into the positioning information analysis unit, which analyzes the character positioning information to obtain the first character attention map. The target character features are determined based on the target feature map, the character nearest neighbor matrix, and the first character attention map.

4. The method according to claim 3, wherein the feature decoding model further comprises a character alignment unit; The step of determining the target character features based on the target feature map, the character nearest neighbor matrix, and the first character attention map includes: The character nearest neighbor matrix and the first character attention map are input into the character alignment unit to obtain the character attention map of each character in the text information; The target character features are determined based on the target feature map and the character attention map of each character.

5. The method according to claim 4, wherein inputting the character nearest neighbor matrix and the first character attention map into the character alignment unit to obtain the character attention map of each character in the text information includes: The character nearest neighbor matrix and the first character attention map are input into the character alignment unit. In the character alignment unit, the second character attention map of the second character is determined based on the character nearest neighbor matrix and the first character attention map of the first character. The first character is the character whose character attention map has been determined, and the second character is the next character after the first character.

6. The method according to claim 5, further comprising, before determining the second character attention map of the second character based on the character nearest neighbor matrix and the first character attention map of the first character: The elements in the first character attention map are sharpened to obtain the updated first character attention map; The step of determining the second character attention map of the second character based on the character nearest neighbor matrix and the first character attention map of the first character includes: Based on the character nearest neighbor matrix and the updated first character attention map, determine the second character attention map.

7. The method according to claim 1, further comprising, after analyzing the character position information in the text information based on the target feature map to obtain the target character features corresponding to the image to be recognized: Based on the target character features, the target feature map is enhanced to obtain an enhanced target feature map, and the process returns to the step of analyzing the character position information in the text information based on the target feature map to obtain the target character features corresponding to the image to be recognized, until the first preset stop condition is reached and the updated target character features are obtained. The step of determining the text recognition result in the image to be recognized based on the target character features includes: Based on the updated target character features, the text recognition result in the image to be recognized is determined.

8. The method according to claim 7, wherein enhancing the target feature map based on the target character features to obtain an enhanced target feature map comprises: Perform inter-character feature relationship analysis on the target character features to obtain global character features; Contextual information analysis is performed on the global character features and the target feature map to obtain the enhanced target feature map.

9. The method according to claim 2, wherein the training method of the feature decoding model includes: Obtain a sample set, wherein the sample set includes multiple sample images carrying sample text labels, and the sample images include sample text information; Extract a first sample image from the sample set, wherein the first sample image is any sample image in the sample set; Feature extraction is performed on the first sample image to obtain the first sample feature map corresponding to the first sample image; The first sample feature map is input into the feature decoding model to obtain the first sample character features corresponding to the first sample image; Based on the character features of the first sample, determine the first predicted text corresponding to the first sample image; Calculate the training loss value based on the first predicted text and the first sample text label carried by the first sample image; Based on the training loss value, adjust the model parameters of the feature decoding model, and return to the step of extracting the first sample image from the sample set until the second preset stopping condition is met, thereby obtaining the trained feature decoding model.

10. The method according to claim 9, wherein the feature decoding model comprises a positional relationship analysis unit, a positioning information analysis unit, and a character alignment unit; The step of inputting the first sample feature map into the feature decoding model to obtain the first sample character features corresponding to the first sample image includes: The first sample feature map is input into the positional relationship analysis unit, and the positional relationship analysis unit analyzes the positional relationship between characters in the first sample text information to obtain the first sample character nearest neighbor matrix; The first sample feature map is input into the positioning information analysis unit, and the positioning information analysis unit analyzes the character positioning information in the first sample text information to obtain the first sample first character attention map; Input the first sample character nearest neighbor matrix and the first sample first character attention map into the character alignment unit to obtain the first sample character attention map of each character in the first sample text information; The first sample character features are determined based on the first sample feature map and the first sample character attention map of each character.

11. The method according to claim 10, further comprising, before adjusting the model parameters of the feature decoding model based on the training loss value: Calculate the attention map loss value based on the first sample character attention map of each character; The step of adjusting the model parameters of the feature decoding model based on the training loss value includes: The model parameters of the feature decoding model are adjusted based on the training loss value and the attention map loss value.

12. A feature decoding model training method, applied to cloud-side devices, comprising: Obtain a sample set, wherein the sample set includes multiple sample images carrying sample text labels, the sample images include sample text information, the sample text information includes the positional relationship between characters and the character positioning information, the positional relationship between characters includes the positional adjacency relationship between each character feature in the text information, and the character positioning information includes the positional information of the character feature in the text information; Extract a first sample image from the sample set, wherein the first sample image is any sample image in the sample set; Feature extraction is performed on the first sample image to obtain the first sample feature map corresponding to the first sample image; The first sample feature map is input into the feature decoding model to obtain the first sample character features corresponding to the first sample image; Based on the character features of the first sample, determine the first predicted text corresponding to the first sample image; Calculate the training loss value based on the first predicted text and the first sample text label carried by the first sample image; Based on the training loss value, adjust the model parameters of the feature decoding model, and return to the step of extracting the first sample image from the sample set until the second preset stopping condition is reached, and obtain the model parameters of the trained feature decoding model. Send the model parameters of the trained feature decoding model to the edge device.

13. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 11 or claim 12.

14. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of any one of claims 1 to 11 or the method of claim 12.

Images