Model recognition result positioning method and device, electronic device, and computer medium

Abstract

The present disclosure provides a model recognition result positioning method and device, and relates to the technical fields of image recognition, deep learning and the like. A specific embodiment of the method comprises: adopting a sequence recognition model to recognize text line materials to obtain an encoding sequence corresponding to the text line materials output by the sequence recognition model; detecting whether the position of the encoding needs to be updated based on the content of each encoding in the encoding sequence; and in response to the detection result being that the position of the encoding needs to be updated, calculating and storing the updated position of the encoding based on the position of the encoding in the encoding sequence and the sequence recognition model, the updated position being the position of each text in the text line materials. The embodiment achieves positioning of the model recognition result.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, specifically to the fields of image recognition, deep learning, and other related technologies, and in particular to methods and apparatuses for locating model recognition results, electronic devices, computer-readable media, and computer program products. Background Technology

[0002] In CTC decoding algorithms, the greedy search decoding method is widely used. This method selects the output value with the highest probability at each step of the text sequence output by the text sequence recognition network to obtain the final decoded text sequence. However, since the text sequence recognition network cannot label the actual position of each character in the source material, it's impossible to know from which part of the source material the greedy search decoding method identified the characters in the text sequence. Summary of the Invention

[0003] Embodiments of this disclosure provide methods and apparatus for locating model recognition results, electronic devices, computer-readable media, and computer program products.

[0004] In a first aspect, embodiments of this disclosure provide a method for locating model recognition results. The method includes: using a sequence recognition model to recognize text line material to obtain an encoding sequence output by the sequence recognition model corresponding to the text line material; detecting whether the position of the encoding needs to be updated based on the content of each encoding in the encoding sequence; and in response to the detection result indicating that the position of the encoding needs to be updated, calculating and storing the updated position of the encoding based on the position of the encoding in the encoding sequence and the sequence recognition model, wherein the updated position is the position of each text in the text line material.

[0005] In some embodiments, based on the content of each code in the encoding sequence, detecting whether the position of the code needs to be updated includes the following steps: S1: Detecting whether the content of the current code in the encoding sequence is the same as the content of the previous code adjacent to the current code; S2: In response to the detection result that the content of the current code is different from the content of the previous code adjacent to the current code, determining that the position of the current code needs to be updated.

[0006] In some embodiments, the above-mentioned detection of whether the position of the encoding needs to be updated based on the content of each encoding in the encoding sequence further includes: in response to the detection result that the content of the current encoding is the same as the content of the previous encoding adjacent to the current encoding, taking the next encoding adjacent to the current encoding as the current encoding, and continuing to execute step S1.

[0007] In some embodiments, the above-mentioned detection of whether the position of the encoding needs to be updated based on the content of each encoding in the encoding sequence further includes: detecting whether the content of the current encoding in the encoding sequence is a preset termination symbol; in response to detecting that the content of the current encoding is not a preset termination symbol, taking the next encoding adjacent to the current encoding as the current encoding and continuing to execute step S1.

[0008] In some embodiments, the above-mentioned detection of whether the position of the encoding needs to be updated based on the content of each encoding in the encoding sequence further includes: detecting whether the content of the current encoding in the encoding sequence is a preset isolation label; in response to the detection result that the content of the current encoding is a preset isolation label, taking the next encoding adjacent to the current encoding as the current encoding, and continuing to execute step S1.

[0009] In some embodiments, the above method further includes: using a greedy search algorithm to predict the encoded sequence to obtain the text sequence corresponding to the encoded sequence; and setting each text in the text sequence to correspond one-to-one with each update position.

[0010] In some embodiments, the above-mentioned calculation of the updated position of the code based on the position of the code in the encoded sequence and the sequence recognition model includes: inputting the position of the code in the encoded sequence into the position update formula to obtain the updated position of the current code; the position update formula is: L=(i+0.5)*t / s; where L represents the updated position, t represents the compression ratio of the sequence recognition model to the image width, i represents the position of the current code in the encoded sequence, and s represents the hyperparameter.

[0011] In some embodiments, the sequence recognition model includes: an image feature extraction network and a text sequence extraction network; the text line material includes: at least one text line image with the same height; the above-mentioned use of the sequence recognition model to recognize the text line material and obtain the encoding sequence output by the sequence recognition model corresponding to the text line of the text line material includes: inputting at least one text line image with the same height into the image feature extraction network to obtain the image feature vector sequence of each text line image; inputting each image feature vector sequence into the text sequence extraction network to obtain the encoding sequence corresponding to the text line of the text line image.

[0012] Secondly, embodiments of this disclosure provide a model recognition result localization device, which includes: a recognition unit configured to recognize text line material using a sequence recognition model to obtain an encoding sequence output by the sequence recognition model corresponding to the text line material; a detection unit configured to detect whether the position of the encoding needs to be updated based on the content of each encoding in the encoding sequence; and a calculation unit configured to, in response to the detection result indicating that the position of the encoding needs to be updated, calculate and store the updated position of the encoding based on the position of the encoding in the encoding sequence and the sequence recognition model, wherein the updated position is the position of each text in the text line material.

[0013] In some embodiments, the detection unit includes: an adjacent detection module configured to detect whether the content of the current code in the encoding sequence is the same as the content of the previous code adjacent to the current code; and a determination module configured to determine the position where the current code needs to be updated in response to the detection result that the content of the current code is different from the content of the previous code adjacent to the current code.

[0014] In some embodiments, the detection unit further includes: a same detection module, configured to, in response to the detection result that the content of the current code is the same as the content of the previous code adjacent to the current code, take the next code adjacent to the current code as the current code, and the adjacent detection module continues to work.

[0015] In some embodiments, the detection unit further includes: a termination detection module configured to detect whether the content of the current code in the encoding sequence is a preset termination symbol; and a termination cursor module configured to, in response to detecting that the content of the current code is not a preset termination symbol, take the next code adjacent to the current code as the current code, and the adjacent detection module continues to work.

[0016] In some embodiments, the detection unit further includes: an isolation detection module configured to detect whether the content of the current code in the encoding sequence is a preset isolation label; and an isolation cursor module configured to, in response to the detection result that the content of the current code is a preset isolation label, take the next code adjacent to the current code as the current code, and the adjacent detection module continues to work.

[0017] In some embodiments, the above apparatus further includes: a obtaining unit configured to use a greedy search algorithm to predict the encoded sequence to obtain a text sequence corresponding to the encoded sequence; and a setting unit configured to set a one-to-one correspondence between each text in the text sequence and each update position.

[0018] In some embodiments, the above-mentioned calculation unit is further configured to input the position of the encoding in the encoding sequence into the position update formula to obtain the updated position of the encoding; the position update formula is: L=(i+0.5)*t / s; where L represents the updated position, t represents the compression ratio of the sequence recognition model to the image width, i represents the current position of the encoding in the encoding sequence, and s represents the hyperparameter.

[0019] In some embodiments, the sequence recognition model includes an image feature extraction network and a text sequence extraction network; the text line material includes at least one text line image with the same height; the recognition unit includes a feature input unit configured to input at least one text line image with the same height into the image feature extraction network to obtain an image feature vector sequence for each text line image; and an encoding input unit configured to input each image feature vector sequence into the text sequence extraction network to obtain an encoding sequence corresponding to the text line of the text line image.

[0020] Thirdly, embodiments of this disclosure provide an electronic device comprising: one or more processors; a storage device having one or more programs stored thereon; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the method described in any implementation of the first aspect.

[0021] Fourthly, embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon that, when executed by a processor, implements the method as described in any implementation of the first aspect.

[0022] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the method described in any implementation of the first aspect.

[0023] The model recognition result localization method and apparatus provided in the embodiments of this disclosure first use a sequence recognition model to recognize text line material, obtaining the encoded sequence output by the sequence recognition model corresponding to the text line material; secondly, based on the content of each code in the encoded sequence, it is detected whether the position of the code needs to be updated; finally, in response to the detection result indicating that the position of the code needs to be updated, based on the position of the code in the encoded sequence and the sequence recognition model, the updated position of the code is calculated and stored, where the updated position is the position of each text in the text line material. Thus, by detecting whether the position of the code needs to be updated through the content of each code in the encoded sequence, and calculating the updated position when an update is needed, the updated position is related to the sequence recognition model and the position of the code in the encoded sequence, thereby locating the actual position corresponding to each code in the encoded sequence, providing auxiliary localization interpretation for the recognition results of the sequence recognition model. Attached Figure Description

[0024] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0025] Figure 1 This is an exemplary system architecture diagram to which one embodiment of this disclosure can be applied;

[0026] Figure 2 This is a flowchart of an embodiment of the model recognition result localization method based on the present disclosure;

[0027] Figure 3 This is a flowchart of another embodiment of the model recognition result localization method based on this disclosure;

[0028] Figure 4 This is a schematic diagram illustrating the use of model recognition result localization method to predict text line images in this disclosure;

[0029] Figure 5 This is a schematic diagram of an embodiment of the positioning device based on the model recognition result of this disclosure;

[0030] Figure 6 This is a schematic diagram of the structure of an electronic device suitable for implementing embodiments of the present disclosure. Detailed Implementation

[0031] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0032] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0033] Figure 1 An exemplary system architecture 100 is shown in which the model recognition result localization method of this disclosure can be applied.

[0034] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, typically including wireless communication links, etc.

[0035] Terminal devices 101, 102, and 103 interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications, such as instant messaging tools and email clients, can be installed on terminal devices 101, 102, and 103.

[0036] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be terminal devices with communication and control functions, or terminals or autonomous mobile robots equipped with sequence recognition models. These devices can communicate with the sequence recognition models of the terminals or autonomous mobile robots. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned terminals or autonomous mobile robots. They can be implemented as multiple software programs or software modules (e.g., software or software modules used to implement the sequence recognition model) or as a single software program or software module. No specific limitations are made here.

[0037] Server 105 can be a server that provides various services, such as a model recognition result localization server that supports sequence recognition models on terminal devices 101, 102, and 103. The model recognition result localization server can analyze and process relevant information of various terminals or autonomous mobile robots in the network and feed back the processing results (such as location sets, updated locations, etc.) to the terminal devices.

[0038] It should be noted that a server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules (such as software programs or software modules used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0039] It should be noted that the model recognition result localization method provided in the embodiments of this disclosure is generally executed by server 105.

[0040] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0041] like Figure 2 The flowchart 200 illustrates an embodiment of the model recognition result localization method according to the present disclosure, the model recognition result localization method comprising:

[0042] Step 201: Use a sequence recognition model to recognize the text line material and obtain the encoded sequence output by the sequence recognition model that corresponds to the text line material.

[0043] In this embodiment, text line material refers to material containing multiple text lines. The material itself is not limited; for example, the material can be an image that includes multiple text lines. Alternatively, the material can be speech that can be converted into multiple text lines through audio conversion.

[0044] Sequence recognition models are encoded sequences obtained by encoding different task sequences (e.g., text sequences in images, text sequences corresponding to speech) in text materials. Each code in the encoded sequence corresponds to a text sequence in the task sequence. To reconstruct the text sequence from the encoded sequence, a decoder such as CTC (Connection-Temporal Classification) is needed to decode the encoded sequence. However, generally, a sequence recognition model is combined with CTC. This only yields the overall recognition result of the text sequence after CTC decoding. Since the sequence recognition model is a black box model, it cannot know which images or speech samples CTC used to recognize the text, i.e., it cannot locate the actual position of each text sample in the image or speech sequence.

[0045] In this embodiment, the sequence recognition model can be a trained model, such as a trained text recognition model used to recognize text sequences in images and speech. When a text line image is input into the text recognition model, the encoded sequence output by the text recognition model corresponds to the text sequence in the text line image. By using the model recognition result localization method of this embodiment, the updated position of each text in the position set of the text sequence can be obtained. The recognition result of the model can be visualized on the original image, so that users can understand the specific position of each text in real time.

[0046] Optionally, the sequence recognition model can also be an untrained model. For example, the sequence recognition model can be an untrained text recognition model. After inputting a text line sample into the sequence recognition model, the encoded sequence output by the sequence recognition model corresponds to the text in the text line sample. The updated position in the position set obtained by the model recognition result localization method in this embodiment can correctly represent the position of the text in the text line sample. This can guide the sequence recognition model to make better parameter adjustments, and explain from a visual perspective where the sequence recognition model makes predictions about the sequence. It can also help the sequence recognition model explain some failed examples.

[0047] The model recognition result localization method runs on the execution entity (such as...) Figure 1 The server 105 shown can obtain the encoded sequence in various ways, for example, by acquiring the user terminal (e.g., ...) in real time. Figure 1The encoded sequence output by the sequence recognition model on the terminal devices 101, 102, and 103 shown; or, for example, obtaining the encoded sequence of the sequence recognition model pre-stored in the data server.

[0048] Step 202: Based on the content of each code in the encoding sequence, detect whether the position of the code needs to be updated.

[0049] In this embodiment, the sequence recognition model can only encode the task and cannot locate each text in the input task sequence. The encoded sequence corresponds to the task sequence input to the sequence recognition model. The order of each code in the encoded sequence generally corresponds to the order of the text sequence in the task sequence. In order to locate the task sequence, the content of the code in the encoded sequence can be detected to determine whether the position of the code in the encoded sequence needs to be updated.

[0050] In this embodiment, the task sequence output to the sequence recognition model is a text line image. The text line image contains a text line, and the encoding sequence corresponds to the text line. Each encoding in the encoding sequence has a corresponding text in the text line, and the update position of the encoding can be equal to the position of the text in the text line image, or correspond to the position of the text in the text line image.

[0051] Step 203: In response to the detection result indicating that the code needs to be updated, the update position of the code is calculated and stored based on the position of the code in the code sequence and the sequence recognition model.

[0052] The update position refers to the position of each text element within the text line material.

[0053] In this embodiment, the update positions of the encoding can be stored sequentially in a preset position set according to the order of each encoding in the encoding sequence. The preset position set is a collection of storage positions, including at least one update position. The region of each result in the sequence recognition result can be determined by each update position. For example, if the recognition result corresponding to the encoding sequence is a text sequence, then the position set stores the position of each text in the text sequence in the image, and the region of each text in the image is determined by this position.

[0054] In this embodiment, the encoding sequence contains multiple codes. In order to accurately locate the recognition result of the sequence recognition model, the update positions of all codes in the entire encoding sequence are stored sequentially into a preset position set, which can locate the actual position of each code in the encoding sequence.

[0055] Optionally, in response to a detection result indicating that the position does not require updating the encoding, the system continues to detect whether the position needs updating the encoding based on the content of the next encoding in the encoding sequence, until all encodings in the encoding sequence have been traversed.

[0056] The model recognition result localization method provided in this embodiment first uses a sequence recognition model to recognize text line material, obtaining the encoding sequence output by the sequence recognition model corresponding to the text line material. Second, based on the content of each encoding in the encoding sequence, it detects whether the position of the encoding needs to be updated. Finally, in response to the detection result indicating that the position of the encoding needs to be updated, the update position of the encoding is calculated and stored based on the position of the encoding in the encoding sequence and the sequence recognition model. Thus, by detecting whether the position of the encoding needs to be updated through the content of each encoding in the encoding sequence, and calculating the update position of the encoding when the position of the encoding needs to be updated, the update position is related to the sequence recognition model and the position of the encoding in the encoding sequence, thereby locating the actual position corresponding to each encoding in the encoding sequence, providing auxiliary localization interpretation for the recognition result of the sequence recognition model.

[0057] In some optional implementations of this embodiment, the above-mentioned detection of whether the position of the encoding needs to be updated based on the content of each encoding in the encoding sequence includes the following steps:

[0058] Step S1: Detect whether the content of the current code in the encoding sequence is the same as the content of the previous code adjacent to the current code; when the detection result is that the content of the current code is not the same as the content of the previous code adjacent to the current code, proceed to step S2.

[0059] Step S2: Determine the location where the current encoding needs to be updated.

[0060] In this embodiment, by detecting whether the content of the current code is the same as the content of the previous code adjacent to the current code, it can be determined whether the current code and the previous code are the same code. If they are not the same code, the position of the current code in the actual task sequence (the initial sequence input into the sequence recognition model) can be calculated through the transformation parameters of the sequence recognition model, and the current code can be accurately located.

[0061] Generally, when a sequence recognition model identifies a task sequence and obtains an encoded sequence, if there is duplicate content in the task sequence, the sequence recognition model will distinguish the duplicate content (for example, by adding a blank label). This is reflected in the encoded sequence as each code in the encoded sequence is different. Therefore, when the content of the current code is the same as the content of the previous code adjacent to the current code, it indicates that the current code is an abnormal code. There is no need to locate the actual position of the current code; instead, the position of the next code adjacent to the current code can be located directly.

[0062] In this optional implementation, the recognition content of the model recognition result is distinguished by detecting whether the content of the current code and the previous code adjacent to the current code are the same. When the content of the current code and the previous code adjacent to the current code are different, the update position of the current code that needs to be updated is determined. The update position can be related to the width compression degree of the sequence recognition model, thereby locating the actual position corresponding to the current code in the encoding sequence, and providing auxiliary localization interpretation for the recognition result of the sequence recognition model.

[0063] In some optional implementations of this embodiment, the above-mentioned detection of whether the position of the encoding needs to be updated based on the content of each encoding in the encoding sequence further includes: in response to the detection result that the content of the current encoding is the same as the content of the previous encoding adjacent to the current encoding, taking the next encoding adjacent to the current encoding as the current encoding, and continuing to execute the above step S1.

[0064] In this optional implementation, the recognition content of the model recognition result is distinguished by detecting whether the content of the current code and the previous code adjacent to the current code are the same. When the content of the current code and the previous code adjacent to the current code are the same, the next code adjacent to the current code is taken as the current code, and the current code is detected again, thus providing a reliable foundation for the sequential detection task sequence.

[0065] In some optional implementations of this embodiment, the above-mentioned detection of whether the position of the encoding needs to be updated based on the content of each encoding in the encoding sequence further includes: detecting whether the content of the current encoding in the encoding sequence is a preset termination symbol; in response to detecting that the content of the current encoding is not a preset termination symbol, taking the next encoding adjacent to the current encoding as the current encoding, and then continuing to execute step S1.

[0066] In this embodiment, after the sequence recognition model has completed the recognition of the task sequence, a termination symbol is generally added to the encoded sequence to indicate that the current task sequence has been recognized.

[0067] In this embodiment, when the content of the current encoding is not the preset termination symbol for the sequence recognition model, it means that the detection of the encoding sequence has not yet reached the last encoding, and it is necessary to continue to judge the encoding in the encoding sequence. That is, the next encoding adjacent to the current encoding is taken as the current encoding, and the detection of whether the content of the current encoding is the same as the content of the previous encoding adjacent to the current encoding continues.

[0068] In this optional implementation, when the content of the current encoding is not the preset termination symbol, the next encoding adjacent to the current encoding can be used as the current encoding to effectively cycle through the encoding sequence, providing a reliable basis for traversing the entire encoding sequence.

[0069] Optionally, when the content of the current encoding is a preset termination symbol of the sequence recognition model, it is determined that the current encoding sequence has been located, and there is no need to judge the encoding content again. In some optional implementations of this embodiment, when the content of the current encoding is a preset termination symbol, the location of the encoding sequence is stopped. Optionally, in one embodiment of this disclosure, the model recognition result location method further includes: when the content of the current encoding in the encoding sequence is different from the content of the previous encoding adjacent to the current encoding, detecting that the content of the current encoding is not a preset termination symbol; in response to detecting that the content of the current encoding is not a preset termination symbol, obtaining the updated position of the current encoding based on the position of the current encoding in the encoding sequence, storing the updated position in a preset position set in sequence; when the content of the current encoding is not a preset termination symbol, taking the next encoding adjacent to the current encoding as the current encoding, and then continuing to check whether the content of the current encoding in the encoding sequence is the same as the content of the previous encoding adjacent to the current encoding; in response to detecting that the content of the current encoding is a preset termination symbol, stopping the model recognition result location.

[0070] In some optional implementations of this embodiment, detecting whether the position of the encoding needs to be updated based on the content of each encoding in the encoding sequence further includes: detecting whether the content of the current encoding in the encoding sequence is a preset isolation label; in response to the detection result that the content of the current encoding is a preset isolation label, taking the next encoding adjacent to the current encoding as the current encoding, and continuing to execute step S1.

[0071] In embodiments of this disclosure, such as Figure 3 The flowchart 300 shows another embodiment of the model recognition result localization method according to the present disclosure, which includes the following steps:

[0072] Step 301: Obtain the encoded sequence output by the sequence recognition model.

[0073] In this embodiment, the encoded sequence can be a sequence of text lines obtained by a sequence recognition model recognizing text lines. Therefore, a sequence model can be used to recognize text lines and obtain the encoded sequence output by the sequence recognition model.

[0074] Step 302: Detect whether the currently encoded content in the encoded sequence is a preset isolation label; if the currently encoded content is a preset isolation label, proceed to step 303. If the currently encoded content is not a preset isolation label, proceed to step 304.

[0075] In this embodiment, the isolation label is a code set by the sequence recognition model in the middle of the same content in order to distinguish the task sequence. Therefore, there may be multiple isolation labels with intervals set between different codes in the encoded sequence.

[0076] The model recognition result localization method operates on an execution entity that detects the content of each code in the encoding sequence in real time. When the content of a code is exactly the same as the preset isolation label, there is no need to perform any position update processing on the code, and the next code adjacent to the code in the encoding sequence is directly detected.

[0077] Step 303: Take the next code adjacent to the current code as the current code, and then execute step 304.

[0078] Step 304: Detect whether the content of the current code in the encoding sequence is the same as the content of the previous code adjacent to the current code; if the detection result is that the content of the current code is not the same as the content of the previous code adjacent to the current code, proceed to step 305. If the detection result is that the content of the current code is the same as the content of the previous code adjacent to the current code, proceed to step 303.

[0079] Step 305: Calculate the updated position of the current code based on its position in the encoded sequence and the sequence recognition model. Then, proceed to step 306.

[0080] In this embodiment, the update position of the current encoding is the position of the text corresponding to the current encoding in the text material.

[0081] Step 306: Store the updated positions in a preset position set in order, and then execute step 307.

[0082] Step 307: Detect whether the content of the current encoding is a preset termination symbol; if the content of the current encoding is not a preset termination symbol, proceed to step 308. If the content of the current encoding is a preset execution symbol, proceed to step 309.

[0083] Step 308: Take the next code adjacent to the current code as the current code, and then continue to execute step 302.

[0084] Step 309, End.

[0085] The model recognition result localization method provided in this embodiment takes the next code adjacent to the current code as the current code when the content of the current code is a preset isolation label or the content of the current code is the same as the content of the previous code adjacent to the current code. This ensures that the location of the code is not located for the code that is the content of the isolation label or the code with incorrect content, thus guaranteeing the effectiveness of the model recognition result localization.

[0086] Optionally, when the content of the current encoding is detected to be not a preset isolation label, the above model recognition result localization method further includes: detecting whether the content of the current encoding is a preset termination symbol; in response to detecting that the content of the current encoding is not a preset termination symbol, detecting whether the content of the current encoding in the encoding sequence is the same as the content of the previous encoding adjacent to the current encoding; when the detection result is that the content of the current encoding is not the same as the content of the previous encoding adjacent to the current encoding, obtaining the updated position of the current encoding based on the position of the current encoding in the encoding sequence, storing the updated position in a preset position set in order, taking the next encoding adjacent to the current encoding as the current encoding, and then continuing to detect whether the content of the current encoding is a preset isolation label.

[0087] In some optional implementations of this embodiment, the above model recognition result localization method further includes: using a greedy search algorithm to predict the encoded sequence to obtain the text sequence corresponding to the encoded sequence; and setting a one-to-one correspondence between each text in the text sequence and each update position.

[0088] In this optional implementation, the task sequence is input into the sequence model, and the encoded sequence output by the sequence model is obtained, such as... Figure 4 As shown, the task sequence is text line image 1, which contains a text sequence "Miss Jeranan". One letter in this text sequence is the text of the text sequence.

[0089] In this optional implementation, the encoded sequence 2 output by the sequence model can be processed in two branches. One branch is to use the model recognition result localization method described in the above embodiment to obtain the position in the position set, and determine the text visualization region based on each updated position 3 in the position set. The other branch is to use a greedy search algorithm to predict the encoded sequence 2 to obtain the text sequence 4 corresponding to the encoded sequence (e.g., ...). Figure 4 The "Miss Jeranan" in the text sequence 4 is set to correspond one-to-one with the update position 3, which makes it easier to analyze the visual text area corresponding to each text in the text sequence.

[0090] In this optional implementation, the greedy search algorithm selects the output value with the highest probability at each step of the output sequence in the network to obtain the final decoded output sequence. By visualizing the CTC network output on the original image and combining it with the final text recognition result after CTC decoding, auxiliary interpretation of the text recognition result can be achieved.

[0091] In this optional implementation, the model recognition result localization method provided in this embodiment is applied to the text sequence recognition of the greedy search algorithm, providing reliable location guidance for text localization in the text sequence.

[0092] In a specific example, the current encoding position can be changed by moving the cursor. After CTC decoding, the encoded sequence yields the final recognized text sequence. The model recognition result localization method disclosed herein, while performing CTC decoding, simulates the CTC decoding algorithm process based on the encoded sequence extracted from the sequence model, and visualizes each extracted valid text region. The specific steps are as follows:

[0093] Step 1: Obtain the encoded sequence T output by the loop layer. The encoded sequence T includes at least one code.

[0094] Step 2: Initialize the cursor index = 1.

[0095] a) Traverse the encoded sequence T.

[0096] If the encoded content T[index] corresponding to the current sequence value index is the isolation label blank, that is, T[index] == <blank>At this point, no operation is performed, and the cursor is incremented by 1, i.e., index = index + 1; then return to a).

[0097] If the encoded content T[index] corresponding to the current sequence value index is not equal to the label T[index-1] corresponding to the previous sequence value, i.e., T[index] != T[index-1], update the position of the current sequence value, obtaining the updated position l = (index+0.5)*CNN_STRIDE / scale, where 0.5 is the position offset, CNN_STRIDE is the compression ratio of the image width after passing through the sequence model, and scale is the ratio of the width of the adjusted image to the original image when inputting into the sequence model. Then, store the updated position l corresponding to the current sequence value into the position set L, increment the cursor by 1, index = index+1, and return to a).

[0098] If the label T[index] corresponding to the current sequence value index is equal to the label T[index-1] corresponding to the previous sequence value, that is, T[index] == T[index-1], no operation is performed, the cursor is incremented by 1, index = index + 1, and the process returns to a).

[0099] Step 3: Traverse each updated position in the position set L of the encoded sequence T, and visualize each updated position in the position set L.

[0100] When the input text material of the sequence model includes at least one image, after fixing each image at the same height, the actual position in the image corresponding to each code in the encoding sequence can be obtained based on the image processing results of the sequence model. The actual position can be used to determine the actual update position of the code that needs to be updated.

[0101] In some optional implementations of this embodiment, the above-mentioned calculation of the updated position of the code based on the position of the code in the encoded sequence and the sequence recognition model includes: inputting the position of the code in the encoded sequence into the position update formula to obtain the updated position of the code; the position update formula is: L=(i+0.5)*t / s; where L represents the updated position, t represents the compression ratio of the sequence recognition model to the image width, i represents the current position of the code in the encoded sequence, and s represents the hyperparameter.

[0102] In this optional implementation, the position update formula is determined by the compression ratio of the image width by the sequence recognition model. The sequence recognition model is used to recognize the image and obtain the sequence display of the content in the image.

[0103] In this optional implementation, by inputting the position of the current code in the encoding sequence into the position update formula, the updated position of the current code can be effectively obtained. The updated position can determine the actual position of the text in the image corresponding to the current code, thus providing a reliable means for locating the model recognition results.

[0104] When the task sequence is an image sequence, the sequence recognition model can be implemented using an image feature extraction network and a text sequence extraction network. In some optional implementations of this embodiment, the above-mentioned sequence recognition model includes: an image feature extraction network and a text sequence extraction network; the text line material includes: at least one text line image with the same height. The above-mentioned use of the sequence recognition model to recognize the text line material and obtain the encoding sequence output by the sequence recognition model corresponding to the text line of the text line material includes: inputting at least one text line image with the same height into the image feature extraction network to obtain the image feature vector sequence of each text line image; inputting each image feature vector sequence into the text sequence extraction network to obtain the encoding sequence corresponding to the text line of the text line image.

[0105] In this optional implementation, before extracting the image feature vector sequence from multiple text line images 1, such as Figure 4 As shown, it is necessary to adjust each text line image 1 to the same height, and then obtain the image feature vector sequence of each text line image 1 through the image feature extraction network.

[0106] In this optional implementation, the image feature extraction network has convolutional layers, max pooling layers, and activation function layers. The image feature extraction network arranges each feature vector in the image feature vector sequence in order from left to right. The height of each image feature vector is set to a single pixel. With the action of the convolutional layer, max pooling layer, and activation function, each column of text in each text line image corresponds to a rectangular region of the text line image.

[0107] In this optional implementation, the text sequence extraction network converts the predictions for each rectangular region into encoded sequences, thus completing the prediction of the text sequence. Specifically, the final encoded sequence can be found based on the maximum predicted probability for each bounding box.

[0108] In this optional implementation, the image feature extraction network can be a convolutional neural network, and the text sequence extraction network can also be a convolutional neural network. Specifically, the text sequence extraction network can also be an LSTM (Long Short-Term Memory) network. LSTM is a temporal recurrent neural network, suitable for processing and predicting important events with relatively long intervals and delays in time series.

[0109] In this optional implementation, a sequence recognition model composed of an image feature extraction network and a text sequence extraction network is used to recognize the task sequence, thereby obtaining the encoded sequence of the text sequence corresponding to the task sequence. The image feature extraction network extracts the features of the image text line image, and the text sequence extraction network extracts the sequence features of the image feature vector sequence output by the image feature extraction network, thus improving the reliability of the encoded sequence.

[0110] Further reference Figure 5 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a model recognition result localization device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0111] like Figure 5 As shown, an embodiment of this disclosure provides a model recognition result localization device 500, which includes: a recognition unit 501, a detection unit 502, and a calculation unit 503. The recognition unit 501 can be configured to use a sequence recognition model to recognize text line material, obtaining an encoded sequence output by the sequence recognition model corresponding to the text lines of the text line material. The detection unit 502 can be configured to detect whether the position of the encoded code needs to be updated based on the content of each encoded code in the encoded sequence. The calculation unit 503 can be configured to, in response to the detection result indicating that the position of the encoded code needs to be updated, calculate and store the updated position of the encoded code based on the position of the encoded code in the encoded sequence and the sequence recognition model; the updated position is the position of each text element in the text line material.

[0112] In this embodiment, the specific processing of the recognition unit 501, detection unit 502, and calculation unit 503 in the model recognition result localization device 500, and the resulting technical effects, can be referred to respectively. Figure 2 Steps 201, 202, and 203 in the corresponding embodiment.

[0113] In some embodiments, the detection unit 502 includes: an adjacent detection module (not shown in the figure) and a determination module (not shown in the figure). The adjacent detection module can be configured to detect whether the content of the current code in the encoding sequence is the same as the content of the previous code adjacent to the current code. The determination module can be configured to determine the position where the current code needs to be updated in response to the detection result that the content of the current code is different from the content of the previous code adjacent to the current code.

[0114] In some embodiments, the detection unit 502 further includes a same detection module (not shown in the figure). The same detection module is configured to, in response to the detection result that the content of the current code is the same as the content of the previous code adjacent to the current code, take the next code adjacent to the current code as the current code, and the adjacent detection module continues to work.

[0115] In some embodiments, the detection unit 502 further includes a termination detection module (not shown in the figure) and a termination cursor module (not shown in the figure). The termination detection module can be configured to detect whether the content of the current code in the encoding sequence is a preset termination symbol. The termination cursor module can be configured to, in response to detecting that the content of the current code is not a preset termination symbol, take the next code adjacent to the current code as the current code, and the adjacent detection module continues to work.

[0116] In some embodiments, the detection unit further includes: an isolation detection module (not shown in the figure) and an isolation cursor module (not shown in the figure). The isolation detection module can be configured to detect whether the content of the current code in the encoding sequence is a preset isolation label. The isolation cursor module can be configured to, in response to the detection result indicating that the content of the current code is a preset isolation label, take the next adjacent code as the current code, and allow the adjacent detection module to continue operating.

[0117] In some embodiments, the model recognition result localization device 500 further includes: an obtaining unit (not shown in the figure) and a setting unit (not shown in the figure). The obtaining unit can be configured to use a greedy search algorithm to predict the encoded sequence to obtain the text sequence corresponding to the encoded sequence. The setting unit can be configured to set a one-to-one correspondence between each text in the text sequence and each update position.

[0118] In some embodiments, the above-mentioned calculation unit 503 is further configured to input the position of the code in the encoding sequence into the position update formula to obtain the updated position of the code; the position update formula is: L=(i+0.5)*t / s; where L represents the updated position, t represents the compression ratio of the sequence recognition model to the image width, i represents the position of the current code in the encoding sequence, and s represents the hyperparameter.

[0119] In some embodiments, the sequence recognition model includes an image feature extraction network and a text sequence extraction network; the text line material includes at least one text line image with the same height; the recognition unit 501 includes a feature input unit configured to input at least one text line image with the same height into the image feature extraction network to obtain an image feature vector sequence of each text line image; and an encoding input unit configured to input each image feature vector sequence into the text sequence extraction network to obtain an encoding sequence corresponding to the text line of the text line image.

[0120] The model recognition result localization device provided in the embodiments of this disclosure firstly identifies the text line material using a sequence recognition model, obtaining the encoded sequence output by the sequence recognition model corresponding to the text line material. Secondly, the detection unit 502 detects whether the position of the encoded code needs to be updated based on the content of each encoded code in the encoded sequence. Finally, the calculation unit 503, in response to the detection result indicating that the position of the encoded code needs to be updated, calculates and stores the updated position of the encoded code based on the position of the encoded code in the encoded sequence and the sequence recognition model. The updated position is the position of each text in the text line material. Thus, by detecting whether the position of the encoded code needs to be updated through the content of each encoded code in the encoded sequence, and calculating the updated position of the encoded code when an update is needed, the updated position is related to the sequence recognition model and the position of the encoded code in the encoded sequence, thereby locating the actual position corresponding to each encoded code in the encoded sequence, providing auxiliary localization interpretation for the recognition result of the sequence recognition model.

[0121] The following is for reference. Figure 6 It shows a schematic diagram of the structure of an electronic device 600 suitable for implementing embodiments of the present disclosure.

[0122] like Figure 6 As shown, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. Processing device 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0123] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 6 Each box shown can represent a device or multiple devices as needed.

[0124] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by a processing device 601, it performs the functions defined in the methods of embodiments of this disclosure.

[0125] It should be noted that the computer-readable medium in the embodiments of this disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (Radio Frequency), etc., or any suitable combination thereof.

[0126] The aforementioned computer-readable medium may be included in the aforementioned server; or it may exist independently and not assembled into the server. The aforementioned computer-readable medium carries one or more programs that, when executed by the server, cause the server to: use a sequence recognition model to recognize text lines, obtaining an encoded sequence output by the sequence recognition model corresponding to the text lines of the text line material; based on the content of each encoding in the encoded sequence, detect whether the position of the encoding needs to be updated; in response to the detection result indicating that the position of the encoding needs to be updated, calculate and store the updated position of the encoding based on the position of the encoding in the encoded sequence and the sequence recognition model, wherein the updated position is the position of each text element in the text line material.

[0127] Computer program code for performing the operations of embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0128] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0129] The units described in the embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, it can be described as: a processor including a recognition unit, a detection unit, and a calculation unit. The names of these units do not necessarily limit the unit itself; for example, the recognition unit can also be described as a unit "configured to recognize text line material using a sequence recognition model, and obtain an encoded sequence output by the sequence recognition model corresponding to the text line material".

[0130] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.< / blank>

Claims

1. A method for locating model recognition results, the method comprising: A sequence recognition model is used to identify text lines, and the encoded sequence output by the sequence recognition model corresponding to the text lines of the text line material is obtained. Based on the content of each code in the encoded sequence, detect whether the position of the code needs to be updated; In response to the detection result indicating that the position of the encoding needs to be updated, the update position of the encoding is calculated and stored based on the position of the encoding in the encoding sequence and the compression ratio of the sequence recognition model to the image width. The update position is the position of each text in the text line material.

2. The method according to claim 1, wherein, The step of detecting whether the position of the encoding needs to be updated based on the content of each encoding in the encoding sequence includes the following steps: S1: Detect whether the content of the current code in the encoding sequence is the same as the content of the previous code adjacent to the current code; S2: In response to the detection result that the content of the current code is different from the content of the previous code adjacent to the current code, determine the position where the current code needs to be updated.

3. The method according to claim 2, wherein, The step of detecting whether the position of the encoding needs to be updated based on the content of each encoding in the encoding sequence further includes: In response to the detection result that the content of the current code is the same as the content of the previous code adjacent to the current code, the next code adjacent to the current code is taken as the current code, and step S1 is continued.

4. The method according to claim 2, wherein, The step of detecting whether the position of the encoding needs to be updated based on the content of each encoding in the encoding sequence further includes: Detect whether the content currently encoded in the encoded sequence is a preset termination symbol; When it is detected that the content of the current encoding is not a preset termination symbol, the next encoding adjacent to the current encoding is taken as the current encoding, and step S1 is continued.

5. The method according to claim 2, wherein, The step of detecting whether the position of the encoding needs to be updated based on the content of each encoding in the encoding sequence further includes: Detect whether the currently encoded content in the encoded sequence is a preset isolation label; In response to the detection result indicating that the content of the current code is a preset isolation label, the next code adjacent to the current code is taken as the current code, and step S1 is continued.

6. The method according to any one of claims 1-5, further comprising: A greedy search algorithm is used to predict the encoded sequence to obtain the text sequence corresponding to the encoded sequence; Each text in the text sequence is assigned a one-to-one correspondence with each update position.

7. The method according to claim 1, wherein, The step of calculating the update position of the encoding based on its position in the encoded sequence and the compression ratio of the image width by the sequence recognition model includes: Input the position of the code in the encoding sequence into the position update formula to obtain the updated position of the code; The position update formula is: L = (i + 0.5) t / s; Where L represents the update position, t represents the compression ratio of the image width by the sequence recognition model, i represents the position of the current encoding in the encoding sequence, and s represents the hyperparameter.

8. The method according to claim 1, wherein, The sequence recognition model includes an image feature extraction network and a text sequence extraction network; the text line material includes at least one text line image with the same height. The step of using a sequence recognition model to identify text lines and obtaining the encoded sequence output by the sequence recognition model corresponding to the text lines of the text line material includes: The image feature extraction network is input into the at least one text line image with the same height to obtain the image feature vector sequence of each text line image; Each image feature vector sequence is input into the text sequence extraction network to obtain the encoding sequence corresponding to the text line of the text line image.

9. A model recognition result localization device, the device comprising: The recognition unit is configured to use a sequence recognition model to recognize the text line material and obtain the encoded sequence output by the sequence recognition model corresponding to the text line material. The detection unit is configured to detect whether the position of the encoding needs to be updated based on the content of each encoding in the encoding sequence. The calculation unit is configured to, in response to a detection result indicating that the encoding needs to be updated, calculate and store the updated position of the encoding based on the position of the encoding in the encoding sequence and the compression ratio of the sequence recognition model to the image width, wherein the updated position is the position of each text in the text line material.

10. An electronic device, comprising: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-8.

11. A computer-readable medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the method as described in any one of claims 1-8.

12. A computer program product comprising a computer program that, when executed by a processor, implements the method of any one of claims 1-8.

Images