Scene character recognition method based on single vector query decoding and related equipment

Abstract

The invention discloses a scene character recognition method based on single vector query decoding and related equipment. The method comprises the following steps: performing visual feature extraction on an image to obtain visual features; coding the character to be coded to obtain a character vector; performing global information fusion on the visual features and the character vectors to obtain context vectors; inputting the context vector and the visual feature into a single vector decoder for decoding to obtain a target decoded character; and if the characters in the image are not completely decoded, taking the target decoded character as a to-be-coded character, and returning to the step of coding the to-be-coded character by adopting the position sensing hash coding method to obtain the character vector until the characters in the image are completely decoded, thereby obtaining a scene character recognition result. According to the method, the recognition accuracy of the model on shielded and fuzzy samples can be improved, the problems of attention drift and calculation amount increase faced by a traditional autoregression decoding method can be relieved, and the method can be widely applied to the technical field of computer vision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a scene text recognition method based on single vector query decoding and related equipment. BACKGROUND

[0002] As an example, scene text recognition (STR) aims to extract text from natural images, and its application in automatic driving, product identification, robot navigation and other fields makes it a popular research direction in recent years. Current scene text recognition methods are mainly divided into two categories: (1) CTC (Connectionist Temporal Classification) based methods, which usually use connectionist temporal classification as a decoding scheme, but due to its inherent limitation of converting images into one-dimensional input, CTC methods perform poorly when dealing with irregularly arranged text. (2) Encoder-Decoder methods usually use attention mechanisms to decode sequences, which have good modeling ability for irregular text and good ability to capture long-range dependencies, but often show weak context modeling ability and inaccurate sequence length estimation, thus limiting their effectiveness in complex scenarios. In order to obtain higher prediction accuracy, current methods usually use an auto-regressive (AR) decoding strategy, however, the AR decoder still has limitations: generating sequences character by character, and the decoded semantic features will linearly increase over time, leading to attention drift and slow reasoning speed. In addition, the lack of global context modeling is still a key challenge faced by AR decoders.

[0003] In summary, the technical problems in the related art need to be improved. SUMMARY

[0004] Embodiments of the present application aim to at least partially address one of the technical problems in the related art. To this end, the main purpose of the embodiments of the present application is to propose a scene text recognition method based on single vector query decoding and related equipment, which can improve the recognition accuracy of the model on occluded and blurred samples, and can alleviate the attention drift and increased computational complexity problems faced by traditional auto-regressive decoding methods, thereby reducing the reasoning computational complexity in the decoding stage.

[0005] To achieve the above purpose, one aspect of the embodiments of the present application proposes a scene text recognition method based on single vector query decoding, which comprises the following steps: obtaining an image to be recognized and a character to be encoded; wherein the character to be encoded for the first time is a text start symbol; extracting a visual feature from the image to be recognized; encoding the character to be encoded by using a position-aware hash encoding method to obtain a character vector; fusing global information of the visual feature and the character vector to obtain a target context vector; inputting the target context vector and the visual feature into a one-vector decoder for decoding to obtain a target decoding character corresponding to a current time step; If the target decoding character represents that the character in the image to be recognized is not completely decoded, the target decoding character is taken as the character to be encoded, and the step of encoding the character to be encoded by using the position-aware hash encoding method to obtain the character vector is returned to be executed until the target decoding character represents that the character in the image to be recognized is completely decoded, and a scene text recognition result is obtained.

[0006] In some embodiments, the step of extracting a visual feature from the image to be recognized includes: inputting the image to be recognized into a backbone network, and extracting a visual feature from the image to be recognized by the backbone network.

[0007] In some embodiments, the step of encoding the character to be encoded by using the position-aware hash encoding method to obtain a character vector includes: constructing an index position pair corresponding to the character to be encoded; mapping the index position pair into a corresponding hash value by using a hash function; taking the hash value as an index to retrieve the character vector from an embedding matrix.

[0008] In some embodiments, the step of fusing global information of the visual feature and the character vector to obtain a target context vector includes: aggregating the visual feature by using a global average pooling method to obtain a visual vector; aggregating the character vector by using a data average processing method to obtain a language vector; dynamically fusing the visual vector and the language vector by using a gated fusion method to obtain an initial context vector; adding the initial context vector and a position encoding to obtain the target context vector; wherein the position encoding is used to represent an encoding corresponding to a position of a character to be decoded in a text.

[0009] In some embodiments, the decoding of the context vector and the visual feature in the single vector decoder to obtain the target decoding character corresponding to the current time step comprises: inputting the context vector and the visual feature into the single vector decoder, performing fusion processing on the context vector and the visual feature through a multi-head cross attention module in the single vector decoder to obtain a visual language fusion feature, and performing layer normalization processing on the visual language fusion feature; extracting channel-in feature of the layer normalization processed visual language fusion feature through a feedforward neural network in the single vector decoder to obtain a channel-in fusion feature, and performing layer normalization processing on the channel-in fusion feature; performing linear projection processing on the layer normalization processed channel-in fusion feature through a linear projection module in the single vector decoder to generate the target decoding character corresponding to the current time step.

[0010] In some embodiments, if the target decoding character represents that the character in the image to be recognized is not completely decoded, the target decoding character is taken as the character to be encoded, and the step of encoding the character to be encoded by using the position-aware hash encoding method is returned to be executed until the target decoding character represents that the character in the image to be recognized is completely decoded, and a scene text recognition result is obtained, comprising: If the target decoding character represents that the character in the image to be recognized is not completely decoded, the target decoding character is taken as the character to be encoded, and the step of encoding the character to be encoded by using the position-aware hash encoding method is returned to be executed until the target decoding character represents that the character in the image to be recognized is completely decoded, and a scene text recognition result is obtained, comprising: If the updated target decoding character represents that the character in the image to be recognized is not completely decoded, the updated target decoding character and the target decoding character before updating are combined and taken as the character to be encoded, and the step of encoding the character to be encoded by using the position-aware hash encoding method is returned to be executed until the updated target decoding character represents that the character in the image to be recognized is completely decoded, and the scene text recognition result is obtained.

[0011] To achieve the above object, another aspect of the embodiment of the present application proposes a scene text recognition device based on single vector query decoding, which comprises the following modules: An image-text acquisition module is configured to acquire an image to be recognized and a character to be encoded; wherein the character to be encoded for the first time is a text start symbol. A visual feature extraction module is configured to perform visual feature extraction processing on the image to be recognized to obtain a visual feature. The character hash encoding module is used to encode the character to be encoded using a position-aware hash encoding method to obtain a character vector; A global information fusion module is used to perform global information fusion processing on the visual features and the character vector to obtain a target context vector; The single-vector decoding module is used to input the target context vector and the visual features into the single-vector decoder for decoding to obtain the target decoded character corresponding to the current time step; The scene text recognition result determination module is used to determine the scene text recognition result if the target decoded character represents that not all characters in the image to be recognized have been decoded. If so, the target decoded character is used as the character to be encoded, and the step of encoding the character to be encoded using the position-aware hash encoding method to obtain the character vector is returned, until the target decoded character represents that all characters in the image to be recognized have been decoded, and the scene text recognition result is obtained.

[0012] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0013] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0014] To achieve the above objectives, another aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0015] The embodiments of this application include at least the following beneficial effects: This application provides a scene text recognition method and related equipment based on single vector query decoding. This scheme obtains an image to be recognized and a character to be encoded; wherein, the character to be encoded for the first time is a character start symbol; visual feature extraction processing is performed on the image to be recognized to obtain visual features; a position-aware hash encoding method is used to encode the character to be encoded to obtain a character vector; global information fusion processing is performed on the visual features and the character vector to obtain a target context vector; the target context vector and the visual features are input into a single vector decoder for decoding to obtain the target decoded character corresponding to the current time step; if the target decoded character represents that not all characters in the image to be recognized have been decoded, the target decoded character is used as the character to be encoded, and the step of encoding the character to be encoded using a position-aware hash encoding method to obtain a character vector is returned until the target decoded character represents that all characters in the image to be recognized have been decoded to obtain the scene text recognition result. In this embodiment, except for the character to be encoded in the first round which is a character start symbol, the characters to be encoded in the remaining rounds are the decoded target characters. In this embodiment, the main focus is on the case where the character to be encoded is the decoded target character: by using a position-aware hash encoding method to encode the decoded target character, the sequence information of the decoded character can be preserved in the context vector; by performing global information fusion processing on visual features and character vectors, it can be understood as dynamically fusing global visual features with decoded language features, improving the model's ability to perceive global information, thereby improving the model's recognition accuracy on occluded and blurred samples; by inputting the context vector and visual features into a single vector decoder for decoding, that is, by encoding the semantic information of the decoded character into a single context vector, and using the context vector as a query vector to input a single vector decoder for autoregressive decoding, the attention drift and increased computation problem faced by traditional autoregressive decoding methods can be alleviated, thereby reducing the inference computation in the decoding stage. Attached Figure Description

[0016] Figure 1 This is a flowchart of the steps of the scene text recognition method based on single vector query decoding provided in the embodiments of this application; Figure 2 This is a schematic diagram of the overall process of the scene text recognition method based on single vector query decoding provided in the embodiments of this application; Figure 3 This is a schematic diagram of the hash encoding process provided in an embodiment of this application; Figure 4 This is a schematic diagram of the global information fusion process provided in an embodiment of this application; Figure 5 This is a schematic diagram of the single-vector decoding process provided in an embodiment of this application; Figure 6 This application provides an attention visualization diagram of a single vector decoder and other decoders in its embodiments; Figure 7 This is a schematic diagram of the structure of the scene text recognition device based on single vector query decoding provided in the embodiments of this application; Figure 8 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0018] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0019] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0021] As an example, Scene Text Recognition (STR) aims to extract text from natural images. Its applications in fields such as autonomous driving, product recognition, and robot navigation have made it a popular research direction in recent years. Current scene text recognition methods are mainly divided into two categories: (1) CTC (Connectionist Temporal Classification) based methods, which usually use connectionist temporal classification as the decoding scheme. CTC models the output as an implicit alignment of the input features and processes variable-length sequences by inserting blank tokens without explicit alignment. Among them, CRNN (Convolutional Recurrent Neural Network) stacks RNN (Recurrent Neural Network) on top of CNN (Convolutional Neural Network) and uses CTC for decoding; SVTR (Scene Text Vision Transformer) first divides the image into patches and captures the patterns between and within characters through mixing blocks. However, due to its inherent limitation of converting the image into a one-dimensional input, the CTC method performs poorly when dealing with irregularly arranged text (such as curved text or multi-directional text). (2) The encoder-decoder method usually uses an attention mechanism to decode the sequence. Its ability to model irregular text and capture long-range dependencies makes it the mainstream choice for current high-performance STR systems. Among them, the parallel decoding method can predict all characters at once. Although this method has a faster inference speed, it often shows weak context modeling ability and inaccurate sequence length estimation, thus limiting its effectiveness in complex scenes. To achieve higher prediction accuracy, current methods typically employ autoregressive (AR) decoding strategies. Early AR decoders often consisted of RNNs with attention mechanisms. These methods leveraged the sequential memory capabilities of RNNs to enhance contextual modeling. However, due to the inherent limitations of RNNs in capturing long-range dependencies, their performance degrades when processing long text sequences. With the successful application of Transformers in various visual tasks, researchers have incorporated them into STR (Speech Character Recognition) and used their decoders for decoding. Compared to RNN-based decoders, Transformer-based decoders are better at modeling complex language structures and long-range dependencies. Furthermore, they naturally support parallel computation, enabling more efficient training. Therefore, they have become the mainstream choice for AR decoders in recent years' scene text recognition research.However, AR decoders still have limitations: generating sequences character by character, the decoded semantic features grow linearly over time, leading to attention drift and slow inference speed. Furthermore, insufficient global context modeling remains a key challenge for AR decoders.

[0022] In view of this, this application provides a scene text recognition method and related device based on single vector query decoding. This scheme obtains an image to be recognized and a character to be encoded; wherein, the character to be encoded for the first time is a character start character; visual feature extraction processing is performed on the image to be recognized to obtain visual features; a position-aware hash coding method is used to encode the character to be encoded to obtain a character vector; global information fusion processing is performed on the visual features and the character vector to obtain a target context vector; the target context vector and the visual features are input into a single vector decoder for decoding to obtain the target decoded character corresponding to the current time step; if the target decoded character represents that not all characters in the image to be recognized have been decoded, the target decoded character is used as the character to be encoded, and the step of encoding the character to be encoded using a position-aware hash coding method to obtain a character vector is returned until the target decoded character represents that all characters in the image to be recognized have been decoded to obtain the scene text recognition result. In this embodiment, except for the character to be encoded in the first round which is a character start symbol, the characters to be encoded in the remaining rounds are the decoded target characters. In this embodiment, the main focus is on the case where the character to be encoded is the decoded target character: by using a position-aware hash encoding method to encode the decoded target character, the sequence information of the decoded character can be preserved in the context vector; by performing global information fusion processing on visual features and character vectors, it can be understood as dynamically fusing global visual features with decoded language features, improving the model's ability to perceive global information, thereby improving the model's recognition accuracy on occluded and blurred samples; by inputting the context vector and visual features into a single vector decoder for decoding, that is, by encoding the semantic information of the decoded character into a single context vector, and using the context vector as a query vector to input a single vector decoder for autoregressive decoding, the attention drift and increased computation problem faced by traditional autoregressive decoding methods can be alleviated, thereby reducing the inference computation in the decoding stage.

[0023] The scene text recognition method based on single vector query decoding provided in this application relates to the field of computer vision technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the scene text recognition method based on single vector query decoding, but is not limited to the above forms.

[0024] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0025] Please see Figure 1 , Figure 1 This is an optional flowchart of a scene text recognition method based on single vector query decoding provided in an embodiment of this application. Figure 1 The method may include, but is not limited to, steps S101 to S106.

[0026] Step S101: Obtain the image to be recognized and the character to be encoded; wherein, the character to be encoded for the first time is a text start character; The image to be identified is a natural scene text image.

[0027] For characters to be encoded, the first character to be encoded only includes the start character. Except for the first character on the surface, which is the start character of the text. In addition, the characters to be encoded in the remaining rounds are the total number of characters decoded in all previous rounds. , Let i be the sequence of characters to be decoded at time steps 0 to i.

[0028] It should be noted that the scene text recognition method based on single vector query decoding provided in this application embodiment is mainly based on the already decoded characters for subsequent scene text recognition.

[0029] Step S102: Perform visual feature extraction processing on the image to be identified to obtain visual features; In some embodiments, step S102 may include: inputting the image to be identified into a backbone network, and performing visual feature extraction processing on the image to be identified through the backbone network to obtain visual features.

[0030] In the specific implementation, the backbone network SVTR (Scene Text Vision Transformer) is used to process the input natural scene text image. Perform visual feature extraction and output visual features. .

[0031] Step S103: Encode the character to be encoded using a position-aware hash coding method to obtain a character vector; In some embodiments, step S103 may include: constructing an index position tuple corresponding to the character to be encoded; mapping the index position tuple to the corresponding hash value using a hash function; and retrieving the character vector from the embedding matrix using the hash value as an index.

[0032] Since the scene text recognition method based on single vector query decoding provided in this application mainly performs subsequent scene text recognition based on the already decoded characters, in the specific implementation, a position-aware hash encoding method is used to process the already decoded characters. Encode the decoded characters to achieve the desired result. Encode into character embeddings (i.e., character vectors) Specifically, the position-aware hashing method decouples the position and index of the decoded character, maps different characters at different positions to hash values ​​through a hash function, and finally selects the character embedding (i.e., character vector) using the hash value as the index.

[0033] In the first round of decoding, the "decoded characters" only include the start character; in subsequent decoding rounds, the "decoded characters" refer to the total number of characters decoded in all previous rounds.

[0034] Step S104: Perform global information fusion processing on the visual features and the character vector to obtain the target context vector; In some embodiments, step S104 may include: aggregating visual features using a global average pooling method to obtain a visual vector; aggregating character vectors using a data averaging method to obtain a language vector; dynamically fusing the visual vector and language vector using a gating fusion method to obtain an initial context vector; and adding the initial context vector to the positional encoding to obtain a target context vector; wherein the positional encoding is used to characterize the encoding corresponding to the position of the currently decoded character in the text.

[0035] In the specific implementation, visual features and existing character embedding Encode it into a single context vector using a global information fusion method. That is, obtaining visual features. and character vectors Then, visual features are fused through global information. and character vectors To obtain a context vector containing rich semantic information Specifically, this involves using visual aggregation mechanisms and language aggregation mechanisms to analyze visual features. and character vectors The visual vectors are obtained by aggregation. and text vector And through a gating fusion mechanism, visual vectors are... and text vector By dynamically combining them, an initial context vector with rich contextual semantic information is obtained. Finally, based on the initially obtained context vector Add position encoding This yields the final target context vector. .

[0036] Step S105: Input the target context vector and the visual features into a single vector decoder for decoding to obtain the target decoded character corresponding to the current time step; In some embodiments, step S105 may include: inputting the context vector and visual features into a single-vector decoder; fusing the context vector and visual features through a multi-head cross-attention module in the single-vector decoder to obtain visual-language fusion features; and performing layer normalization on the visual-language fusion features; extracting intra-channel features of the layer-normalized visual-language fusion features through a feedforward neural network in the single-vector decoder to obtain intra-channel fusion features; and performing layer normalization on the intra-channel fusion features; and performing linear projection on the layer-normalized intra-channel fusion features through a linear projection module in the single-vector decoder to generate the target decoded character corresponding to the current time step.

[0037] In this embodiment of the application, a context vector is input to the single-vector decoder. and visual features The context vector is obtained through multi-head cross-attention in the single-vector decoder. and visual features Perform fusion and output data (i.e., the aforementioned visual language fusion features), where data This indicates that the context vector As a query vector, and for the data Layer normalization is performed; subsequently, a feedforward neural network is used to extract the data after layer normalization. The intra-channel features are used to obtain intra-channel fused features, which are then subjected to layer normalization. After obtaining the output of the last decoder layer, a linear projection is used to generate the predicted character for the current time step. .

[0038] In the specific implementation, the context vector The input is processed by a single-vector decoder to decode the character corresponding to the current time step. The single-vector decoder consists of cross-attention and a feedforward neural network. By accepting a single context vector at each time step for prediction, it alleviates the problems of attention drift and increased computation caused by the increase in semantic vectors in traditional autoregressive decoders.

[0039] Step S106: If the target decoded character represents that not all characters in the image to be recognized have been decoded, then the target decoded character is used as the character to be encoded, and the step of encoding the character to be encoded using the position-aware hash encoding method to obtain a character vector is returned until the target decoded character represents that all characters in the image to be recognized have been decoded, and the scene text recognition result is obtained.

[0040] In some embodiments, step S106 may include: if the target decoded character represents that not all characters in the image to be recognized have been decoded, then the target decoded character is used as the character to be encoded, and the step of encoding the character to be encoded using a position-aware hash encoding method to obtain a character vector is returned, thus obtaining the updated target decoded character; if the updated target decoded character represents that not all characters in the image to be recognized have been decoded, then the updated target decoded character is combined with the target decoded character before the update and used as the character to be encoded, and the step of encoding the character to be encoded using a position-aware hash encoding method to obtain a character vector is returned, until the updated target decoded character represents that all characters in the image to be recognized have been decoded, thus obtaining the scene text recognition result.

[0041] Finally, the decoded In step S103, steps S103-S106 are repeated until all characters in the scene text image are decoded, resulting in a prediction sequence. Among them, the predicted sequence To include all sequence, In time step Decoded characters, Let i be the sequence of characters to be decoded at time steps 0 to i.

[0042] Steps S101 to S106 as shown in this embodiment involve acquiring an image to be recognized and a character to be encoded; wherein, the character to be encoded for the first time is a text start character; visual feature extraction processing is performed on the image to be recognized to obtain visual features; a position-aware hash coding method is used to encode the character to be encoded to obtain a character vector; global information fusion processing is performed on the visual features and the character vector to obtain a target context vector; the target context vector and the visual features are input into a single-vector decoder for decoding to obtain the target decoded character corresponding to the current time step; if the target decoded character represents that not all characters in the image to be recognized have been decoded, the target decoded character is used as the character to be encoded, and the process returns to the step of using a position-aware hash coding method to encode the character to be encoded to obtain a character vector, until the target decoded character represents that all characters in the image to be recognized have been decoded, and the scene text recognition result is obtained. In this embodiment, except for the character to be encoded in the first round which is a character start symbol, the characters to be encoded in the remaining rounds are the decoded target characters. In this embodiment, the main focus is on the case where the character to be encoded is the decoded target character: by using a position-aware hash encoding method to encode the decoded target character, the sequence information of the decoded character can be preserved in the context vector; by performing global information fusion processing on visual features and character vectors, it can be understood as dynamically fusing global visual features with decoded language features, improving the model's ability to perceive global information, thereby improving the model's recognition accuracy on occluded and blurred samples; by inputting the context vector and visual features into a single vector decoder for decoding, that is, by encoding the semantic information of the decoded character into a single context vector, and using the context vector as a query vector to input a single vector decoder for autoregressive decoding, the attention drift and increased computation problem faced by traditional autoregressive decoding methods can be alleviated, thereby reducing the inference computation in the decoding stage.

[0043] To explain in detail the principle of the technical solution of the present invention, the overall process of the present invention will be described below with reference to some specific embodiments. It is easy to understand that the following is an explanation of the technical principle of the present invention and should not be regarded as a limitation of the present invention.

[0044] This application proposes a scene text recognition method based on single-vector query decoding for natural scene text images. This method primarily involves encoding the semantic information of decoded characters into a single context vector, which is then used as the query vector input to a single-vector decoder for autoregressive decoding, thereby reducing the computational load of inference during the decoding stage. Furthermore, this application designs a dynamic global information fusion method to inject global visual information into the context vector, enhancing the model's ability to perceive global information. Finally, to preserve the order information of the decoded sequence in the context vector, this application designs a position-aware hash encoding method to encode the decoded characters. Please see Figure 2 , Figure 2 This is a schematic diagram of the overall process of the scene text recognition method based on single vector query decoding provided in the embodiments of this application, as follows: Figure 2 As shown, the overall process of the scene text recognition method based on single vector query decoding provided in this application embodiment includes the following steps: Step (1), transfer the scene text image The input is fed into a visual encoder for visual feature extraction to obtain visual features. This application utilizes the SVTR backbone network for processing input images. Visual features are extracted to obtain visual features for decoding.

[0045] Step (2): Using a position-aware hash encoding method, the decoded characters are processed... Encode the decoded characters to achieve the desired result. Encode into character embeddings (i.e., character vectors) The position-aware hashing method decouples the position and index of the decoded character, maps different characters at different positions to hash values ​​through a hash function, and finally selects the character embedding (i.e., character vector) using the hash value as the index. In the first round of decoding, the "decoded characters" only include the start character; in subsequent decoding rounds, the "decoded characters" refer to the total number of characters decoded in all previous rounds.

[0046] Step (3), visual features and existing character embedding Encode it into a single context vector using a global information fusion method. That is, after obtaining visual features and character vectors, the visual features and character vectors are fused through global information to obtain a context vector containing rich semantic information. Specifically, visual vectors and text vectors are obtained through visual aggregation mechanism and language aggregation mechanism respectively, and the two are dynamically combined through gating fusion mechanism to obtain a context vector with rich contextual semantic information.

[0047] Step (4) transfer the context vector The input is processed by a single-vector decoder to decode the character corresponding to the current time step. Specifically, after obtaining the context vector, it is input into a single-vector decoder for decoding to obtain the character corresponding to the current time step. The single-vector decoder consists of cross-attention and a feedforward neural network. By accepting a single context vector for prediction at each time step, it alleviates the attention drift and increased computational cost problems caused by the increase in semantic vectors in traditional autoregressive decoders. Finally, In input (2), repeat (2)-(4) until all characters in the scene text image are decoded.

[0048] In this embodiment of the application, the input is an RGB scene text image. The output is the predicted sequence. Among them, the predicted sequence To include all sequence, In time step Decoded characters. All input images were scaled down. Specifically, the implementation process of the scene text recognition method based on single vector query decoding is as follows: (1) Using a backbone network to process the input image Visual feature extraction is performed, with scene text images as input. The output is visual features. Specifically, scene text images Input into the backbone network In the process of generating text image features The calculation formula for visual feature extraction is as follows: ; This application selects SVTR-B as the backbone network. ,in, , and These are the input images. Height and width, and These are the input images. The number of channels, height, and width.

[0049] (2) Using a position-aware hashing method to process the decoded characters Encode it. For example... Figure 3 As shown, the input to the hash encoding is the already decoded character. (in, Indicates the current time step. (Indicates the start character), the output is the encoded character vector. Specifically, such as Figure 3 As shown, the input is a decoded string. First, for the string Each character in Construct a <index, position> tuple The specific calculation formula is as follows: ; in, , Indicates the first The index of a character in the character set. Indicates the position of the character in the decoded substring. and These represent the size of the character set and the maximum text length, respectively.

[0050] Then, each tuple is mapped to a hash value using a hash function. The specific calculation formula is as follows: ; in, Indicates the maximum text length of the character set.

[0051] Finally, the hash value of each character... For indexing, from the embedding matrix Retrieve the corresponding character vector from the middle character vector The calculation formula is as follows: ; in, This represents a trainable embedding matrix. Indicates the retrieved first Character embedding.

[0052] (3) Context vector generation. After obtaining visual features... and character vectors Then, visual features are fused through global information. and character vectors To obtain a context vector containing rich semantic information .like Figure 4 As shown, the global information fusion process includes three steps: visual aggregation, linguistic aggregation, and gating fusion. Specifically, for the input visual features... The visual aggregation process uses Global Average Pooling (GAP) to aggregate the visual vectors. Visual vectors The calculation formula is as follows: ; Language aggregation then converts the input character vector By averaging, we obtain the language vector. Language vectors The calculation formula is as follows: ; in, Indicates the retrieved first Character embedding, i.e. For the retrieved character vector The first in item, Character vectors This represents all characters that have been decoded at time step i. Embedded.

[0053] Next, visual vectors are fused using gating. and language vectors Dynamic fusion is performed to obtain a preliminary context vector with rich semantic information. The specific calculation formula is as follows: ; ; in, For the gated vector, These are candidate fusion features.

[0054] Subsequently, the gating vector is controlled through a gating mechanism. and candidate fusion features Adaptive fusion is performed to obtain the fused feature representation. (i.e., a preliminary context vector with rich semantic information) ), The calculation formula is as follows: ; in, , , All are learnable parameters. It is the sigmoid function. This indicates element-wise multiplication.

[0055] Finally, based on the initially obtained context vector Add position encoding This yields the final context vector. , The specific calculation formula is as follows: ; Among them, position coding This indicates the encoding corresponding to the position of the character in the text that needs to be decoded.

[0056] (4) Decoding based on context vectors. After obtaining the context vector... Then, it is input into a single-vector decoder for decoding to obtain the character corresponding to the current time step. Specifically, such as... Figure 5 As shown, the context vector is input to the single-vector decoder. and visual features By using multi-head cross attention to combine context vectors and visual features Perform fusion and output data Among them, data This indicates that the context vector As a query vector, with visual features The specific calculation formula for the output obtained after multi-head cross-attention fusion is as follows: ; ; in, These are learnable parameters. This represents the dimension of each attention head. Indicates the number of attention heads. It is the output mapping matrix. Represented as Any one of them.

[0057] Subsequently, a feedforward neural network is used to extract intra-channel features, and the specific calculation formula is as follows: ; in, These are learnable parameters; decoders can be stacked. Layer. After obtaining the output of the last decoder layer, a linear projection is used to generate the predicted character for the current time step. The structure of a single-vector decoder is as follows: Figure 5 As shown, for simplicity, layer normalization is omitted in the formula. Then, the decoded character sequences at time steps 0 to i... In input step (2), repeat steps (2) to (4) until all characters are decoded to obtain the predicted sequence. .

[0058] The loss function used in the overall training process of the scene text recognition network based on single vector query decoding is as follows: .

[0059] It should be noted that this embodiment is only a brief illustrative description of the overall process of the scene text recognition method based on single vector query decoding. Detailed descriptions of each step can be found in the relevant content of the foregoing embodiments, and will not be repeated here. It is understood that the present invention does not limit this.

[0060] This application embodiment acquires an image to be recognized and characters to be encoded; wherein, the character to be encoded for the first time is a text start character; visual feature extraction processing is performed on the image to be recognized to obtain visual features; a position-aware hash coding method is used to encode the character to be encoded to obtain a character vector; global information fusion processing is performed on the visual features and character vector to obtain a target context vector; the target context vector and visual features are input into a single-vector decoder for decoding to obtain the target decoded character corresponding to the current time step; if the target decoded character represents that not all characters in the image to be recognized have been decoded, the target decoded character is used as the character to be encoded, and the step of encoding the character to be encoded using a position-aware hash coding method to obtain a character vector is returned until the target decoded character represents that all characters in the image to be recognized have been decoded to obtain the scene text recognition result. In this embodiment, except for the character to be encoded in the first round which is a character start symbol, the characters to be encoded in the remaining rounds are the decoded target characters. In this embodiment, the main focus is on the case where the character to be encoded is the decoded target character: by using a position-aware hash encoding method to encode the decoded target character, the sequence information of the decoded character can be preserved in the context vector; by performing global information fusion processing on visual features and character vectors, it can be understood as dynamically fusing global visual features with decoded language features, improving the model's ability to perceive global information, thereby improving the model's recognition accuracy on occluded and blurred samples; by inputting the context vector and visual features into a single vector decoder for decoding, that is, by encoding the semantic information of the decoded character into a single context vector, and using the context vector as a query vector to input a single vector decoder for autoregressive decoding, the attention drift and increased computation problem faced by traditional autoregressive decoding methods can be alleviated, thereby reducing the inference computation in the decoding stage.

[0061] In addition, this application also conducted relevant test experiments on the proposed scene text recognition method based on single vector query decoding. The test environment and experimental results are as follows: (1) Test environment: System environment: Ubuntu 22.04.3 LTS; Hardware environment: Memory: 80GB, GPU: NVIDIA A100 Tensor Core GPU, CPU: Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz.

[0062] (2) Experimental data: This application trains on the real dataset Union14M-Filter and tests it on several commonly used STR test sets, including the most commonly used public datasets Common Benchmarks (CoB, which includes IC13, SVT, IIIT5k, IC15, SVTP and CUTE80), Union14M-benchmarks (U14M-B, which includes seven subsets: CUR, MO, ART, CTL, SAL, MTW and GEN) and more challenging test sets (such as More ChallengingBenchmarks, MCB, which includes Uber, ArT, COCOv1.4, WordArt and OST).

[0063] In this embodiment, further experimental settings are as follows: During training, the AdamW optimizer is used, with weight decay set to 0.05 and learning rate set to [missing value]. The batch size is 1024. Training lasts for 25 epochs, using a one-cycle learning rate scheduler with linear warm-up in the first 1.5 epochs. All input images are resized to 32×128. For data augmentation, this embodiment uses RandAugment, consistent with PARSeq, with a maximum text length L of 25 and a character set length N of 96 (including uppercase and lowercase letters, numbers, punctuation marks, and start, end, and padding tokens).

[0064] Specifically, the experimental results are as follows: 1) Comparative experiment: To verify the effectiveness of the method proposed in this application, it is compared with decoding methods for text recognition in other scenarios. Specifically, this application uses the same backbone network for feature extraction and compares the performance of the decoding method proposed in this application with other common decoding methods. As shown in Table 1, Table 1 illustrates the effectiveness verification of the single-vector decoder proposed in this application: Table 1: Validation diagram of a single-vector decoder

[0065] As shown in Table 1, FLOPS represents the number of floating-point operations, in units of 10 ... Time (G); Time represents inference time in milliseconds (ms). As shown in Table 1, although CTC-based and PD-based decoders have advantages in decoding speed and computational efficiency, their recognition performance is significantly lower than that of AR-based decoders. Compared with these decoders, the single-vector decoder proposed in this application achieves the highest recognition performance while maintaining low computational cost and decoding time. Specifically, compared with NRTR, the single-vector decoder (Ours in Table 1) proposed in this application improves the accuracy of the three benchmark categories by 0.26%, 0.45%, and 1.17%, respectively. This improvement is mainly attributed to: (1) the effectiveness of the single-vector decoder in mitigating attention drift. Please refer to Figure 6 , Figure 6 This application provides an attention visualization diagram of a single-vector decoder and other decoders in its embodiments. Figure 6 (a) in the text represents the NRTR method. Figure 6 In the example (b), the method proposed in the embodiment of this application is shown, such as... Figure 6 As shown, compared with NRTR, the method proposed in this application provides a more focused and accurate representation of the position of the corresponding character in the input image; (2) explicit utilization of visual tokens (to be discussed in the ablation experiment stage below). More notably, due to its single-vector input design, the computational complexity of the single-vector decoder is greatly reduced, and the decoding time is only 58.2% of NRTR.

[0066] This application evaluates the performance of the proposed method on the most commonly used scene text recognition (STR) benchmark and compares it with related methods, as shown in Tables 2 and 3. Table 2 shows the performance comparison of the proposed method in the embodiments of this application with various scene text recognition methods on CommonBenchmarks, and Table 3 shows the performance comparison of the proposed method in the embodiments of this application with various scene text recognition methods on Union14M Benchmarks.

[0067] Table 2: Performance comparison of various text recognition methods in Common Benchmarks

[0068] Table 3: Performance comparison of various scene text recognition methods on Union14MBenchmarks

[0069] In Table 2, Avg refers to the average accuracy measured on Common Benchmarks, and in Table 3, Avg refers to the average accuracy measured on Union14M Benchmarks. As shown in Tables 2 and 3, the method proposed in this application outperforms previous STR methods and reaches the state-of-the-art performance level, highlighting the effectiveness of the proposed decoding strategy. Compared with the latest CTC method SVTRv2 and the latest parallel decoding method CPPD, the method proposed in this application improves the accuracy by 0.15% and 0.46% on CoB, respectively, and by 0.15% and 4.89% on U14M-B, respectively. These results demonstrate the superiority of the method proposed in this application. Furthermore, compared with the latest autoregressive decoding methods (including OTE-AR, IGTR-AR, and SMTR), the method proposed in this application outperforms by 0.78%, 0.31%, and 2.59% on CoB, respectively, and by 6.36%, 0.85%, and 0.69% on U14M-B, respectively. Since all models use the same encoder, the performance differences can be mainly attributed to the differences in decoder design, thus verifying the effectiveness of the decoding paradigm proposed in this application.

[0070] 2) Ablation experiment: To further verify the effectiveness of the other modules proposed in this application, ablation experiments were conducted. Table 4 shows the effectiveness verification of the other modules proposed in this application: Table 4: Validation Schematic Table for Other Modules

[0071] As shown in Table 4, introducing global visual information improved the accuracy of the three benchmark categories by 0.02%, 0.37%, and 0.93%, respectively (row 1 vs. row 2). This improvement can be attributed to the fact that autoregressive decoders often lack a comprehensive understanding of the input image in the early stages of decoding. In some challenging scenarios (such as occluded text or blurred characters), the fusion of global context can help the model more accurately interpret the local semantic regions of interest at each decoding step, thereby improving recognition performance. Subsequently, the introduction of a gated fusion mechanism (row 3) further improved the recognition accuracy compared to simple aggregation, with improvements of 0.24%, 0.34%, and 0.84% ​​on the three benchmarks, respectively. This is because the gating mechanism allows for weighting during the fusion of visual and semantic information, thereby enhancing the model's representational ability in complex tasks. In addition, the gating mechanism can also suppress irrelevant information during the fusion process, thereby reducing noise. Finally, the introduction of hash encoding (row 4) further improved the model performance. This improvement can be attributed to the gating fusion mechanism, where previously decoded characters are averaged after embedding. However, if Word2Vec is used for embedding, the character order information is lost, which is detrimental to AR decoding that relies on character order. Hash encoding preserves the positional information of the decoded characters, thus generating more informative embeddings. Results show that using hash encoding improves accuracy by 0.18%, 0.74%, and 0.55% on three benchmarks, respectively.

[0072] In summary, the scene text recognition method based on single vector query decoding provided in this application can achieve efficient scene text recognition, and its specific advantages are as follows: (1) Use a single vector decoder for autoregressive decoding to alleviate the problems of attention drift and increased computation faced by traditional autoregressive decoding methods. That is, encode the semantic information of the decoded characters into a single context vector and use it as a query vector to input a single vector decoder for autoregressive decoding to reduce the inference computation in the decoding stage. (2) By dynamically fusing global visual features with decoded language features through global information fusion, the model’s ability to perceive global information is enhanced, thereby improving the model’s recognition accuracy on occluded and blurred samples. (3) By using a position-aware hashing method, the hash value is calculated and encoded based on the position and index information of the character in the sequence, so as to ensure that the sequence information of the decoded character is preserved in the context vector.

[0073] Please see Figure 7 This application embodiment also provides a scene text recognition device 700 based on single vector query decoding, which can implement the above method. The device includes the following modules: Image and text acquisition module 701 is used to acquire an image to be recognized and a character to be encoded; wherein, the character to be encoded for the first time is a text start character; The visual feature extraction module 702 is used to perform visual feature extraction processing on the image to be identified to obtain visual features; The character hash encoding module 703 is used to encode the character to be encoded using a position-aware hash encoding method to obtain a character vector; The global information fusion module 704 is used to perform global information fusion processing on the visual features and the character vector to obtain the target context vector; The single-vector decoding module 705 is used to input the target context vector and the visual features into the single-vector decoder for decoding to obtain the target decoded character corresponding to the current time step; The scene text recognition result determination module 706 is used to determine the scene text recognition result if the target decoded character represents that not all characters in the image to be recognized have been decoded. If so, the target decoded character is used as the character to be encoded, and the step of encoding the character to be encoded using the position-aware hash encoding method to obtain the character vector is returned until the target decoded character represents that all characters in the image to be recognized have been decoded, and the scene text recognition result is obtained.

[0074] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0075] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0076] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0077] Please see Figure 8 , Figure 8 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 801 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 802 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802 and is called and executed by the processor 801 using the methods described in the embodiments of this application. The 803 input / output interface is used to implement information input and output. The communication interface 804 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 805 transmits information between various components of the device (e.g., processor 801, memory 802, input / output interface 803, and communication interface 804); The processor 801, memory 802, input / output interface 803, and communication interface 804 are connected to each other within the device via bus 805.

[0078] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0079] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0080] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0081] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0082] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0083] The scene text recognition method, apparatus, electronic device, storage medium, and program product based on single vector query decoding provided in this application embodiment acquire an image to be recognized and a character to be encoded. The first character to be encoded is a character start symbol. Visual features are extracted from the image to be recognized to obtain visual features. A position-aware hash coding method is used to encode the character to be encoded to obtain a character vector. Global information fusion processing is performed on the visual features and the character vector to obtain a target context vector. The target context vector and visual features are input into a single-vector decoder for decoding to obtain the target decoded character corresponding to the current time step. If the target decoded character represents that not all characters in the image to be recognized have been decoded, the target decoded character is used as the character to be encoded, and the process returns to the step of using a position-aware hash coding method to encode the character to be encoded to obtain a character vector, until the target decoded character represents that all characters in the image to be recognized have been decoded, resulting in a scene text recognition result. In this embodiment, except for the character to be encoded in the first round which is a character start symbol, the characters to be encoded in the remaining rounds are the decoded target characters. In this embodiment, the main focus is on the case where the character to be encoded is the decoded target character: by using a position-aware hash encoding method to encode the decoded target character, the sequence information of the decoded character can be preserved in the context vector; by performing global information fusion processing on visual features and character vectors, it can be understood as dynamically fusing global visual features with decoded language features, improving the model's ability to perceive global information, thereby improving the model's recognition accuracy on occluded and blurred samples; by inputting the context vector and visual features into a single vector decoder for decoding, that is, by encoding the semantic information of the decoded character into a single context vector, and using the context vector as a query vector to input a single vector decoder for autoregressive decoding, the attention drift and increased computation problem faced by traditional autoregressive decoding methods can be alleviated, thereby reducing the inference computation in the decoding stage.

[0084] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0085] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0086] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0087] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0088] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0089] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0090] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0091] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0092] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0093] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0094] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A scene text recognition method based on single vector query decoding, characterized in that, The method includes the following steps: Obtain the image to be recognized and the character to be encoded; wherein, the character to be encoded for the first time is a text start character; The image to be identified is subjected to visual feature extraction processing to obtain visual features; The character to be encoded is encoded using a position-aware hash encoding method to obtain a character vector; The visual features and the character vector are subjected to global information fusion processing to obtain the target context vector; The target context vector and the visual features are input into a single-vector decoder for decoding to obtain the target decoded character corresponding to the current time step; If the target decoded character represents that not all characters in the image to be recognized have been decoded, then the target decoded character is used as the character to be encoded, and the step of encoding the character to be encoded using the position-aware hash encoding method is returned to obtain the character vector, until the target decoded character represents that all characters in the image to be recognized have been decoded, and the scene text recognition result is obtained.

2. The method according to claim 1, characterized in that, The step of performing visual feature extraction processing on the image to be identified to obtain visual features includes: The image to be identified is input into the backbone network, and the visual features are extracted from the image through the backbone network to obtain the visual features.

3. The method according to claim 1, characterized in that, The method of encoding the character to be encoded using a position-aware hash encoding method to obtain a character vector includes: Construct the index position tuple corresponding to the character to be encoded; A hash function is used to map the index position tuple to the corresponding hash value; The character vector is retrieved from the embedding matrix using the hash value as an index.

4. The method according to claim 1, characterized in that, The step of performing global information fusion processing on the visual features and the character vector to obtain the target context vector includes: The visual features are aggregated using a global average pooling method to obtain a visual vector; The character vectors are aggregated using a data averaging method to obtain language vectors; A gating fusion method is used to dynamically fuse the visual vector and the language vector to obtain an initial context vector; The initial context vector is added to the positional encoding to obtain the target context vector; wherein the positional encoding is used to characterize the encoding corresponding to the position of the currently decoded character in the text.

5. The method according to claim 1, characterized in that, The step of inputting the context vector and the visual features into a single-vector decoder for decoding to obtain the target decoded character corresponding to the current time step includes: The context vector and the visual features are input into the single-vector decoder. The multi-head cross-attention module in the single-vector decoder fuses the context vector and the visual features to obtain visual-language fusion features. The visual-language fusion features are then subjected to layer normalization. The feedforward neural network in the single vector decoder extracts the intra-channel features of the visual-language fusion features after layer normalization, thus obtaining the intra-channel fusion features, and then performs layer normalization on the intra-channel fusion features. The linear projection module in the single-vector decoder performs linear projection processing on the channel-fused features after layer normalization to generate the target decoded character corresponding to the current time step.

6. The method according to claim 1, characterized in that, If the target decoded character represents that not all characters in the image to be recognized have been decoded, then the target decoded character is used as the character to be encoded, and the step of encoding the character to be encoded using a position-aware hash encoding method is returned to obtain a character vector, until the target decoded character represents that all characters in the image to be recognized have been decoded, and the scene text recognition result is obtained, including: If the target decoded character indicates that not all characters in the image to be recognized have been decoded, then the target decoded character is used as the character to be encoded, and the step of encoding the character to be encoded using the position-aware hash encoding method to obtain a character vector is returned to obtain the updated target decoded character; If the updated target decoded character represents that not all characters in the image to be recognized have been decoded, then the updated target decoded character is combined with the original target decoded character and used as the character to be encoded. The step of using the position-aware hash encoding method to encode the character to be encoded to obtain a character vector is returned until the updated target decoded character represents that all characters in the image to be recognized have been decoded, and the scene text recognition result is obtained.

7. A scene text recognition device based on single vector query decoding, characterized in that, The device includes the following modules: The image and text acquisition module is used to acquire the image to be recognized and the character to be encoded; wherein, the character to be encoded for the first time is a text start character. The visual feature extraction module is used to perform visual feature extraction processing on the image to be identified to obtain visual features; The character hash encoding module is used to encode the character to be encoded using a position-aware hash encoding method to obtain a character vector; A global information fusion module is used to perform global information fusion processing on the visual features and the character vector to obtain a target context vector; The single-vector decoding module is used to input the target context vector and the visual features into the single-vector decoder for decoding to obtain the target decoded character corresponding to the current time step; The scene text recognition result determination module is used to determine the scene text recognition result if the target decoded character represents that not all characters in the image to be recognized have been decoded. If so, the target decoded character is used as the character to be encoded, and the step of encoding the character to be encoded using the position-aware hash encoding method to obtain the character vector is returned, until the target decoded character represents that all characters in the image to be recognized have been decoded, and the scene text recognition result is obtained.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images