Image text category recognition method, device, medium and equipment

Abstract

The application provides a method and device for identifying the category of image text, a computer readable storage medium and an electronic device, and relates to the technical field of computers. The method comprises: extracting text detection features, text region information and text recognition features of an image to be identified; performing structural processing on the text detection features, the text region information and the text recognition features respectively to obtain multi-dimensional reference features; and identifying the category of the text content of the image to be identified according to the multi-dimensional reference features. In this way, the text detection features, the text region information and the text recognition features corresponding to different feature extraction stages are subjected to feature extraction to obtain multi-dimensional reference features, and then the category of the image text is identified based on the multi-dimensional reference features, so that a more accurate category recognition result can be obtained.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a method for recognizing text categories in an image, a device for recognizing text categories in an image, a computer-readable storage medium, and an electronic device. Background Technology

[0002] When performing specific type recognition on text in images, models such as Optical Character Recognition (OCR) are typically relied upon. These models first extract features from the image, then recognize the text within the image based on the final features, and finally perform specific type recognition on the text. In other words, it's a method that first obtains the text recognition result, then performs post-processing on the text recognition result to obtain the text type recognition result. This method relies on the text recognition capabilities of OCR; if the OCR text recognition result is inaccurate, it can easily lead to inaccurate determination of the specific text type.

[0003] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] The purpose of this application is to provide a method, device, computer-readable storage medium, and electronic device for recognizing text categories in images. This method extracts text detection features, text region information, and text recognition features corresponding to different feature extraction stages to obtain multi-dimensional reference features. Based on these multi-dimensional reference features, image text category recognition can be performed, resulting in more accurate category recognition results. Furthermore, because this application extracts text detection features, text region information, and text recognition features corresponding to different feature extraction stages and performs image text recognition based on the extracted multi-dimensional features, compared to existing technologies that rely on OCR and other models to perform image text category recognition using the final features extracted from the image, this method does not depend on OCR's recognition capabilities or any complex post-processing techniques. It can complete the text category recognition work and avoids the influence of noise during the feature extraction stage on the recognition results of text in images, thus further improving the accuracy of text category recognition in images.

[0005] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0006] According to one aspect of this application, a method for recognizing text categories in an image is provided, the method comprising:

[0007] Extract text detection features, text region information, and text recognition features from the image to be recognized;

[0008] Text detection features, text region information, and text recognition features are processed in a structured manner to obtain multi-dimensional reference features;

[0009] Identify the category of text content in the image to be identified based on multi-dimensional reference features.

[0010] In one exemplary embodiment of this application, before identifying the category of text content in the image to be identified based on multi-dimensional reference features, the method further includes:

[0011] The text content of the image to be recognized is obtained based on text detection features, text region information, and text recognition features.

[0012] In one exemplary embodiment of this application, extracting text detection features, text region information, and text recognition features from the image to be recognized includes:

[0013] Text detection features, text region information, and text recognition features of the image to be recognized are extracted from each processing stage of the sequential arrangement of the text recognition model.

[0014] In one exemplary embodiment of this application, after identifying the category of text content in the image to be identified based on multi-dimensional reference features, the method further includes:

[0015] Recognition results are generated based on the category and text content, corresponding to the image to be recognized.

[0016] In one exemplary embodiment of this application, the multi-dimensional reference features include a first reference feature, a second reference feature, and a third reference feature. The text detection features, text region information, and text recognition features are respectively subjected to structured processing to obtain the multi-dimensional reference features, including:

[0017] The text detection features and text region information are subjected to specific region pooling to obtain the first reference features;

[0018] The text region information is encoded with region features to obtain the second reference feature;

[0019] Text recognition features are encoded to obtain the third reference feature.

[0020] In one exemplary embodiment of this application, text detection features and text region information are subjected to specific region pooling processing to obtain a first reference feature, including:

[0021] Determine the set of region coordinates based on the text region information;

[0022] The text detection features are subjected to specific region pooling based on the set of region coordinates to obtain the first reference feature.

[0023] In one exemplary embodiment of this application, regional feature encoding is performed on text region information to obtain a second reference feature, including:

[0024] Regional feature encoding is performed on text region information to obtain the normalized centroid and normalized area corresponding to the text region information;

[0025] A second reference feature is generated based on the normalized centroid and normalized area.

[0026] In one exemplary embodiment of this application, text recognition features are encoded to obtain a third reference feature, including:

[0027] The text recognition features are encoded using a text encoder that includes multiple encoders and multiple decoders to obtain the third reference feature.

[0028] In one exemplary embodiment of this application, identifying the category of text content in an image to be identified based on multi-dimensional reference features includes:

[0029] The first reference feature, the second reference feature, and the third reference feature are fused to obtain the target feature.

[0030] The text content in the image to be identified is classified based on the target features to determine the category of the text content in the image to be identified.

[0031] According to one aspect of this application, a device for recognizing text categories in an image is provided, comprising:

[0032] The first feature extraction unit is used to extract text detection features, text region information, and text recognition features from the image to be recognized.

[0033] The second feature extraction unit is used to perform structured processing on text detection features, text region information, and text recognition features to obtain multi-dimensional reference features.

[0034] The classification unit is used to identify the category of text content in the image to be identified based on multi-dimensional reference features.

[0035] In one exemplary embodiment of this application, the above-described apparatus further includes:

[0036] The text content acquisition unit is used to acquire the text content of the image to be recognized based on text detection features, text region information, and text recognition features before the classification unit identifies the category of the text content in the image to be recognized based on multi-dimensional reference features.

[0037] In one exemplary embodiment of this application, the first feature extraction unit extracts text detection features, text region information, and text recognition features from the image to be recognized, including:

[0038] Text detection features, text region information, and text recognition features of the image to be recognized are extracted from each processing stage of the sequential arrangement of the text recognition model.

[0039] In one exemplary embodiment of this application, the above-described apparatus further includes:

[0040] The recognition result generation unit is used to generate a recognition result corresponding to the image to be recognized based on the category and text content after the classification unit identifies the category of the text content in the image to be recognized according to the multi-dimensional reference features.

[0041] In one exemplary embodiment of this application, the multi-dimensional reference features include a first reference feature, a second reference feature, and a third reference feature. The second feature extraction unit performs structured processing on the text detection features, text region information, and text recognition features respectively to obtain the multi-dimensional reference features, including:

[0042] The text detection features and text region information are subjected to specific region pooling to obtain the first reference features;

[0043] The text region information is encoded with region features to obtain the second reference feature;

[0044] Text recognition features are encoded to obtain the third reference feature.

[0045] In one exemplary embodiment of this application, the second feature extraction unit performs specific region pooling processing on text detection features and text region information to obtain a first reference feature, including:

[0046] Determine the set of region coordinates based on the text region information;

[0047] The text detection features are subjected to specific region pooling based on the set of region coordinates to obtain the first reference feature.

[0048] In one exemplary embodiment of this application, the second feature extraction unit performs region feature encoding on the text region information to obtain a second reference feature, including:

[0049] Regional feature encoding is performed on text region information to obtain the normalized centroid and normalized area corresponding to the text region information;

[0050] A second reference feature is generated based on the normalized centroid and normalized area.

[0051] In one exemplary embodiment of this application, the second feature extraction unit performs text feature encoding on the text recognition features to obtain a third reference feature, including:

[0052] The text recognition features are encoded using a text encoder that includes multiple encoders and multiple decoders to obtain the third reference feature.

[0053] In one exemplary embodiment of this application, the classification unit identifies the category of text content in the image to be identified based on multi-dimensional reference features, including:

[0054] The first reference feature, the second reference feature, and the third reference feature are fused to obtain the target feature.

[0055] The text content in the image to be identified is classified based on the target features to determine the category of the text content in the image to be identified.

[0056] According to one aspect of this application, a computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the method of any one of the above.

[0057] According to one aspect of this application, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the method of any of the above by executing the executable instructions.

[0058] The exemplary embodiments of this application may have some or all of the following beneficial effects:

[0059] In an example embodiment of this application, a method for recognizing text categories in an image is provided. This method extracts text detection features, text region information, and text recognition features from the image to be recognized. The text detection features, text region information, and text recognition features are then processed in a structured manner to obtain multi-dimensional reference features. The category of the text content in the image to be recognized is then identified based on these multi-dimensional reference features. This allows for feature extraction of text detection features, text region information, and text recognition features corresponding to different feature extraction stages, resulting in multi-dimensional reference features. Recognizing the image text category based on these multi-dimensional reference features yields more accurate category recognition results. Furthermore, since this application can extract text detection features, text region information, and text recognition features corresponding to different feature extraction stages and perform image text recognition based on the extracted multi-dimensional features, compared to existing technologies that rely on OCR and other models to perform image text category recognition using the final features extracted from the image, this method does not depend on the recognition capabilities of OCR or any complex post-processing techniques. It can complete the text category recognition work and avoids the influence of noise during the feature extraction stage on the recognition results of the text in the image, thus further improving the accuracy of text category recognition in the image.

[0060] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0061] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0062] Figure 1 A schematic diagram of an exemplary system architecture for a method and apparatus for recognizing text categories in an image, which can be applied to embodiments of this application, is shown.

[0063] Figure 2 A flowchart illustrating a method for recognizing text categories in an image according to an embodiment of this application is shown schematically.

[0064] Figure 3 A flowchart illustrating a method for recognizing text categories in an image according to another embodiment of this application is shown schematically;

[0065] Figure 4 A schematic diagram of an image recognition system according to this application is shown.

[0066] Figure 5A schematic diagram of a text encoder according to this application is shown.

[0067] Figure 6 A schematic diagram illustrating the feature fusion process according to this application is shown.

[0068] Figure 7 The illustration shows a schematic diagram of the text recognition result in an image according to this application;

[0069] Figure 8 This schematically illustrates a structural block diagram of a text category recognition device in an image according to one embodiment of the present application;

[0070] Figure 9 The schematic diagram illustrates the structure of a computer system suitable for implementing the electronic devices of the present application. Detailed Implementation

[0071] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of the embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this application.

[0072] Furthermore, the accompanying drawings are merely illustrative of this application and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0073] Please see Figure 1 , Figure 1 This diagram illustrates a system architecture of an exemplary application environment for a method and apparatus for recognizing text categories in images, which can be applied according to embodiments of this application. Figure 1As shown, system architecture 100 may include one or more of terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 is used as a medium to provide a communication link between terminal devices 101, 102, and 103 and server 105.

[0074] Network 104 can include various connection types, such as wired, wireless communication links, or fiber optic cables. Terminal devices 101, 102, and 103 can be devices providing voice and / or data connectivity to users, handheld devices with wireless connectivity, or other processing devices connected to a wireless modem. The wireless terminal can communicate with one or more core networks via the RAN. The wireless terminal can be user equipment (UE), a handheld terminal, a laptop computer, a subscriber unit, a cellular phone, a smartphone, a wireless data card, a personal digital assistant (PDA) computer, a tablet computer, a wireless modem, a handheld device, a laptop computer, a cordless phone, a wireless local loop (WLL) station, a machine-type communication (MTC) terminal, or other devices capable of accessing the network. The terminal communicates with the access network equipment using some air interface technology (e.g., 3GPP access technology or non-3GPP access technology). It should be understood that... Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, there can be any number of terminal devices, networks, and servers. For example, server 105 could be a server cluster composed of multiple servers.

[0075] The image text category recognition method provided in this embodiment can be executed by server 105, and correspondingly, the image text category recognition device is generally set in server 105. However, those skilled in the art will readily understand that the image text category recognition method provided in this embodiment can also be executed by terminal device 101, 102, or 103, and correspondingly, the image text category recognition device can also be set in terminal device 101, 102, or 103. This exemplary embodiment does not impose any special limitations on this. For example, in one exemplary embodiment, terminal device 101, 102, or 103 can upload an image to be recognized, and server 105 can extract text detection features, text region information, and text recognition features from the image to be recognized; perform structured processing on the text detection features, text region information, and text recognition features respectively to obtain multi-dimensional reference features; and identify the category of text content in the image to be recognized based on the multi-dimensional reference features.

[0076] Please see Figure 2 , Figure 2 A flowchart illustrating a method for recognizing text categories in an image according to an embodiment of this application is shown schematically. Figure 2 As shown, the method for recognizing text categories in the image may include steps S210 to S230.

[0077] Step S210: Extract text detection features, text region information, and text recognition features from the image to be recognized.

[0078] Step S220: Perform structured processing on the text detection features, text region information, and text recognition features respectively to obtain multi-dimensional reference features.

[0079] Step S230: Identify the category of text content in the image to be identified based on multi-dimensional reference features.

[0080] Implementation Figure 2 The method described herein can extract text detection features, text region information, and text recognition features corresponding to different feature extraction stages, thereby obtaining multi-dimensional reference features. Based on these multi-dimensional reference features, image text category recognition can be performed, resulting in more accurate category recognition results. Furthermore, since this application can extract text detection features, text region information, and text recognition features corresponding to different feature extraction stages and perform image text recognition based on the extracted multi-dimensional features, compared to existing technologies that rely on OCR and other models to perform image text category recognition through the final features extracted from the image, it does not depend on the recognition capabilities of OCR or any complex post-processing techniques. It can complete the text category recognition work without relying on OCR's recognition capabilities or any complex post-processing techniques. It also avoids the influence of noise during the feature extraction stage on the text recognition results in the image, thus further improving the accuracy of text category recognition in the image.

[0081] The steps described above in this example implementation will now be explained in more detail.

[0082] In step S210, text detection features, text region information, and text recognition features of the image to be recognized are extracted.

[0083] Specifically, text detection features, text region information, and text recognition features can correspond to different representation methods or the same representation method, and the embodiments of this application do not limit them.

[0084] The process involves extracting text detection features, text region information, and text recognition features from the image to be recognized. This includes extracting these features based on an Optical Character Recognition (OCR) model. The text detection features, text region information, and text recognition features correspond to different stages of OCR feature extraction. This facilitates the joint application of text recognition models and category recognition models, thereby improving the accuracy of text category recognition.

[0085] Specifically, this can include: extracting features from the image to be recognized using convolutional layers of the OCR text detection model backbone; further extracting features from the extracted features using a text detection model neck (e.g., Feature Pyramid Networks, FPN) to obtain text detection features; then, inputting the text detection features into the text detection model head to obtain text region information of the image to be recognized; and finally, using convolutional layers of the OCR text recognition model backbone to extract features from the text region information, and inputting the extracted features into the text recognition model neck to further extract features to obtain text recognition features.

[0086] Furthermore, the above method may also include: inputting text recognition features into the text recognition model content output network (head) to obtain the text content of the image to be recognized.

[0087] In step S220, the text detection features, text region information, and text recognition features are respectively processed in a structured manner to obtain multi-dimensional reference features.

[0088] Specifically, multi-dimensional reference features include reference features of multiple dimensions. This application does not limit the number of reference features. Reference features of different dimensions are different and have different representational meanings.

[0089] As an optional embodiment, the multi-dimensional reference features include a first reference feature, a second reference feature, and a third reference feature. The text detection features, text region information, and text recognition features are respectively structured to obtain the multi-dimensional reference features, including: performing specific region pooling on the text detection features and text region information to obtain the first reference feature; performing region feature encoding on the text region information to obtain the second reference feature; and performing text feature encoding on the text recognition features to obtain the third reference feature. This approach enhances the image representation capability based on the first, second, and third reference features, thereby improving image recognition accuracy.

[0090] Specifically, the first reference feature, the second reference feature, and the third reference feature may correspond to the same dimension or different dimensions, and this application embodiment does not limit this.

[0091] As an optional embodiment, specific region pooling is performed on text detection features and text region information to obtain a first reference feature, including: determining a set of region coordinates based on the text region information; and performing specific region pooling on the text detection features based on the set of region coordinates to obtain the first reference feature. This allows obtaining the region pooling results corresponding to the text detection features and text region information, and image text recognition based on the region pooling results can yield more accurate results.

[0092] Specifically, determining the set of region coordinates based on text region information includes: inputting the text region information into the Region of Interest Pooling (RoI pooling) module so that the RoI pooling module can determine the set of region coordinates S corresponding to the text region information.

[0093] Furthermore, based on the set of region coordinates, specific region pooling is performed on the text detection features to obtain the first reference feature, including: substituting the set of region coordinates and the text detection feature D into the expression. To calculate the first reference feature F D ;in, H represents the height of the image to be recognized, W represents the width of the image to be recognized, and F represents the height of the image to be recognized. D For dimension R C The feature vector, S, is the set of coordinates of the region enclosed by the text region P, P = {(x1,y1),(x2,y2),……,(x...}. n ,y n )}, where n is a positive integer, and (i,j) is used to represent the coordinates in the set of region coordinates, where i represents the x-coordinate and j represents the y-coordinate.

[0094] As an optional embodiment, regional feature encoding is performed on the text region information to obtain a second reference feature, including: performing regional feature encoding on the text region information to obtain the normalized centroid and normalized area corresponding to the text region information; and generating the second reference feature based on the normalized centroid and normalized area. This allows for the acquisition of the regional feature encoding result corresponding to the text region information, and image text recognition based on the regional feature encoding result can yield more accurate results.

[0095] Specifically, the text region information is encoded using region features to obtain the normalized centroid and normalized area corresponding to the text region information. This includes: inputting the text region information into a text region feature encoder (Polygon Encoder, PE) to encode the region coordinate set P = {(x1,y1), 9x2,y2), ..., (x n ,y n Region feature encoding is performed to obtain the normalized centroids (x) corresponding to the text region information. c ,y c ) and the normalized area A, where, Furthermore, a second reference feature is generated based on the normalized centroid and normalized area, including: [the normalized centroid (x...]] c ,y c And the normalized area A is substituted into F. p =[A,x c ,y c ], to generate the second reference feature F p .

[0096] As an optional embodiment, text feature encoding is performed on the text recognition features to obtain a third reference feature. This includes: performing text feature encoding on the text recognition features based on a text encoder that includes multiple encoders and multiple decoders to obtain the third reference feature. This allows obtaining the text feature encoding result corresponding to the text recognition features, and performing image text recognition based on the text feature encoding result can yield more accurate results.

[0097] Specifically, text recognition features are encoded using a text encoder that includes multiple encoders and multiple decoders to obtain a third reference feature, including: an expression F based on the text feature encoder (TE) that includes multiple encoders and multiple decoders. T =ViT(T) encodes the text recognition features T to obtain the third reference feature F. T Where, T∈R N×K F T ∈R KK is the text feature embedding dimension (e.g., K = 512), and N is used to characterize the text length.

[0098] In step S230, the category of text content in the image to be identified is determined based on multi-dimensional reference features.

[0099] Specifically, when the embodiments of this application are applied to the advertising field, the categories may include: advertising category and non-advertising category. When the embodiments of this application are applied to the academic field, the categories may include: academic category and non-academic category.

[0100] As an optional embodiment, identifying the category of text content in an image to be identified based on multi-dimensional reference features includes: fusing first, second, and third reference features to obtain target features; and classifying the text content in the image to be identified based on the target features to determine the category of the text content. This allows for text category identification in an image based on the fusion result of the first, second, and third reference features, improving the accuracy of the identification results, reducing the impact of noise in the OCR feature extraction results on the identification results, and enhancing the generalization ability of the image text recognition scheme.

[0101] Specifically, the first reference feature, the second reference feature, and the third reference feature are fused to obtain the target feature, including: fusing the first reference feature F... D Second reference feature F P Third reference feature F T Input the Feature Fusion (FF) module to perform feature fusion based on the expression. The target features F, F∈R are calculated. 3d d is a hyperparameter of the embedding dimension (e.g., d = 128); where, Where ReLU is a non-linear activation function, W D b D W P b P W T b T These are the learnable model parameters.

[0102] Furthermore, based on the target features, the text content in the image to be recognized is classified to determine the category of the text content in the image to be recognized. This includes: inputting the target features F into the classifier, so that the classifier is based on the expression Q = Softmax((FW) Q )+b Q ) Determine the category of the image to be identified, W Q ∈R 3d×2 bQ ∈R 2 The classifier can be understood as a fully connected neural network, which can be a binary classification network or a multi-class classification network. This application does not limit the implementation. Softmax is the activation function, and W... Q and b Q These are the learnable model parameters. Q represents the probability that the text in the image to be recognized belongs to each category; the category corresponding to the highest probability can be determined as the category of the corresponding text in the image to be recognized.

[0103] As an optional embodiment, before identifying the category of text content in the image to be identified based on multi-dimensional reference features, the method further includes: obtaining the text content of the image to be identified based on text detection features, text region information, and text recognition features. This allows the text content of the image to be identified to be obtained, so that subsequent generation of recognition results that can more accurately represent the image to be identified can be achieved.

[0104] Specifically, the text content of the image to be recognized is obtained based on text detection features, text region information, and text recognition features, including: extracting text detection features, text region information, and text recognition features of the image to be recognized based on the Optical Character Recognition (OCR) model.

[0105] As an optional embodiment, after identifying the category of the text content in the image to be identified based on multi-dimensional reference features, the method further includes: generating a recognition result corresponding to the image to be identified based on the category and text content. This allows for the generation of recognition results with stronger representational capabilities based on the category and text content.

[0106] Specifically, generating a recognition result corresponding to the image to be recognized based on the category and text content includes: associating the image to be recognized, the category, and the text content, and outputting the association result as the recognition result. The recognition result can be represented in the form of text, image, table, etc., and this embodiment does not limit the representation. It should be noted that if the text content consists of multiple lines, there can also be multiple categories, each representing the category to which a different line of text belongs.

[0107] Please see Figure 3 , Figure 3 A flowchart illustrating a method for recognizing text categories in an image according to another embodiment of this application is shown schematically. Figure 3 As shown, the method for recognizing text categories in the image may include steps S310 to S380.

[0108] Step S310: Extract text detection features, text region information, and text recognition features from the image to be recognized.

[0109] Step S320: Obtain the text content of the image to be recognized based on text detection features, text region information, and text recognition features.

[0110] Step S330: Determine the set of region coordinates based on the text region information, and perform specific region pooling processing on the text detection features based on the set of region coordinates to obtain the first reference feature.

[0111] Step S340: Encode the text region information using region features to obtain the normalized centroid and normalized area corresponding to the text region information, and generate a second reference feature based on the normalized centroid and normalized area.

[0112] Step S350: Based on a text encoder containing multiple encoders and multiple decoders, text recognition features are encoded to obtain a third reference feature.

[0113] Step S360: Perform feature fusion on the first reference feature, the second reference feature, and the third reference feature to obtain the target feature.

[0114] Step S370: Classify the text content in the image to be identified based on the target features to determine the category of the text content in the image to be identified.

[0115] Step S380: Generate recognition results corresponding to the image to be recognized based on the category and text content.

[0116] It should be noted that steps S310 to S380 are the same as... Figure 2 For the specific implementation details of steps S310 to S380, please refer to the examples shown. Figure 2 The steps and their embodiments shown are not repeated here.

[0117] It is evident that implementation Figure 3 The method described herein can extract text detection features, text region information, and text recognition features corresponding to different feature extraction stages, thereby obtaining multi-dimensional reference features. Based on these multi-dimensional reference features, image text category recognition can be performed, resulting in more accurate category recognition results. Furthermore, since this application can extract text detection features, text region information, and text recognition features corresponding to different feature extraction stages and perform image text recognition based on the extracted multi-dimensional features, compared to existing technologies that rely on OCR and other models to perform image text category recognition through the final features extracted from the image, it does not depend on the recognition capabilities of OCR or any complex post-processing techniques. It can complete the text category recognition work without relying on OCR's recognition capabilities or any complex post-processing techniques. It also avoids the influence of noise during the feature extraction stage on the text recognition results in the image, thus further improving the accuracy of text category recognition in the image.

[0118] Please see Figure 4 , Figure 4 A schematic diagram of an image recognition system according to this application is shown. Figure 4 As shown, the image recognition system may include: OCR 410 and category recognition model 420. Specifically, OCR 410 may include a text detection model 411 and a text recognition model 412. The text detection model 411 may include a backbone network 4111, a neck network 4112, and a content output network 4113. The text recognition model 412 may include a backbone network 4121, a neck network 4122, and a content output network 4123. The category recognition model 420 may include a structured feature extraction module 421 and a classifier 422. The structured feature extraction module 421 may include a Region of Interest Pooling (RoI pooling) module 4211, a Polygon Encoder (PE) 4212, a Text Feature Encoder (TE) 4213, and a feature fusion module 4214.

[0119] Specifically, the image to be recognized can be input into the text detection model 411, so that the text detection model 411 can extract text detection features and text region information of the image to be recognized through the backbone network 4111, the neck network 4112, and the content output network 4113. Furthermore, the text detection model 411 can also input the output text region information into the text recognition model 412, so that the text recognition model 412 can extract text recognition features of the image to be recognized through the backbone network 4121, the neck network 4122, and the content output network 4123. Thus, the text content of the image to be recognized can be obtained based on the text recognition features.

[0120] Furthermore, a first reference feature can be obtained by using the Region of Interest Pooling (RoIpooling) module 4211 to determine the set of region coordinates based on the text region information, and by performing specific region pooling processing on the text detection features based on the set of region coordinates; a second reference feature can be obtained by using the Polygon Encoder (PE) 4212 to encode the text region information into region features, thereby obtaining the normalized centroid and normalized area corresponding to the text region information, and generating the second reference feature based on the normalized centroid and normalized area; and a third reference feature can be obtained by using the Text Feature Encoder (TE) 4213 to encode the text recognition features into text features based on the text encoder containing multiple encoders and multiple decoders.

[0121] Furthermore, the feature fusion module 4214 can fuse the first reference feature, the second reference feature, and the third reference feature to obtain the target feature. This target feature is then input into the classifier 422, which classifies the text content in the image to be recognized based on the target feature to determine the category of the text content. Subsequently, based on the category and text content, a recognition result corresponding to the image to be recognized can be generated. The classifier 422 can be either a binary classification model or a multi-class classification model.

[0122] Please see Figure 5 , Figure 5 A schematic diagram of a text encoder according to this application is shown. Figure 5 As shown, the text encoder may include a Transformer Encoder model 510 and an MLP Head model 520; wherein, the Transformer Encoder model 510 is used to perform multiple feature encoding and decoding, and the MLP Head model 520 is used to perform feature parsing on the output of the Transformer Encoder model 510.

[0123] Specifically, the text recognition features can be constructed from multiple characters, each character can correspond to a feature, and the position embedding features of each character can be determined according to the order of each character. The position embedding features of each character and the features that each character can correspond to are used as input to the Transformer Encoder model 510 so that the Transformer Encoder model 510 outputs the feature extraction result corresponding to the text recognition features. The MLP Head model 520 can parse the feature extraction result into a third reference feature.

[0124] Please see Figure 6 , Figure 6 A schematic diagram illustrating the feature fusion process according to this application is shown. Figure 6 As shown, in the feature fusion module, the first reference feature 610, the second reference feature 620, and the third reference feature 630 can be cross-fused to obtain the target feature 640.

[0125] Please see Figure 7 , Figure 7 The illustration shows a schematic diagram of the text recognition results in an image according to this application. For example... Figure 7 As shown, through Figure 3 The steps shown can identify the image 710 to be recognized as text content 720 and category 730. It can be understood that the category 730 corresponding to "Fun and Simple Code Programming" in text content 720 can be "Non-Advertising," the category 730 corresponding to "QQ: 1234321431534414" in text content 720 can be "Advertising," and the category 730 corresponding to "Long press the image to recognize the QR code and add my enterprise WeChat" in text content 720 can be "Non-Advertising." Therefore, implementing the embodiments of this application can improve the efficiency and accuracy of text category recognition.

[0126] Please see Figure 8 , Figure 8 The diagram schematically illustrates a structural block diagram of a text category recognition device for images according to one embodiment of this application. The text category recognition device 800 for images and... Figure 2 The methods shown correspond to those described, such as Figure 8 As shown, the text category recognition device 800 in the image includes:

[0127] The first feature extraction unit 801 is used to extract text detection features, text region information, and text recognition features of the image to be recognized.

[0128] The second feature extraction unit 802 is used to perform structured processing on text detection features, text region information, and text recognition features to obtain multi-dimensional reference features.

[0129] Classification unit 803 is used to identify the category of text content in the image to be identified based on multi-dimensional reference features.

[0130] It is evident that implementation Figure 8The apparatus shown can extract text detection features, text region information, and text recognition features corresponding to different feature extraction stages, thereby obtaining multi-dimensional reference features. Based on these multi-dimensional reference features, image text category recognition can be performed, resulting in more accurate category recognition results. Furthermore, since this application can extract text detection features, text region information, and text recognition features corresponding to different feature extraction stages and perform image text recognition based on the extracted multi-dimensional features, compared to existing technologies that rely on OCR and other models to perform image text category recognition through the final features extracted from the image, it does not depend on the recognition capabilities of OCR or any complex post-processing techniques. It can complete the text category recognition work without relying on OCR's recognition capabilities or any complex post-processing techniques. It also avoids the influence of noise during the feature extraction stage on the text recognition results in the image, thus further improving the accuracy of text category recognition in the image.

[0131] In one exemplary embodiment of this application, the above-described apparatus further includes:

[0132] The text content acquisition unit is used to acquire the text content of the image to be recognized based on text detection features, text region information, and text recognition features before the classification unit 803 identifies the category of the text content in the image to be recognized based on multi-dimensional reference features.

[0133] As can be seen, by implementing this optional embodiment, the text content of the image to be recognized can be obtained, so that a recognition result that can more accurately represent the image to be recognized can be generated subsequently.

[0134] In one exemplary embodiment of this application, the first feature extraction unit 801 extracts text detection features, text region information, and text recognition features from the image to be recognized, including:

[0135] Text detection features, text region information, and text recognition features of the image to be recognized are extracted from each processing stage of the sequential arrangement of the text recognition model.

[0136] It is evident that implementing this optional embodiment can facilitate the joint application of text recognition models and category recognition models, thereby improving the accuracy of text category recognition.

[0137] In one exemplary embodiment of this application, the above-described apparatus further includes:

[0138] The recognition result generation unit is used to generate a recognition result corresponding to the image to be recognized based on the category and text content after the classification unit 803 identifies the category of the text content in the image to be recognized according to the multi-dimensional reference features.

[0139] As can be seen, implementing this optional embodiment can generate more representative recognition results based on category and text content.

[0140] In one exemplary embodiment of this application, the multi-dimensional reference features include a first reference feature, a second reference feature, and a third reference feature. The second feature extraction unit 802 performs structured processing on the text detection features, text region information, and text recognition features respectively to obtain the multi-dimensional reference features, including:

[0141] The text detection features and text region information are subjected to specific region pooling to obtain the first reference features;

[0142] The text region information is encoded with region features to obtain the second reference feature;

[0143] Text recognition features are encoded to obtain the third reference feature.

[0144] It is evident that implementing this optional embodiment can enhance the representation capability of an image based on the first reference feature, the second reference feature, and the third reference feature, which is beneficial for improving the recognition accuracy of the image.

[0145] In one exemplary embodiment of this application, the second feature extraction unit 802 performs specific region pooling processing on text detection features and text region information to obtain a first reference feature, including:

[0146] Determine the set of region coordinates based on the text region information;

[0147] The text detection features are subjected to specific region pooling based on the set of region coordinates to obtain the first reference feature.

[0148] As can be seen, by implementing this optional embodiment, the region pooling processing results corresponding to the text detection features and text region information can be obtained, and image text recognition based on the region pooling processing results can yield more accurate results.

[0149] In one exemplary embodiment of this application, the second feature extraction unit 802 performs region feature encoding on the text region information to obtain a second reference feature, including:

[0150] Regional feature encoding is performed on text region information to obtain the normalized centroid and normalized area corresponding to the text region information;

[0151] A second reference feature is generated based on the normalized centroid and normalized area.

[0152] As can be seen, by implementing this optional embodiment, the region feature encoding result corresponding to the text region information can be obtained, and image text recognition based on the region feature encoding result can yield more accurate results.

[0153] In one exemplary embodiment of this application, the second feature extraction unit 802 performs text feature encoding on the text recognition features to obtain a third reference feature, including:

[0154] The text recognition features are encoded using a text encoder that includes multiple encoders and multiple decoders to obtain the third reference feature.

[0155] As can be seen, by implementing this optional embodiment, the text feature encoding result corresponding to the text recognition feature can be obtained, and image text recognition based on the text feature encoding result can obtain more accurate results.

[0156] In one exemplary embodiment of this application, the classification unit 803 identifies the category of text content in the image to be identified based on multi-dimensional reference features, including:

[0157] The first reference feature, the second reference feature, and the third reference feature are fused to obtain the target feature.

[0158] The text content in the image to be identified is classified based on the target features to determine the category of the text content in the image to be identified.

[0159] As can be seen, implementing this optional embodiment can perform text category recognition in an image based on the fusion result of the first reference feature, the second reference feature, and the third reference feature, thereby improving the accuracy of the recognition result, reducing the impact of noise in the OCR feature extraction result on the recognition result, and improving the generalization ability of the image text recognition scheme.

[0160] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0161] Since the functional modules of the image text category recognition device in the example embodiments of this application correspond to the steps of the image text category recognition method in the example embodiments described above, for details not disclosed in the device embodiments of this application, please refer to the embodiments of the image text category recognition method described above in this application.

[0162] Please see Figure 9 , Figure 9 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown.

[0163] It should be noted that, Figure 9The computer system 900 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0164] like Figure 9 As shown, the computer system 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 902 or programs loaded from storage section 908 into random access memory (RAM) 903. The RAM 903 also stores various programs and data required for system operation. The CPU 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0165] The following components are connected to I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to I / O interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 910 as needed so that computer programs read from it can be installed into storage section 908 as needed.

[0166] Specifically, according to embodiments of this application, the processes described in the above-described flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 909, and / or installed from removable medium 911. When the computer program is executed by central processing unit (CPU) 901, it performs the various functions defined in the methods and apparatus of this application.

[0167] In another aspect, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods described in the above embodiments.

[0168] It should be noted that the computer-readable medium shown in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0169] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0170] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0171] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the foregoing claims.

Claims

1. A method for recognizing text categories in an image, characterized in that, include: Extract text detection features, text region information, and text recognition features from the image to be recognized; The text detection features, text region information, and text recognition features are respectively subjected to structured processing to obtain multi-dimensional reference features; wherein, the multi-dimensional reference features include a first reference feature, a second reference feature, and a third reference feature; obtaining the multi-dimensional reference features includes: performing specific region pooling processing on the text detection features and the text region information to obtain the first reference feature; performing region feature encoding on the text region information to obtain the second reference feature; and performing text feature encoding on the text recognition features to obtain the third reference feature; Identifying the category of text content in the image to be identified based on the multi-dimensional reference features includes: fusing the first reference feature, the second reference feature, and the third reference feature to obtain a target feature; and classifying the text content in the image to be identified based on the target feature to determine the category of the text content in the image to be identified. The extraction of text detection features, text region information, and text recognition features from the image to be recognized includes: extracting text detection features, text region information, and text recognition features from each of the sequential processing stages of the text recognition model; specifically, feature extraction is performed on the image to be recognized based on the convolutional layer of the backbone network of the text detection model in OCR, and further feature extraction is performed on the feature extraction results through the intermediate network of the text detection model to obtain text detection features; the text detection features are input into the content output network of the text detection model to obtain the text region information of the image to be recognized; feature extraction is performed on the text region information based on the convolutional layer of the backbone network of the text recognition model in OCR, and the extracted features are input into the intermediate network of the text recognition model to further extract the feature extraction results to obtain text recognition features.

2. The method according to claim 1, characterized in that, The method further includes: The text content of the image to be recognized is obtained based on the text detection features, the text region information, and the text recognition features.

3. The method according to claim 1, characterized in that, The text detection features and the text region information are subjected to specific region pooling processing to obtain the first reference feature, including: Determine the set of region coordinates based on the text region information; Based on the set of region coordinates, the text detection features are subjected to specific region pooling to obtain the first reference feature.

4. The method according to claim 1, characterized in that, The text region information is encoded using region feature encoding to obtain the second reference feature, including: The text region information is encoded with regional features to obtain the normalized centroid and normalized area corresponding to the text region information; The second reference feature is generated based on the normalized centroid and the normalized area.

5. The method according to claim 1, characterized in that, The text recognition features are encoded to obtain the third reference feature, including: The text recognition features are encoded using a text encoder that includes multiple encoders and multiple decoders to obtain the third reference feature.

6. A device for recognizing text categories in an image, characterized in that, include: The first feature extraction unit is used to extract text detection features, text region information, and text recognition features from the image to be recognized. The second feature extraction unit is used to perform structured processing on the text detection features, the text region information, and the text recognition features respectively to obtain multi-dimensional reference features; wherein, the multi-dimensional reference features include a first reference feature, a second reference feature, and a third reference feature; obtaining the multi-dimensional reference features includes: performing specific region pooling processing on the text detection features and the text region information to obtain the first reference feature; performing region feature encoding on the text region information to obtain the second reference feature; and performing text feature encoding on the text recognition features to obtain the third reference feature; A classification unit is configured to identify the category of text content in the image to be identified based on the multi-dimensional reference features, comprising: fusing the first reference feature, the second reference feature, and the third reference feature to obtain a target feature; and classifying the text content in the image to be identified based on the target feature to determine the category of the text content in the image to be identified. The extraction of text detection features, text region information, and text recognition features from the image to be recognized includes: extracting text detection features, text region information, and text recognition features from each of the sequential processing stages of the text recognition model; specifically, feature extraction is performed on the image to be recognized based on the convolutional layer of the backbone network of the text detection model in OCR, and further feature extraction is performed on the feature extraction results through the intermediate network of the text detection model to obtain text detection features; the text detection features are input into the content output network of the text detection model to obtain the text region information of the image to be recognized; feature extraction is performed on the text region information based on the convolutional layer of the backbone network of the text recognition model in OCR, and the extracted features are input into the intermediate network of the text recognition model to further extract the feature extraction results to obtain text recognition features.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-5.

8. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the method of any one of claims 1-5 by executing the executable instructions.

Images