Text line language recognition method, device, and electronic device

Through the branch structure of the deep learning network, text line detection and language recognition are integrated, which solves the problem of large error accumulation and calculation in the prior art, and improves recognition accuracy and efficiency.

CN113468979BActive Publication Date: 2024-12-06HANVON CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110650294.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-06-10
Publication Date
2024-12-06
Estimated Expiration
2041-06-10

AI Technical Summary

Technical Problem

In the prior art, text line detection and language recognition are processed as two independent tasks, which can easily lead to error accumulation, reduce the accuracy of language recognition, and increase the calculation amount.

Method used

The feature extraction module of the preset deep learning network performs feature extraction and fusion processing on text images, and combines the attention module to weight vector representations. The branch structure of the deep learning network simultaneously completes text line detection and language recognition, reducing error accumulation .

Benefits of technology

It improves the accuracy of the text line language recognition algorithm, reduces the amount of calculation, and completes text line detection and language recognition tasks at the same time during the training stage, greatly reducing time and maintenance costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113468979B_ABST
    Figure CN113468979B_ABST
Patent Text Reader

Abstract

The present application discloses a method for identifying the language of a text line, which belongs to the field of computer vision technology and helps to improve the accuracy of text line language recognition in an image. The method comprises: performing feature extraction and fusion processing on a target text image through a feature extraction module of a preset deep learning network to obtain a first vector representation; performing binary classification mapping on the first vector representation through a first classification module of the network to determine a binary probability prediction map of the target text image; based on the binary probability prediction map, performing weighted processing on the vector corresponding to the foreground pixel point in the first vector representation through an attention module of the network to obtain a second vector representation; performing multi-classification mapping on the second vector representation through a second classification module of the network to determine a multi-class probability prediction map; according to the multi-class probability prediction map, determining the position information of the text line matching each preset language in the target text image, thereby improving the accuracy of text line and language recognition in the image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer vision technology, and in particular, to a text line language recognition method, device, electronic device, and computer-readable storage medium. Background Art

[0002] Optical character recognition (OCR) is one of the current research hotspots in the field of computer vision. With the expansion of OCR technology application scenarios, images to be detected usually contain texts in multiple languages. For the recognition of multilingual text, it is necessary not only to locate the text line position on the image, but also to obtain the language of the corresponding text line. Only after the position and language of the text line are obtained can text recognition be performed and the text image can be converted into text information. In the prior art, text line detection and language recognition are usually treated as two independent tasks and processed in two stages. The first stage is general text line detection; the second stage is to crop the corresponding text line image according to the detected text line position and perform language recognition on it. When performing text line detection and language recognition based on deep learning methods, the prior art uses two independent models for text line detection and language recognition respectively, which is easy to cause error accumulation. The error of text line detection will affect the accuracy of language recognition, thereby reducing the accuracy of language recognition and increasing the amount of calculation.

[0003] It can be seen that the text image language recognition method in the prior art still needs to be improved. Summary of the invention

[0004] The embodiment of the present application provides a method for identifying the language of a text line, which helps to improve the accuracy of the language recognition algorithm of a text line in an image and reduce the amount of calculation.

[0005] In a first aspect, an embodiment of the present application provides a method for identifying a language of a text line, comprising:

[0006] Performing feature extraction and fusion processing on the target text image through a feature extraction module of a preset deep learning network to obtain a first vector representation of the target text image;

[0007] Performing binary classification mapping on the first vector representation through the first classification module of the deep learning network to determine a binary probability prediction map of the target text image, wherein the binary probability prediction map is used to indicate the probability that each pixel in the target text image belongs to a foreground pixel;

[0008] Based on the two-category probability prediction graph, weighting the first vector representation through the attention module of the deep learning network to obtain a second vector representation;

[0009] The second vector representation is subjected to multi-classification mapping by the second classification module of the deep learning network to determine a multi-class probability prediction map of the target text image; wherein the multi-class probability prediction map is used to indicate the probability value of each pixel point in the target text image matching a preset text attribute category, and the preset text attribute category includes: background and each preset language;

[0010] According to the probability value of each pixel point in the target text image matching the preset text attribute category indicated by the multi-category probability prediction map, the position information of the text line matching each of the preset languages ​​in the target text image is determined.

[0011] In a second aspect, an embodiment of the present application provides a text line language identification device, comprising:

[0012] A feature extraction module, used to perform feature extraction and fusion processing on the target text image through a feature extraction module of a preset deep learning network to obtain a first vector representation of the target text image;

[0013] A foreground-background prediction module, configured to perform binary classification mapping on the first vector representation through the first classification module of the deep learning network, and determine a binary probability prediction map of the target text image, wherein the binary probability prediction map is used to indicate the probability that each pixel in the target text image belongs to a foreground pixel;

[0014] A feature weighted processing module, configured to perform weighted processing on the first vector representation through an attention module of the deep learning network based on the two-category probability prediction graph to obtain a second vector representation;

[0015] A multi-category prediction module, used to perform multi-category mapping on the second vector representation through the second classification module of the deep learning network, and determine a multi-category probability prediction map of the target text image; wherein the multi-category probability prediction map is used to indicate the probability value of each pixel point in the target text image matching a preset text attribute category, and the preset text attribute category includes: background, each preset language;

[0016] The text line position and language determination module is used to determine the position information of the text lines matching each preset language in the target text image according to the probability value of each pixel point in the target text image matching the preset text attribute category indicated by the multi-category probability prediction map.

[0017] In a third aspect, an embodiment of the present application further discloses an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the text line language recognition method described in the embodiment of the present application when executing the computer program.

[0018] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the text line language recognition method disclosed in the embodiment of the present application.

[0019] The text line language recognition method disclosed in the embodiment of the present application performs feature extraction and fusion processing on the target text image through a feature extraction module of a preset deep learning network to obtain a first vector representation of the target text image; performs binary classification mapping on the first vector representation through a first classification module of the deep learning network to determine a two-category probability prediction map of the target text image, wherein the two-category probability prediction map is used to indicate the probability that each pixel point in the target text image belongs to a foreground pixel point; based on the two-category probability prediction map, the first vector representation is weighted through an attention module of the deep learning network to obtain a second vector representation ; The second vector representation is multi-classified and mapped by the second classification module of the deep learning network to determine a multi-class probability prediction map of the target text image; wherein the multi-class probability prediction map is used to indicate the probability value of each pixel point in the target text image matching a preset text attribute category, and the preset text attribute categories include: background and preset languages; according to the probability value of each pixel point in the target text image matching a preset text attribute category indicated by the multi-class probability prediction map, the position information of the text lines matching each of the preset languages ​​in the target text image is determined, thereby effectively improving the accuracy of the language recognition algorithm for text lines in the image and reducing the amount of calculation.

[0020] The text line language recognition method disclosed in the embodiment of the present application uses a deep learning network to simultaneously complete the two tasks of text line detection and language recognition, and obtain the position of the text line and the corresponding language. A branch network structure of the deep learning network is used to detect the text line of the input image, and a two-category prediction map is used to enhance the features of the first vector representation. Another branch network structure of the deep learning network is used to perform multi-category probability prediction of multiple languages ​​and backgrounds on the input image based on the enhanced features. The above method improves the accuracy of text line detection and language recognition. Moreover, in the training stage, only one training data set is needed to train a deep learning network to simultaneously complete the tasks of text line detection and language recognition, and obtain the text line position and the corresponding language detection results, which greatly reduces the time cost and maintenance cost of the training process.

[0021] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0023] Figure 1 is a flow chart of a method for identifying a language of a text line according to Embodiment 1 of the present application;

[0024] Figure 2 is a schematic diagram of text lines in a target text image in Example 1 of the present application;

[0025] Figure 3 This is one of the deep learning network structure diagrams in Example 1 of the present application;

[0026] Figure 4 This is the second schematic diagram of the deep learning network structure in the first embodiment of the present application;

[0027] Figure 5 This is the third schematic diagram of the deep learning network structure in the first embodiment of the present application;

[0028] Figure 6 This is one of the structural schematic diagrams of the text line language identification device of the second embodiment of the present application;

[0029] Figure 7 This is the second structural diagram of the text line language recognition device of the second embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0031] Embodiment 1

[0032] The present application discloses a method for identifying the language of a text line. Figure 1 As shown, the method includes: steps 110 to 150.

[0033] Step 110, performing feature extraction and fusion processing on the target text image through a feature extraction module of a preset deep learning network to obtain a first vector representation of the target text image.

[0034] In an embodiment of the present application, the target text image includes at least a scanned document image (such as a text image in a document) and a natural scene image (such as a text image in a natural scene). The scanned document image and the natural scene image can be regular text images and irregular text images, such as regular text such as horizontal or vertical text, and irregular text such as curved or inclined text, which exist in the text image.

[0035] In an embodiment of the present application, the position information of the text line is the position information of the text line in the text image represented based on the pixel coordinates, for example, the pixel position of the upper left corner pixel of the text line in the target text image and the position information represented by the pixel length and pixel width of the text line. In an embodiment, a rectangular area in the text image can be determined by the pixel points, pixel length and pixel width in the position information of the text line. In an embodiment of the present application, each text line corresponds to a language, and the position and matching language of one or more text lines included in the target text image are identified by a preset deep learning network.

[0036] by Figure 2 As an example, the target text image shown in FIG. 1 includes: text line 220, text lines 2101 to 2105, and a preset deep learning network is used to Figure 2 After the target text image shown in is recognized, the position information of each text line in the text line 220 and the text lines 2101 to 2105 will be obtained, and the preset language matched by the text line 220 and the text lines 2101 to 2105 will be obtained. For example, the position information of the Chinese text lines 220, 2101, 2103 and 2105 and the position information of the English text lines 2102 and 2104 can be obtained. Among them, the position information of each text line can be represented as a rectangular area by the pixel position of the upper left corner pixel point of the text line in the target text image and the pixel width and pixel height of the text line, and the rectangular area is the image area corresponding to the text line.

[0037] In the following, in combination with the structure of the deep learning network, a specific scheme for determining the position information and language of the text line in the target text image is described. In the embodiment of the present application, the deep learning network used for text line recognition adopts the following Figure 3 The network structure shown, wherein the deep learning network includes a feature extraction module 310, a first classification module 320, a second classification module 330 and an attention module 340.

[0038] The feature extraction module 310 is used to extract and fuse features of an input image (such as a target text image). In some embodiments of the present application, Figure 3As shown, the feature extraction module 310 may further include: a feature extraction submodule 3101 and a feature fusion submodule 3102. The feature extraction submodule 3101 is used to extract features of multiple different levels of an input image (such as a target text image). The feature extraction submodule 3101 may use common network structures including resnet, densenet, mobilenet, etc., but is not limited to these network structures. The feature fusion submodule 3102 is used to fuse the hierarchical features extracted by the feature extraction submodule 3101. The feature fusion submodule 3102 may use a feature fusion module commonly used in the prior art, such as a feature pyramid network FPN, etc., but is not limited to an FPN network. For an input image, the features of different levels of the input image extracted by the feature extraction submodule 3101 generally refer to feature maps of different resolutions of the input image. The low-resolution feature map is the high-level feature of the input image, and the high-resolution feature map is the low-level feature of the input image. Taking the use of a convolutional neural network (such as Resnet) to extract feature maps of four levels of an input image as an example, the feature extraction submodule 3101 can obtain feature maps of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 sizes of the input image. The feature fusion submodule 3102 fuses the obtained feature maps of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 sizes of the input image to obtain a feature map of 1 / 4 size of the input image as the first vector representation of the input image.

[0039] Step 120: Perform binary classification mapping on the first vector representation through the first classification module of the deep learning network to determine a binary probability prediction map of the target text image.

[0040] The two-category probability prediction map is used to indicate the probability that each pixel in the target text image belongs to a foreground pixel.

[0041] In some embodiments of the present application, the first classification module 320 can adopt a general convolutional neural network structure, for example, a convolutional neural network including a convolution layer, a batch normalization layer, an activation layer and a transposed convolution layer. The first classification module performs a binary classification mapping on the first vector representation to determine a two-category probability prediction map of the input image (such as a target text image). In some embodiments of the present application, the pixel value of each pixel point of the two-category probability prediction map is the probability value of the corresponding pixel point of the input image as a foreground pixel point, for example, the pixel value of each pixel point of the two-category probability prediction map is a value between 0 and 1.

[0042] Step 130: Based on the two-category probability prediction graph, the first vector representation is weightedly processed by the attention module of the deep learning network to obtain a second vector representation.

[0043] In some embodiments of the present application, based on the two-category probability prediction graph, the first vector representation is weightedly processed by the attention module of the deep learning network to obtain the second vector representation, including the following two schemes.

[0044] The first method is to use the pixel value of each pixel point in the two-category probability prediction map as a weight, and perform spatial weighted processing on the first vector representation through the attention module to obtain a second vector representation. For example, the attention module 340 includes a spatial attention submodule constructed based on the spatial attention mechanism, and the spatial attention submodule uses the pixel value of each pixel point in the two-category probability prediction map output by the first classification module 320 as a weight, and multiplies the pixel value of each pixel point in the two-category probability prediction map with the first vector representation output by the feature extraction module 310 pixel by pixel, and uses the multiplication result as the second vector representation. In the two-category probability prediction map, the pixel value of the pixel point in the text line area will be relatively high, and the pixel value of the pixel point in the non-text line area (i.e., the background area) will be relatively low. Therefore, the pixel value of each pixel point in the two-category probability prediction map is multiplied with the first vector representation to obtain the second vector representation. In this operation process, the features of the text line area in the first vector representation (i.e., the vector representation) will be highlighted, and the features of the non-text line area will be suppressed. When the second classification module 330 performs multi-category probability prediction based on the second vector representation, it is beneficial to improve the output of the second classification module 330 to be more accurate multi-category probability prediction map.

[0045] The second method is to use the pixel value of each pixel point in the two-category probability prediction map as a weight, and perform spatial weighted processing on the first vector representation through the attention module; then, the feature map obtained by the spatial weighted processing is input into the attention module, the weight corresponding to each channel is calculated, and then the feature map is channel-weighted with the calculated weight to obtain the second vector representation. For example, the attention module 340 includes: a spatial attention submodule constructed based on a spatial attention mechanism and a channel attention submodule constructed based on a channel attention mechanism. First, the pixel value of each pixel point in the two-category probability prediction map output by the first classification module 320 can be used as a weight through the spatial attention submodule, and the pixel value of each pixel point in the two-category probability prediction map is multiplied pixel by pixel with the first vector representation output by the feature extraction module 310; then, the result obtained after the pixel-by-pixel multiplication (i.e., the feature map) is input into the channel attention submodule, and the weight of each channel in the channel attention submodule is calculated; and the result obtained after the pixel-by-pixel multiplication is further weighted with the calculated channel weight, and the final result is used as the second vector representation.

[0046] In some embodiments of the present application, the channel attention submodule can adopt the SE-inseption model structure. Taking the vector representation obtained after the spatial weighted processing as a 640x640x3-dimensional vector as an example, if the calculated channel weight is [0.5, 0.3, 0.2], it means that the vector corresponding to the first channel (such as the first layer feature) is of the highest importance, and the vector corresponding to the third channel (such as the third layer feature) is of the lowest importance. The weight of each channel is multiplied by the vector representation of each channel obtained after the spatial weighted processing to obtain a new vector representation as the second vector representation.

[0047] The specific structure of the SE-inseption model structure can be found in the prior art and will not be described in detail in this embodiment. By further performing weighted processing on the vector representation after weighted processing based on the spatial attention mechanism, a second vector representation is obtained based on the channel attention mechanism, and the feature expression with high weight is highlighted in the channel direction. When the second classification module 330 performs multi-category probability prediction based on the second vector representation, the accuracy of the multi-category probability prediction map output by the second classification module 330 can be further improved.

[0048] Step 140: Perform multi-classification mapping on the second vector representation through the second classification module of the deep learning network to determine a multi-class probability prediction map of the target text image.

[0049] The multi-category probability prediction map is used to indicate the probability value of each pixel point in the target text image matching a preset text attribute category, and the preset text attribute category includes: background and each preset language.

[0050] The preset text attribute categories described in the embodiments of the present application are determined according to the language category of the text line to be detected, and the number of preset text attribute categories is the language category of the text line to be detected + 1. For example, when the language category of the text line to be detected includes Chinese and English, the number of the preset text attribute categories is 3, and the preset text attribute categories are used to indicate that a certain pixel point in the target text image is a background pixel point, a Chinese pixel point, or an English pixel point. In some embodiments of the present application, the second classification module 330 can adopt a general convolutional neural network structure, for example, a convolutional neural network including a convolutional layer, a batch normalization layer, an activation layer, and a transposed convolutional layer.

[0051] In some embodiments of the present application, taking the prediction of Chinese and English as an example, the multi-category probability prediction map can be represented as a multi-channel feature map (i.e., a multi-channel vector representation) of size H×W×C, wherein H and W represent the height and width of the multi-category probability prediction map, respectively, and C represents the depth of the feature map, and C is equal to 3. Among them, each channel data indicates the probability value of each pixel in the target text image belonging to a specified preset text attribute category. For example, the pixel value of each pixel of the multi-category probability prediction map can be represented by a multi-channel vector. For example, the pixel value of each pixel of the multi-category probability prediction map can be represented as (p1, p2, ..., pn), wherein p1+p2+ ... +pn = 1, p1, p2 to pn-1 are used to indicate the probability value of the language corresponding to the pixel, and pn is used to indicate the probability value of the pixel being a background pixel.

[0052] Step 150 , determining the position information of the text lines matching each of the preset languages ​​in the target text image according to the probability value of each pixel in the target text image matching the preset text attribute category indicated by the multi-category probability prediction map.

[0053] In some embodiments of the present application, each channel data in the multi-category probability prediction map respectively indicates the probability value of each pixel in the target text image matching the corresponding preset text attribute category, and determining the position information of the text line matching each preset language in the target text image according to the probability value of each pixel in the target text image matching the preset text attribute category indicated by the multi-category probability prediction map includes: determining the preset text attribute category matched by each pixel in the target text image according to the probability value indicated by each channel data in the multi-category probability prediction map; determining a multi-channel binary map according to the preset text attribute category matched by each pixel in the target text image, wherein each channel data in the multi-channel binary map respectively indicates the position information of a pixel of one of the preset text attribute categories in the target text image; and determining the position information of the text line matching each preset language in the target text image according to each channel data in the multi-channel binary map.

[0054] Taking the example that the pixel values ​​of each pixel point of the multi-channel feature map can be represented by three channel data (p1, p2, p3), if p1 represents the probability that the corresponding pixel point in the input image corresponding to the pixel point matches the Chinese pixel point, p2 represents the probability that the corresponding pixel point in the input image corresponding to the pixel point matches the English pixel point, and p3 represents the probability that the corresponding pixel point in the input image corresponding to the pixel point matches the background pixel point, for each pixel point, the preset text attribute category matched by the pixel point can be determined as Chinese, English or background based on the size relationship of the probability values ​​indicated in the above three channel data of the pixel point.

[0055] In some embodiments of the present application, the preset text attribute category matched by each pixel in the target text image is determined respectively according to the probability value indicated by each channel data in the multi-category probability prediction map, including: for each pixel in the target text image, the maximum probability value among the probability values ​​indicated by each channel data corresponding to the pixel is determined, and the preset text attribute category corresponding to the channel where the maximum probability value is located is used as the preset text attribute category matched by the pixel. Still taking the multi-channel data of each pixel in the multi-category probability prediction map as (p1, p2, p3) as an example, assuming that the pixel value of a certain pixel in the multi-category probability prediction map is (0.6, 0.3, 0.1), that is, p1 = 0.6, p2 = 0.3, p3 = 0.1, p1 is the largest, so the language category corresponding to p1 is determined to be the language category of this pixel, and the language category corresponding to this pixel is Chinese.

[0056] After determining the text attribute category matched by each pixel in the target text image, a multi-channel binary image is further determined according to the preset text attribute category matched by each pixel in the target text image. Taking the text attribute category recognition result in the aforementioned target text image as an example, a three-channel binary image corresponding to the target text image can be determined, wherein the binary image of the first channel indicates the position information of the Chinese pixel in the target text image, the binary image of the second channel indicates the position information of the English pixel in the target text image, and the binary image of the third channel indicates the position information of the background pixel in the target text image.

[0057] In some embodiments of the present application, taking the pixel value of each channel of the pixel point in the binary image obtained in the above steps as 1 or 0 as an example, the following method can be used to determine the multi-channel binary image according to the preset text attribute category matched by each pixel point in the target text image: for a certain pixel point in the target text image, if the text attribute category matched by the pixel point is Chinese, the pixel value of the first channel corresponding to the pixel point in the binary image is set to 1, and the pixel values ​​of the second and third channels are set to 0; if the text attribute category matched by the pixel point is English, the pixel value of the second channel corresponding to the pixel point in the binary image is set to 1, and the pixel values ​​of the first and third channels are set to 0; if the text attribute category matched by the pixel point is background, the pixel value of the third channel corresponding to the pixel point in the binary image is set to 1, and the pixel values ​​of the first and second channels are set to 0.

[0058] According to this method, a three-channel binary image can be obtained, and the pixel value of each pixel in the binary image includes the values ​​of three channels, and the values ​​of the three channels include one 1 and two 0s. It can be concluded from the above binary image generation method that the data of each channel in the binary image respectively represents the position information of the pixel of the text attribute category corresponding to the channel in the target text image. For example, when the pixel value corresponding to the pixel of the Ith row and Jth column in the first channel data of the binary image is 1, it means that the pixel of the Ith row and Jth column of the target text image is a Chinese pixel; when the pixel value corresponding to the pixel of the Ith row and J+10th column in the second channel data of the binary image is 1, it means that the pixel of the Ith row and J+10th column of the target text image is an English pixel; when the pixel value corresponding to the pixel of the Ith row and J+20th column in the third channel data of the binary image is 1, it means that the pixel of the Ith row and J+20th column of the target text image is a background pixel.

[0059] Next, according to the channel data corresponding to each preset language category in the binary image (such as the first channel data corresponding to Chinese and the second channel data corresponding to English), the position information of the text line of the preset language category corresponding to each channel is determined respectively.

[0060] In some embodiments of the present application, the position information of the text lines matching each preset language in the target text image is determined respectively according to each channel data in the multi-channel binary image, including: for each channel data matching the preset language in the multi-channel binary image, the image area corresponding to the text line matching the preset language in the target text image is determined by performing connectivity analysis on the position information of the pixel points of the corresponding preset language in the target text image indicated by the channel data. For example, for the first channel data of the aforementioned binary image, the connected domain of the binary image can be obtained by using the relevant algorithm in the prior art, and the minimum bounding rectangle of each connected domain can be solved. The minimum bounding rectangle of each connected domain represents the text line position information, and the position of each bounding rectangle obtained is the position of the image area corresponding to the Chinese text line in the target text image. Similarly, according to the second channel data of the aforementioned binary image, the position of the image area corresponding to the English text line in the target text image can be determined.

[0061] The text line language recognition method disclosed in the embodiment of the present application performs feature extraction and fusion processing on the target text image through a feature extraction module of a preset deep learning network to obtain a first vector representation of the target text image; performs binary classification mapping on the first vector representation through a first classification module of the deep learning network to determine a two-category probability prediction map of the target text image, wherein the two-category probability prediction map is used to indicate the probability that each pixel point in the target text image belongs to a foreground pixel point; based on the two-category probability prediction map, the first vector representation is weighted through an attention module of the deep learning network to obtain a second vector representation ; The second vector representation is multi-classified and mapped by the second classification module of the deep learning network to determine a multi-class probability prediction map of the target text image; wherein the multi-class probability prediction map is used to indicate the probability value of each pixel point in the target text image matching a preset text attribute category, and the preset text attribute categories include: background and preset languages; according to the probability value of each pixel point in the target text image matching a preset text attribute category indicated by the multi-class probability prediction map, the position information of the text lines matching each of the preset languages ​​in the target text image is determined, thereby effectively improving the accuracy of the language recognition algorithm for text lines in the image and reducing the amount of calculation.

[0062] The text line language recognition method disclosed in the embodiment of the present application uses a deep learning network to simultaneously complete the two tasks of text line detection and language recognition, and obtains the position of the text line and the corresponding language at the same time. A branch network structure of the deep learning network is used to detect the text line of the input image, and a two-category prediction map is used to enhance the features of the first vector representation. Another branch network structure of the deep learning network is used to perform multi-category probability prediction of multiple languages ​​and backgrounds on the input image based on the enhanced features. The above method improves the accuracy of text line detection and language recognition.

[0063] In order to make readers better understand the beneficial effects of the text language recognition method disclosed in this application, Figure 4 and Figure 5 The training scheme of the deep learning network is further explained so that readers can better understand the data processing process when the deep learning network is used for online prediction.

[0064] In some embodiments of the present application, the feature extraction module of the preset deep learning network is used to extract features and fuse the target text image to obtain the first vector representation of the target text image, and the method further includes: constructing a plurality of training samples; training a deep learning network including a feature extraction module, a first classification module, an attention module and a second classification module based on the training samples; wherein the sample data of each training sample is a text image including at least one text line, the language information matched by each text line and the position information in the text image, and each training sample matches at least the following two sample labels: a first label and a second label; the first label is used to indicate the corresponding sample number The second label is used to indicate the true value of each pixel in the corresponding sample data as a foreground pixel or a background pixel, and the second label is used to indicate the true value of each pixel in the corresponding sample data as a background pixel or a preset language pixel; the loss value of the deep learning network includes the loss value of the first classification module and the loss value of the second classification module, and the loss value of the first classification module is calculated based on the cross entropy of the two-category probability prediction map obtained by predicting the training samples by the first classification module and the first label of the corresponding training sample; the loss value of the second classification module is calculated based on the cross entropy of the multi-category probability prediction map obtained by predicting the training samples by the second classification module and the second label of the corresponding training sample.

[0065] See also Figure 4 The deep learning network structure shown in the figure comprises a backbone network structure (i.e., a feature extraction module 310) and two branch network structures (i.e., a branch network composed of a first classification module 320, and a branch network composed of an attention module 340 and a second classification module 330), and the inputs of the two branch network structures are both the outputs of the backbone network structure, and the two branch network structures have their own outputs. Therefore, in the process of constructing training samples of the deep learning network, for each training sample, its matching sample label includes two parts: the first label corresponding to the prediction result of the first classification module 320, and the second label corresponding to the prediction result of the second classification module 330.

[0066] In some embodiments of the present application, the language information and position information of each text line matched by the sample data of the training sample in the text image are the annotation information indicating the text line position and the corresponding language. In the model training process, the text line position information and the corresponding language information of the text image are used to generate the first label and the second label of the training sample matching corresponding to the text image. Among them, the text line position information and the corresponding language information in the text image are manually annotated information.

[0067] Take the sample data as Figure 2Taking the text image shown in as an example, the position information of text lines 220, 2101 to 2105 are manually annotated, and the corresponding languages ​​are annotated, such as text line 220 corresponds to Chinese, and text line 2102 corresponds to English. The first label matched by the text image is a binary segmentation map with the same size as the text image. The binary segmentation map separates the text area and the non-text area in the text image. For example, the pixel values ​​of the binary segmentation map are 0 and 1, where 1 indicates that the corresponding pixel in the text image corresponding to the pixel belongs to the text, and 0 indicates that the corresponding pixel in the text image corresponding to the pixel does not belong to the text (such as the background). In order to avoid the situation where some manually annotated text boxes are inaccurate and the boundaries of different text boxes are unclear due to the close distance, the present application adopts the method of shrinking the text area inward in the prior art to make the first label.

[0068] Still taking the sample data as Figure 2 Taking the text image shown in as an example, the position information and corresponding language information of text lines 220, 2101 to 2105 are manually annotated. The second label matched by the text image is the multivariate segmentation map of the text image. The pixel value of each pixel in the multivariate segmentation map represents the preset text attribute category matched by the corresponding pixel in the text image. Each pixel in the multivariate segmentation map corresponds one-to-one to a pixel of the text image in the sample data. For a text image with a size of H×W, the size of its multivariate segmentation map is H×W×C, where C is the number of language categories plus one. Taking the multivariate segmentation map as a ternary segmentation map as an example, the size of the input text image is 640x640, and when the languages ​​include Chinese and English, the size of the output multivariate segmentation map is 640x640x3. Each pixel point on the input text image corresponds to a three-dimensional vector at the corresponding position on the multivariate segmentation map. The values ​​of this vector include (1, 0, 0), (0, 1, 0), and (0, 0, 1). The three vector values ​​correspond to three different categories, where the category where 1 is located represents the language category or background of the pixel point. For example, if a pixel point in the text image is a Chinese character, the pixel value of the pixel point in the ternary segmentation map corresponding to the pixel point is expressed as (1, 0, 0); for another example, if a pixel point in the text image is an English character, the pixel value of the pixel point in the ternary segmentation map corresponding to the pixel point is expressed as (0, 1, 0); for another example, if a pixel point in the text image is the background, the pixel value of the pixel point in the ternary segmentation map corresponding to the pixel point is expressed as (0, 0, 1).

[0069] During the deep learning network training process, for each training sample, the following learning process is performed respectively: first, after feature extraction and fusion processing by the feature extraction module 310, the first vector representation of the training sample is obtained; then, the first classification module 320 further classifies and maps the first vector representation, and the first classification module 320 outputs a two-category probability prediction map; then, the attention module 340 performs weighted processing on the first vector representation based on the two-category probability prediction map to obtain a second vector representation, and the second classification module 330 further classifies and maps the second vector representation obtained after the weighted processing, and outputs a multi-category probability prediction map; then, the cross entropy between the two-category probability prediction map (i.e., the output of the first classification module 320) and the first label is calculated, and it is used as the first prediction loss value of the training sample; the cross entropy between the multi-category probability prediction map (i.e., the output of the second classification module 330) and the second label is calculated, and it is used as the second prediction loss value of the training sample. After predicting all training samples, the first prediction loss value and the second prediction loss value corresponding to each training sample will be obtained; finally, with the goal of minimizing the first prediction loss value and the second prediction loss value of all training samples (that is, with the goal of minimizing the loss value of the deep learning network (the sum of the first prediction loss value and the second prediction loss value)), the network parameters of the deep learning network are optimized, and the deep learning network is iteratively trained until the first prediction loss value and the second prediction loss value converge, that is, the loss value of the deep learning network converges, and the training process of the deep learning network is completed.

[0070] In some embodiments of the present application, when calculating the cross entropy between a two-category probability prediction map and a first label, the two-category probability prediction map is first binarized according to a preset threshold value to obtain a binary segmentation map corresponding to the two-category probability prediction map, and then the cross entropy between the binary segmentation map and the first label is calculated as the first prediction loss value of the training sample. Similarly, when calculating the cross entropy between a multi-category probability prediction map and a second label, the multi-category probability prediction map is first binarized to obtain a multivariate segmentation map corresponding to the multi-category probability prediction map (for the specific method, see the specific implementation method of generating the multivariate segmentation map mentioned above), and then the cross entropy between the multivariate segmentation map and the second label is calculated as the second prediction loss value of the training sample.

[0071] In some other embodiments of the present application, the sample data of each training sample is a text image including at least one text line, the language information matched by each text line and the position information in the text image, and each training sample matches at least the following three sample labels: a first label, a second label and a third label; the first label is used to indicate the true value of each pixel in the corresponding sample data as a foreground pixel or a background pixel, and the second label is used to indicate the true value of each pixel in the corresponding sample data as a background pixel or a pixel of a preset language; the third label is a threshold map of the corresponding sample data; a deep learning network is trained based on the training samples, and the deep learning network includes a feature extraction module, a first classification module, a second classification module, an attention module and a threshold module. A value map generation module; wherein the loss value of the deep learning network includes the loss value of the first classification module, the loss value of the second classification module, and the loss value of the threshold map generation module. The loss value of the first classification module is calculated based on the cross entropy of the two-category probability prediction map obtained by the first classification module predicting the training sample and the first label of the corresponding training sample; the loss value of the second classification module is calculated based on the cross entropy of the multi-category probability prediction map obtained by the second classification module predicting the training sample and the second label of the corresponding training sample; the loss value of the threshold map generation module is calculated based on the mean absolute error of the threshold map obtained by the threshold map generation module predicting the training sample and the third label of the corresponding training sample.

[0072] See also Figure 5 The deep learning network structure shown in the figure, in other embodiments of the present application, the deep learning network includes a trunk network structure (i.e., a feature extraction module 310) and three branch network structures (i.e., a branch network composed of a first classification module 320, a branch network composed of an attention module 340 and a second classification module 330, and a branch network composed of a threshold map generation module 350), and the inputs of the three branch network structures all include the output of the trunk network structure, and the three branch network structures have their own outputs. Therefore, in the process of constructing training samples of the deep learning network, for each training sample, its matching sample label includes three parts: a first label corresponding to the prediction result of the first classification module 320, a second label corresponding to the prediction result of the second classification module 330, and a third label corresponding to the prediction result of the threshold map generation module 350.

[0073] The meaning and generation method of the first label and the second label refer to the description in the previous paragraph and will not be repeated here.

[0074] The third label is the threshold map of the text image as sample data. The threshold map can be generated using a method similar to the first label. For example, the text box in the text image is expanded inward and outward by 0.4 times to obtain a contraction box and an expansion box, and then the normalized distance D between each pixel point in the area between the contraction box and the expansion box and the nearest text box boundary is calculated, and 1-D is used as the pixel value of the threshold map at this point. At the same time, the maximum and minimum values ​​of the pixels in the threshold map are set. For example, the maximum value of the pixels in the threshold map can be set to 0.7, and the minimum value of the pixels in the threshold map can be set to 0.3.

[0075] In the deep learning network training process, for each training sample, the following learning process is performed respectively: first, after feature extraction and fusion processing by the feature extraction module 310, a first vector representation of the training sample is obtained; then, the first classification module 320 and the threshold map generation module 350 further classify and map the first vector representation respectively, the first classification module 320 will output a two-category probability prediction map, and the threshold map generation module 350 will output a threshold map; then, the attention module 340 performs weighted processing on the first vector representation based on the two-category probability prediction map to obtain a second vector representation, and the second classification module 330 further performs classification mapping on the second vector representation obtained after the weighted processing, and outputs a multi-category probability prediction map; then, the cross entropy between the two-category probability prediction map (i.e., the output of the first classification module 320) and the first label is calculated, and it is used as the first prediction loss value of the training sample; the cross entropy between the multi-category probability prediction map (i.e., the output of the second classification module 330) and the second label is calculated, and it is used as the second prediction loss value of the training sample; the mean absolute error (i.e., L1 loss), and use it as the third predicted loss value of the training sample. After predicting all training samples, the first predicted loss value, the second predicted loss value, and the third predicted loss value corresponding to each training sample will be obtained; finally, with the goal of minimizing the first predicted loss value, the second predicted loss value, and the third predicted loss value of all training samples (that is, with the goal of minimizing the loss value of the deep learning network (the sum of the first predicted loss value, the second predicted loss value, and the third predicted loss value)), the network parameters of the deep learning network are optimized, and the deep learning network is iteratively trained until the first predicted loss value, the second predicted loss value, and the third predicted loss value converge, that is, the loss value of the deep learning network converges, and the training process of the deep learning network is completed.

[0076] In this case, when calculating the cross entropy between the two-category probability prediction map and the first label, the threshold map and the two-category probability prediction map are firstly subjected to differentiable binarization processing to obtain a binary segmentation map corresponding to the two-category probability prediction map, and then the cross entropy between the binary segmentation map and the first label is calculated as the first prediction loss value of the training sample. Similarly, when calculating the cross entropy between the multi-category probability prediction map and the second label, the multi-category probability prediction map is firstly subjected to binarization processing to obtain a multivariate segmentation map corresponding to the multi-category probability prediction map (for the specific method, see the specific implementation method of generating the multivariate segmentation map mentioned above), and then the cross entropy between the multivariate segmentation map and the second label is calculated as the second prediction loss value of the training sample.

[0077] By adding a threshold map during the training phase to constrain the process of calculating the position information of the text line using the two-category probability prediction map, the accuracy of the output results of the first classification module 320 can be improved, further improving the accuracy of the text line and language recognition results.

[0078] In some embodiments of the present application, the feature extraction module 310 and the first classification module 320 may be trained first, and then, based on the parameters of the feature extraction module 310 obtained through the training, the second classification module 330 and the threshold map generation module 350 may be further trained, which will not be described in detail in the embodiments of the present application.

[0079] During the online recognition process, the model trained on the target text image will output text lines divided into different languages.

[0080] In some other embodiments of the present application, a complete line of text containing mixed languages ​​may also be marked as one text line, and the text line may be segmented later, but the present application does not limit this.

[0081] The text line language recognition method disclosed in the embodiment of the present application only needs to use one training data set to train a deep learning network during the deep learning network training stage, so as to simultaneously complete the tasks of text line detection and language recognition, and obtain the text line position and the corresponding language detection result, which greatly reduces the time cost and maintenance cost of the training process.

[0082] On the other hand, the text line language recognition method disclosed in the embodiment of the present application, during the deep learning network training stage, independently determines the text line position information in the target text image, and determines the probability value of the preset text attribute category matching each pixel point in the target text image, and fuses the text line position information and the text category attribute to perform language recognition in the text line. This avoids the problem of error accumulation caused by inaccurate determination of text line position information when first identifying the text line position information in the target text image and then performing language recognition for the specified text line based on the text line position information in the prior art, thereby reducing the accuracy of text line language recognition. This further improves the data processing accuracy of the feature extraction module, the first classification module, the attention module, and the second classification module in the trained deep learning network, thereby improving the accuracy of text line position information and language recognition.

[0083] Embodiment 2

[0084] The embodiment of the present application discloses a text line language recognition device, such as Figure 6 As shown, the device comprises:

[0085] A feature extraction module 610 is used to perform feature extraction and fusion processing on the target text image through a feature extraction module of a preset deep learning network to obtain a first vector representation of the target text image;

[0086] A foreground-background prediction module 620 is used to perform a binary classification mapping on the first vector representation through the first classification module of the deep learning network to determine a binary probability prediction map of the target text image, wherein the binary probability prediction map is used to indicate the probability that each pixel in the target text image belongs to a foreground pixel;

[0087] A feature weighted processing module 630, configured to perform weighted processing on the first vector representation through an attention module of the deep learning network based on the two-category probability prediction graph to obtain a second vector representation;

[0088] A multi-class prediction module 640 is used to perform multi-class mapping on the second vector representation through the second classification module of the deep learning network to determine a multi-class probability prediction map of the target text image; wherein the multi-class probability prediction map is used to indicate the probability value of each pixel point in the target text image matching a preset text attribute category, and the preset text attribute categories include: background and each preset language;

[0089] The text line position and language determination module 650 is used to determine the position information of the text lines matching each preset language in the target text image according to the probability value of each pixel point in the target text image matching the preset text attribute category indicated by the multi-category probability prediction map.

[0090] In some embodiments of the present application, the feature weighting processing module 630 is further used to:

[0091] Using the pixel value of each pixel point of the two-category probability prediction map as a weight, the first vector representation is spatially weighted by the attention module to obtain a second vector representation; or,

[0092] The first vector representation is spatially weighted by the attention module, with the pixel value of each pixel point of the two-category probability prediction map as the weight; then, the feature map obtained by the spatial weighted processing is input into the attention module, the weight corresponding to each channel is calculated, and then the feature map is channel-weighted by the calculated weight to obtain the second vector representation.

[0093] In some embodiments of the present application, each channel data in the multi-category probability prediction map indicates the probability value of each pixel in the target text image matching the corresponding preset text attribute category, and the text line position and language determination module 650 is further used to:

[0094] Determining the preset text attribute category matched by each pixel point in the target text image according to the probability value indicated by each channel data in the multi-category probability prediction map;

[0095] According to the preset text attribute category matched by each pixel point in the target text image, a multi-channel binary image is determined, wherein each channel data in the multi-channel binary image respectively indicates the position information of a pixel point of the preset text attribute category in the target text image;

[0096] According to each channel data in the multi-channel binary image, position information of text lines matching each of the preset languages ​​in the target text image is determined respectively.

[0097] In some embodiments of the present application, determining the position information of the text lines matching each of the preset languages ​​in the target text image according to each channel data in the multi-channel binary image includes:

[0098] For each channel data matching the preset language in the multi-channel binary image, connectivity analysis is performed on the position information of the pixel points corresponding to the preset language in the target text image indicated by the channel data, so as to determine the image area corresponding to the text line matching the preset language in the target text image.

[0099] In some embodiments of the present application, determining the preset text attribute category matched by each pixel point in the target text image according to the probability value indicated by each channel data in the multi-category probability prediction map includes:

[0100] For each pixel in the target text image, the maximum probability value among the probability values ​​indicated by the channel data corresponding to the pixel is determined, and the preset text attribute category corresponding to the channel where the maximum probability value is located is used as the preset text attribute category matched by the pixel.

[0101] In some embodiments of the present application, Figure 7 As shown, the device also includes:

[0102] The training sample construction module 660 is used to construct a plurality of training samples; wherein the sample data of each training sample is a text image including at least one text line, language information matched by each text line and position information in the text image, and each training sample matches at least the following two sample labels: a first label and a second label; the first label is used to indicate the true value of each pixel in the corresponding sample data belonging to a foreground pixel or a background pixel, and the second label is used to indicate the true value of each pixel in the corresponding sample data being a background pixel or a preset language pixel;

[0103] A model training module 670 is used to train a deep learning network based on the training samples, wherein the deep learning network includes a feature extraction module, a first classification module, an attention module, and a second classification module; wherein the loss value of the deep learning network includes the loss value of the first classification module and the loss value of the second classification module, and the loss value of the first classification module is calculated based on the cross entropy of the two-category probability prediction graph obtained by predicting the training samples by the first classification module and the first label of the corresponding training sample; the loss value of the second classification module is calculated based on the cross entropy of the multi-category probability prediction graph obtained by predicting the training samples by the second classification module and the second label of the corresponding training sample.

[0104] In some embodiments of the present application, the sample data of each training sample is a text image including at least one text line, language information matching each text line and position information in the text image, and each training sample matches at least the following three sample labels: a first label, a second label and a third label; the first label is used to indicate the true value of each pixel in the corresponding sample data as a foreground pixel or a background pixel, and the second label is used to indicate the true value of each pixel in the corresponding sample data as a background pixel or a pixel of a preset language; the third label is a threshold map of the corresponding sample data;

[0105] In some embodiments of the present application, the model training module 670 is also used to train a deep learning network based on the training sample, and the deep learning network includes a feature extraction module, a first classification module, a second classification module, an attention module and a threshold map generation module; wherein the loss value of the deep learning network includes the loss value of the first classification module, the loss value of the second classification module, and the loss value of the threshold map generation module, the loss value of the first classification module is calculated according to the cross entropy of the two-category probability prediction map obtained by the first classification module predicting the training sample and the first label of the corresponding training sample; the loss value of the second classification module is calculated according to the cross entropy of the multi-category probability prediction map obtained by the second classification module predicting the training sample and the second label of the corresponding training sample; the loss value of the threshold map generation module is calculated according to the mean absolute error of the threshold map obtained by the threshold map generation module predicting the training sample and the third label of the corresponding training sample. The text line language recognition device disclosed in the embodiment of the present application is used to implement the text line language recognition method described in the first embodiment of the present application. The specific implementation of each module of the device is not repeated here, and the specific implementation of the corresponding steps of the method embodiment can be referred to.

[0106] The text line language recognition device disclosed in the embodiment of the present application performs feature extraction and fusion processing on the target text image through a feature extraction module of a preset deep learning network to obtain a first vector representation of the target text image; performs binary classification mapping on the first vector representation through a first classification module of the deep learning network to determine a two-category probability prediction map of the target text image, wherein the two-category probability prediction map is used to indicate the probability that each pixel point in the target text image belongs to a foreground pixel point; based on the two-category probability prediction map, the first vector representation is weighted through an attention module of the deep learning network to obtain a second vector representation ; The second vector representation is multi-classified and mapped by the second classification module of the deep learning network to determine a multi-class probability prediction map of the target text image; wherein the multi-class probability prediction map is used to indicate the probability value of each pixel point in the target text image matching a preset text attribute category, and the preset text attribute categories include: background and preset languages; according to the probability value of each pixel point in the target text image matching a preset text attribute category indicated by the multi-class probability prediction map, the position information of the text lines matching each of the preset languages ​​in the target text image is determined, thereby effectively improving the accuracy of the language recognition algorithm for text lines in the image and reducing the amount of calculation.

[0107] The text line language recognition device disclosed in the embodiment of the present application simultaneously completes the two tasks of text line detection and language recognition through a deep learning network, and obtains the position of the text line and the corresponding language at the same time. A branch network structure of the deep learning network is used to detect the text line of the input image, and a two-category prediction map is used to enhance the features of the first vector representation. Another branch network structure of the deep learning network is used to perform multi-category probability prediction of multiple languages ​​and backgrounds on the input image based on the enhanced features. The above method improves the accuracy of text line detection and language recognition.

[0108] The text line language recognition device disclosed in the embodiment of the present application only needs to use one training data set to train a deep learning network during the deep learning network training stage, so as to simultaneously complete the tasks of text line detection and language recognition, and obtain the text line position and the corresponding language detection result, which greatly reduces the time cost and maintenance cost of the training process.

[0109] On the other hand, the text line language recognition device disclosed in the embodiment of the present application, during the deep learning network training stage, independently determines the text line position information in the target text image, and determines the probability value of the preset text attribute category matching each pixel point in the target text image, and fuses the text line position information and the text category attribute to perform language recognition in the text line. This avoids the problem of error accumulation caused by inaccurate determination of text line position information when first identifying the text line position information in the target text image and then performing language recognition for the specified text line based on the text line position information in the prior art, thereby reducing the accuracy of text line language recognition. This further improves the data processing accuracy of the feature extraction module, the first classification module, the attention module, and the second classification module in the trained deep learning network, thereby improving the accuracy of text line position information and language recognition.

[0110] Accordingly, the present application also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the text line language recognition method as described in Embodiment 1 of the present application when executing the computer program. The electronic device may be a PC, a mobile terminal, a personal digital assistant, a tablet computer, etc.

[0111] The present application also discloses a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the text line language recognition method as described in the first embodiment of the present application are implemented.

[0112] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0113] The above is a detailed introduction to a text line language recognition method and device provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0114] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0115] The term "one embodiment", "embodiment" or "one or more embodiments" herein means that a particular feature, structure or characteristic described in conjunction with the embodiment is included in at least one embodiment of the present application. In addition, please note that the examples of the term "in one embodiment" here do not necessarily all refer to the same embodiment.

[0116] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0117] In the claims, any reference signs placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application may be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for identifying the language of a text line, It is characterized in that include: Performing feature extraction and fusion processing on the target text image through a feature extraction module of a preset deep learning network to obtain a first vector representation of the target text image; Performing binary classification mapping on the first vector representation through the first classification module of the deep learning network to determine a binary probability prediction map of the target text image, wherein the binary probability prediction map is used to indicate the probability that each pixel in the target text image belongs to a foreground pixel; Based on the two-category probability prediction graph, weighting the first vector representation through the attention module of the deep learning network to obtain a second vector representation; The second vector representation is subjected to multi-classification mapping by the second classification module of the deep learning network to determine a multi-class probability prediction map of the target text image; wherein the multi-class probability prediction map is used to indicate the probability value of each pixel point in the target text image matching a preset text attribute category, and the preset text attribute category includes: background and each preset language; According to the probability value of each pixel point in the target text image matching the preset text attribute category indicated by the multi-category probability prediction map, the position information of the text line matching each of the preset languages ​​in the target text image is determined.

2. The method according to claim 1, It is characterized in that The step of performing weighted processing on the first vector representation based on the two-category probability prediction graph by using an attention module of the deep learning network to obtain a second vector representation includes: Using the pixel value of each pixel point of the two-category probability prediction map as a weight, the first vector representation is spatially weighted by the attention module to obtain a second vector representation; or, The first vector representation is spatially weighted by the attention module, with the pixel value of each pixel point of the two-category probability prediction map as the weight; then, the feature map obtained by the spatial weighted processing is input into the attention module, the weight corresponding to each channel is calculated, and then the feature map is channel-weighted by the calculated weight to obtain the second vector representation.

3. The method according to claim 1, It is characterized in that Each channel data in the multi-category probability prediction map indicates a probability value of each pixel in the target text image matching a corresponding preset text attribute category, and determining position information of text lines matching each preset language in the target text image according to the probability value of each pixel in the target text image matching the preset text attribute category indicated by the multi-category probability prediction map includes: Determining the preset text attribute category matched by each pixel point in the target text image according to the probability value indicated by each channel data in the multi-category probability prediction map; According to the preset text attribute category matched by each pixel point in the target text image, a multi-channel binary image is determined, wherein each channel data in the multi-channel binary image respectively indicates the position information of a pixel point of the preset text attribute category in the target text image; According to each channel data in the multi-channel binary image, position information of text lines matching each of the preset languages ​​in the target text image is determined respectively.

4. The method according to claim 3, It is characterized in that The step of respectively determining the position information of the text lines matching each of the preset languages ​​in the target text image according to each channel data in the multi-channel binary image comprises: For each channel data matching the preset language in the multi-channel binary image, connectivity analysis is performed on the position information of the pixel points corresponding to the preset language in the target text image indicated by the channel data, so as to determine the image area corresponding to the text line matching the preset language in the target text image.

5. The method according to claim 3, It is characterized in that The step of determining the preset text attribute category matched by each pixel point in the target text image according to the probability value indicated by each channel data in the multi-category probability prediction map comprises: For each pixel in the target text image, the maximum probability value among the probability values ​​indicated by the channel data corresponding to the pixel is determined, and the preset text attribute category corresponding to the channel where the maximum probability value is located is used as the preset text attribute category matched by the pixel.

6. The method according to any one of claims 1 to 5, It is characterized in that Before the step of performing feature extraction and fusion processing on the target text image by using a feature extraction module of a preset deep learning network to obtain a first vector representation of the target text image, the step further includes: Constructing a plurality of training samples; wherein the sample data of each training sample is a text image including at least one text line, language information matched by each text line and position information in the text image, and each training sample matches at least the following two sample labels: a first label and a second label; the first label is used to indicate the true value of each pixel in the corresponding sample data belonging to a foreground pixel or a background pixel, and the second label is used to indicate the true value of each pixel in the corresponding sample data being a background pixel or a pixel of a preset language; A deep learning network is trained based on the training samples, and the deep learning network includes a feature extraction module, a first classification module, an attention module, and a second classification module; wherein the loss value of the deep learning network includes the loss value of the first classification module and the loss value of the second classification module, and the loss value of the first classification module is calculated based on the cross entropy of the two-category probability prediction map obtained by predicting the training samples by the first classification module and the first label of the corresponding training sample; the loss value of the second classification module is calculated based on the cross entropy of the multi-category probability prediction map obtained by predicting the training samples by the second classification module and the second label of the corresponding training sample.

7. The method according to any one of claims 1 to 5, It is characterized in that Before the step of performing feature extraction and fusion processing on the target text image by using a feature extraction module of a preset deep learning network to obtain a first vector representation of the target text image, the step further includes: Constructing a plurality of training samples; wherein the sample data of each training sample is a text image including at least one text line, language information matched by each text line and position information in the text image, and each training sample matches at least the following three sample labels: a first label, a second label and a third label; the first label is used to indicate the true value of each pixel in the corresponding sample data belonging to a foreground pixel or a background pixel, and the second label is used to indicate the true value of each pixel in the corresponding sample data being a background pixel or a pixel of a preset language; the third label is a threshold map of the corresponding sample data; A deep learning network is trained based on the training samples, and the deep learning network includes a feature extraction module, a first classification module, a second classification module, an attention module and a threshold map generation module; wherein the loss value of the deep learning network includes the loss value of the first classification module, the loss value of the second classification module, and the loss value of the threshold map generation module, the loss value of the first classification module is calculated based on the cross entropy of the two-category probability prediction map obtained by predicting the training samples by the first classification module and the first label of the corresponding training sample; the loss value of the second classification module is calculated based on the cross entropy of the multi-category probability prediction map obtained by predicting the training samples by the second classification module and the second label of the corresponding training sample; the loss value of the threshold map generation module is calculated based on the mean absolute error between the threshold map obtained by predicting the training samples by the threshold map generation module and the third label of the corresponding training sample.

8. A text line language recognition device, It is characterized in that include: A feature extraction module, used to perform feature extraction and fusion processing on the target text image through the feature extraction module of the pre-trained deep learning network to obtain a first vector representation of the target text image; A foreground-background prediction module, configured to perform binary classification mapping on the first vector representation through the first classification module of the deep learning network, and determine a binary probability prediction map of the target text image, wherein the binary probability prediction map is used to indicate the probability that each pixel in the target text image belongs to a foreground pixel; A feature weighted processing module, configured to perform weighted processing on the first vector representation through an attention module of the deep learning network based on the two-category probability prediction graph to obtain a second vector representation; A multi-category prediction module, used to perform multi-category mapping on the second vector representation through the second classification module of the deep learning network, and determine a multi-category probability prediction map of the target text image; wherein the multi-category probability prediction map is used to indicate the probability value of each pixel point in the target text image matching a preset text attribute category, and the preset text attribute category includes: background, each preset language; The text line position and language determination module is used to determine the position information of the text lines matching each preset language in the target text image according to the probability value of each pixel point in the target text image matching the preset text attribute category indicated by the multi-category probability prediction map.

9. An electronic device comprising a memory, a processor, and a program code stored in the memory and executable on the processor, It is characterized in that When the processor executes the program code, the text line language recognition method described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having program code stored thereon, It is characterized in that When the program code is executed by a processor, the steps of the text line language recognition method described in any one of claims 1 to 7 are implemented.