Text line language recognition method, device, and electronic device

Abstract

The present application discloses a text line language recognition method, which belongs to the field of computer vision technology and helps to improve the accuracy of text line language recognition in an image. The method comprises: respectively determining the text line position information in a target text image and the probability value of a preset text attribute category matched by each pixel point, wherein the preset text attribute category is used to indicate that the corresponding pixel point is a background pixel point or a preset language pixel point; according to the text line position information, determining the image area corresponding to each text line included in the target text image; according to the probability value of the preset text attribute category matched by each pixel point in the image area, respectively determining the number of pixels of each preset language in each image area; for the image area corresponding to each text line, determining the language matched by the text line according to the number of pixels of each preset language in the image area, which not only improves the accuracy of the text line language recognition algorithm in the image, but also reduces the amount of calculation.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer vision technology, and in particular, to a text line language recognition method, device, electronic device, and computer-readable storage medium. Background Art

[0002] Optical character recognition (OCR) is one of the current research hotspots in the field of computer vision. With the expansion of OCR technology application scenarios, images to be detected usually contain texts in multiple languages. For the recognition of multilingual text, it is necessary not only to locate the text line position on the image, but also to obtain the language of the corresponding text line. Only after the position and language of the text line are obtained can text recognition be performed and the text image can be converted into text information. In the prior art, text line detection and language recognition are usually treated as two independent tasks and processed in two stages. The first stage is general text line detection; the second stage is to crop the corresponding text line image according to the detected text line position and perform language recognition on it. When performing text line detection and language recognition based on deep learning methods, the prior art uses two independent models for text line detection and language recognition respectively, which is easy to cause error accumulation. The error of text line detection will affect the accuracy of language recognition, thereby reducing the accuracy of language recognition and increasing the amount of calculation.

[0003] It can be seen that the text image language recognition method in the prior art still needs to be improved. Summary of the invention

[0004] The embodiment of the present application provides a method for identifying the language of a text line, which helps to improve the accuracy of the language recognition algorithm of a text line in an image and reduce the amount of calculation.

[0005] In a first aspect, an embodiment of the present application provides a method for identifying a language of a text line, comprising:

[0006] Determine text line position information in a target text image, and determine a probability value of a preset text attribute category matched by each pixel in the target text image, wherein the preset text attribute category is used to indicate that the corresponding pixel is a background pixel or a preset language pixel;

[0007] Determining, according to the text line position information, an image area corresponding to each text line included in the target text image;

[0008] Determining the number of pixels of each preset language in each of the image regions according to the probability value of the preset text attribute category matched by each pixel in the image region;

[0009] For the image area corresponding to each text line, the language matched by the text line is determined according to the number of pixels of each preset language in the image area.

[0010] In a second aspect, an embodiment of the present application provides a text line language identification device, comprising:

[0011] A text line position information and pixel classification probability value determination module, used to determine the text line position information in the target text image, and to determine the probability value of a preset text attribute category matched by each pixel in the target text image, wherein the preset text attribute category is used to indicate that the corresponding pixel is a background pixel or a preset language pixel;

[0012] A text line corresponding image area determination module, used to determine the image area corresponding to each text line included in the target text image according to the text line position information;

[0013] A module for determining the number of pixels of different languages, used to determine the number of pixels of each preset language in each of the image regions according to the probability value of the preset text attribute category matched by each pixel in the image region;

[0014] The text line language determination module is used to determine the language matched by each text line according to the number of pixels of each preset language in the image area corresponding to each text line.

[0015] In a third aspect, an embodiment of the present application further discloses an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the text line language recognition method described in the embodiment of the present application when executing the computer program.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the text line language recognition method disclosed in the embodiment of the present application.

[0017] The text line language recognition method disclosed in the embodiment of the present application determines the text line position information in the target text image, and determines the probability value of the preset text attribute category matched by each pixel point in the target text image, wherein the preset text attribute category is used to indicate that the corresponding pixel point is a background pixel point or a preset language pixel point; according to the text line position information, the image area corresponding to each text line included in the target text image is determined; according to the probability value of the preset text attribute category matched by each pixel point in the image area, the number of each preset language pixel point in each of the image areas is determined respectively; for the image area corresponding to each text line, the language matched by the text line is determined according to the number of each preset language pixel point in the image area, thereby effectively improving the accuracy of the text line language recognition algorithm in the image and reducing the amount of calculation.

[0018] On the other hand, the text line language recognition method disclosed in the embodiment of the present application uses a deep learning network to simultaneously complete the two tasks of text line detection and language recognition, and obtain the position of the text line and the corresponding language. A branch network structure of the deep learning network is used to detect the text line of the input image, and another branch network structure of the deep learning network is used to perform multi-category probability prediction on the input image. The text line detection results and the multi-category probability prediction results are combined to classify the language of each text line. The above method improves the accuracy of text line detection and language recognition. Moreover, in the training stage, only one training data set is needed to train a deep learning network, so that the tasks of text line detection and language recognition can be completed simultaneously, and the text line position and the corresponding language detection results can be obtained at the same time, which greatly reduces the time cost and maintenance cost of the training process.

[0019] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0021] Figure 1 is a flow chart of a method for identifying a language of a text line according to Embodiment 1 of the present application;

[0022] Figure 2 is a schematic diagram of text lines in a target text image in Example 1 of the present application;

[0023] Figure 3 This is one of the deep learning network structure diagrams in Example 1 of the present application;

[0024] Figure 4 This is the second schematic diagram of the deep learning network structure in the first embodiment of the present application;

[0025] Figure 5 This is one of the structural schematic diagrams of the text line language identification device of the second embodiment of the present application;

[0026] Figure 6 This is the second structural diagram of the text line language recognition device of the second embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0028] Embodiment 1

[0029] The present application discloses a method for identifying the language of a text line. Figure 1 As shown, the method includes: steps 110 to 140.

[0030] Step 110, determining text line position information in a target text image, and determining a probability value of a preset text attribute category matched by each pixel point in the target text image.

[0031] The preset text attribute category is used to indicate that the corresponding pixel is a background pixel or a preset language pixel.

[0032] In an embodiment of the present application, the target text image at least includes a scanned document image (such as a text image in a document) and a natural scene image (such as a text image in a natural scene). The scanned document image and the natural scene image can be regular text images and irregular text images, such as regular text such as horizontal or vertical text, and irregular text such as curved and inclined text in the text image; the text line position information is the position information of the text line represented based on the pixel coordinates, for example, the pixel position of the upper left corner pixel of the text line in the target text image and the position information represented by the pixel length and pixel width of the text line. In one embodiment, the pixel points, pixel length and pixel width in the text line position information can determine a rectangular area. In an embodiment of the present application, taking each text line corresponding to a language as an example, the specific implementation form will be described in the training scheme of the deep learning network later.

[0033] by Figure 2 Taking the target text image shown in as an example, the target text image includes: text line 220, text lines 2101 to 2105, and the position information of each text line can be represented based on the pixel position of the pixel point in the target text image and the pixel length and pixel width of the text line.

[0034] The preset text attribute categories described in the embodiments of the present application are determined according to the language category of the text line to be detected, and the number of preset text attribute categories is the language category of the text line to be detected + 1. For example, when the language category of the text line to be detected includes Chinese and English, the number of the preset text attribute categories is 3, and the preset text attribute categories are used to indicate that a certain pixel point in the target text image is a background pixel point, a Chinese pixel point, or an English pixel point.

[0035] Step 120: Determine the image area corresponding to each text line included in the target text image according to the text line position information.

[0036] As mentioned above, the text line position information can be represented as a rectangular area by the pixel position of the upper left corner pixel of the text line in the target text image and the pixel width and pixel height of the text line. The rectangular area is the image area corresponding to the text line.

[0037] Step 130 , determining the number of pixels of each preset language in each of the image regions according to the probability value of the preset text attribute category matched by each pixel in the image region.

[0038] Next, for each image region, the number of pixels matching different preset text attribute categories in the image region is counted respectively. Taking the preset text attribute categories including three categories, namely: Chinese characters, English characters, and background as an example, for the image region corresponding to a certain text line, each pixel in the image region is traversed, and each pixel matching Chinese characters, English characters, or background pixels is determined according to the pixel value of each pixel, and the number of pixels matching Chinese characters, the number of pixels matching English characters, and the number of pixels matching the background in the image region are counted respectively.

[0039] Step 140: for each image region corresponding to the text line, determine the language matched by the text line according to the number of pixels of each preset language in the image region.

[0040] In some embodiments of the present application, determining the language matched by the text line according to the number of pixels of each preset language in the image area includes: determining the language matched by the pixels of the preset language corresponding to the maximum number as the language matched by the text line corresponding to the image area. That is, counting the number of pixels belonging to different languages ​​in the image area where each text line is located, and taking the language with the largest number of pixels as the language matched by the text line.

[0041] by Figure 2 Taking the image area corresponding to the text line 2102 as an example, the number of pixel points matching Chinese characters in the image area traversed in the above steps is 0, and the number of pixel points matching English characters is greater than 0, then it is determined that the language matched by the image area is English. Figure 2 For the text line 220 shown in the figure, after the above steps, it is determined that the number of pixels matching Chinese characters in the image area corresponding to the text line 220 is a value greater than 0, and the number of pixels matching English characters is 0, then it can be determined that the language matched by the text line 220 is Chinese. Determining the language of the text line based on the number of pixels of different languages ​​can effectively reduce the amount of calculation.

[0042] The text line language recognition method disclosed in the embodiment of the present application determines the text line position information in the target text image, and determines the probability value of the preset text attribute category matched by each pixel in the target text image, wherein the preset text attribute category is used to indicate that the corresponding pixel is a background pixel or a preset language pixel; according to the text line position information, the image area corresponding to each text line included in the target text image is determined; according to the probability value of the preset text attribute category matched by each pixel in the image area, the number of each preset language pixel in each image area is determined respectively; for the image area corresponding to each text line, according to the number of each preset language pixel in the image area, the language matched by the text line is determined. The method performs text line position information detection and text line language recognition respectively based on the same target text image, and then simultaneously obtains the position and corresponding language information of each text line in the target text image, effectively improving the accuracy of the text line language recognition algorithm in the text image and reducing the amount of calculation.

[0043] The text line language recognition method disclosed in the embodiment of the present application independently determines the text line position information in the target text image, determines the probability value of the preset text attribute category matched by each pixel point in the target text image, and fuses the text line position information and the text category attribute to perform language recognition in the text line. This avoids the problem of error accumulation caused by inaccurate determination of the text line position information, thereby reducing the accuracy of text line language recognition, in the prior art, when first identifying the text line position information in the target text image and then performing language recognition for the specified text line based on the text line position information.

[0044] The following examples illustrate the technical solutions used in some embodiments of the present application to determine the text line position information in the target text image, and to determine the probability value of the preset text attribute category matched by each pixel point in the target text image.

[0045] In some embodiments of the present application, the text line position information in the target text image may be determined using methods in the prior art.

[0046] In some other embodiments of the present application, determining the text line position information in the target text image includes: performing feature extraction and fusion processing on the target text image through a feature extraction module of a pre-trained deep learning network to obtain a first vector representation of the target text image; performing binary classification mapping on the first vector representation through a first classification module of the deep learning network to determine a two-category probability prediction map of the target text image, wherein the pixel value of each pixel point of the two-category probability prediction map is used to indicate the probability that the corresponding pixel point in the target text image belongs to a foreground pixel point; and determining the text line position information in the target text image based on the two-category probability prediction map.

[0047] The pre-trained deep learning network described in the embodiments of the present application adopts Figure 3 The network structure shown, wherein the deep learning network includes a feature extraction module 310, a first classification module 320, a second classification module 330 and an attention module 340.

[0048] The feature extraction module 310 is used to extract and fuse features of an input image (such as a target text image). In some embodiments of the present application, Figure 3 As shown, the feature extraction module 310 may further include: a feature extraction submodule 3101 and a feature fusion submodule 3102. The feature extraction submodule 3101 is used to extract features of multiple different levels of an input image (such as a target text image). The feature extraction submodule 3101 may use common network structures including resnet, densenet, mobilenet, etc., but is not limited to these network structures. The feature fusion submodule 3102 is used to fuse the hierarchical features extracted by the feature extraction submodule 3101. The feature fusion submodule 3102 may use a feature fusion module commonly used in the prior art, such as a feature pyramid network FPN, etc., but is not limited to an FPN network. For an input image, the features of different levels of the input image extracted by the feature extraction submodule 3101 generally refer to feature maps of different resolutions of the input image. The low-resolution feature map is the high-level feature of the input image, and the high-resolution feature map is the low-level feature of the input image. Taking the use of a convolutional neural network (such as Resnet) to extract feature maps of four levels of an input image as an example, the feature extraction submodule 3101 can obtain feature maps of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 sizes of the input image. The feature fusion submodule 3102 fuses the obtained feature maps of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 sizes of the input image to obtain a feature map of 1 / 4 size of the input image as the first vector representation of the input image.

[0049] In some embodiments of the present application, the first classification module 320 can adopt a general convolutional neural network structure, for example, a convolutional neural network including a convolution layer, a batch normalization layer, an activation layer and a transposed convolution layer. The first classification module performs a binary classification mapping on the first vector representation to determine a two-category probability prediction map of the input image (such as a target text image). In some embodiments of the present application, the pixel value of each pixel point of the two-category probability prediction map is the probability value of the corresponding pixel point of the input image as a foreground pixel point, for example, the pixel value of each pixel point of the two-category probability prediction map is a value between 0 and 1.

[0050] Next, the text line position information in the target text image is determined according to the two-category probability prediction map. In some embodiments of the present application, the text line position information in the target text image is determined according to the two-category probability prediction map, including: binarizing the two-category probability prediction map according to a preset global probability threshold to obtain a binary segmentation map matching the target text image, wherein the pixel value of each pixel in the binary segmentation map is used to indicate whether the corresponding pixel in the target text image belongs to a foreground pixel or a background pixel; and determining the text line position information in the target text image according to the binary segmentation map. The preset global probability threshold is set based on experience, for example, the preset global probability threshold can be set to 0.3.

[0051] Specifically, according to the binary segmentation map, the text line position information in the target text image is determined, including: first, obtaining an initialized binary segmentation map corresponding to the target text image, that is, initializing the pixel values ​​of all pixels in the binary segmentation map to 0, and then, according to the comparison result of the pixel value of the pixel at the corresponding position in the two-category probability prediction map and the preset global probability threshold, modifying the pixel value of the corresponding pixel in the initialized binary segmentation map, thereby modifying the pixel value of all pixels in the initialized binary segmentation map to be either 0 or 1. For example, when a certain pixel value in the two-category probability prediction map is greater than or equal to the preset global probability threshold, the pixel value of the pixel corresponding to the pixel in the initialized binary segmentation map is set to 1; when a certain pixel value in the two-category probability prediction map is less than the preset global probability threshold, the pixel value of the pixel corresponding to the pixel in the initialized binary segmentation map is kept unchanged at 0, so that the two-category probability prediction map can be binarized to obtain the binary segmentation map corresponding to the two-category probability prediction map. The pixels in the binary segmentation map correspond one-to-one to the pixels in the target text image, that is, the pixels in the binary segmentation map correspond one-to-one to the pixels in the input image.

[0052] Furthermore, the connected domain of the binary segmentation graph is obtained by using the related algorithm in the prior art, and the minimum bounding rectangle of each connected domain is solved. The minimum bounding rectangle of each connected domain represents the text line position information, and the position of each bounding rectangle obtained is the position of the image area corresponding to each text line in the target text image. The text line position information obtained by detecting the target text image through the first classification module 320 (especially irregular text images) is more accurate.

[0053] In some embodiments of the present application, Figure 4 As shown, the deep learning network also includes: a threshold map generation module 350, after the feature extraction and fusion processing of the target text image by the feature extraction module 310 of the pre-trained deep learning network to obtain the first vector representation of the target text image, it also includes: feature mapping of the first vector representation by the threshold map generation module 350 of the deep learning network to determine the threshold map matching the target text image; determining the text line position information in the target text image according to the two-category probability prediction map, including: differentiable binarization of the threshold map and the two-category probability prediction map to obtain a binary segmentation map matching the target text image, wherein the pixel value of each pixel point in the binary segmentation map is used to indicate whether the corresponding pixel point in the target text image belongs to a foreground pixel point or a background pixel point; determining the text line position information in the target text image according to the binary segmentation map.

[0054] Among them, the threshold map generation module 350 further performs feature mapping on the first vector representation output by the feature extraction module 310 to determine the threshold map for matching the target text image, and the pixel value of each pixel in the threshold map represents the dynamic threshold for binarizing the corresponding pixel in the target text image. Next, the threshold map generated by the threshold map generation module 350 and the two-category probability prediction map output by the first classification module 320 are subjected to differentiable binarization processing to obtain an approximate binarization map, which is used as the binary segmentation map for matching the target text image. According to the binary segmentation map, the specific implementation method for determining the text line position information in the target text image is described in the previous paragraph, which will not be repeated here. It should be noted that in this embodiment, the pixel value of the pixel in the two-category probability prediction map is compared with the dynamic threshold of the pixel at the corresponding position in the threshold map, and the pixel value of the corresponding pixel in the initialized binary segmentation map is modified with the comparison result, so that the pixel values ​​of all pixels in the initialized binary segmentation map are modified to be either 0 or 1.

[0055] By setting a threshold map generation module 350 in the deep learning network, the threshold map generation module 350 performs feature mapping according to the first vector representation output by the feature extraction module 310, determines the dynamic threshold when each pixel in the target text image is binarized, and effectively improves the accuracy of the obtained binary segmentation map, thereby improving the accuracy of the text line position and the language recognition results corresponding to the text line.

[0056] In some embodiments of the present application, Figure 3 , Figure 4 As shown, the second classification module 330 can adopt a general convolutional neural network structure, for example, a convolutional neural network including a convolutional layer, a batch normalization layer, an activation layer and a transposed convolutional layer; the attention module 340 can adopt a general attention mechanism network. After the feature extraction and fusion processing of the target text image is performed by the feature extraction module 310 of the pre-trained deep learning network to obtain the first vector representation of the target text image, it also includes: based on the two-category probability prediction map, the first vector representation is spatially weighted by the attention module to obtain a second vector representation; the second vector representation is multi-classified mapped by the second classification module 330 of the deep learning network to determine the multi-class probability prediction map of the target text image; wherein the pixel value of each pixel point in the multi-class probability prediction map is used to indicate the probability value of the corresponding pixel point in the target text image matching the preset text attribute category.

[0057] In some embodiments of the present application, the pixel value of each pixel of the multi-category probability prediction map can be represented by a multi-dimensional vector. For example, the pixel value of each pixel of the multi-category probability prediction map can be represented as (p1, p2, ..., pn), where p1 + p2 + ... + pn = 1, p1, p2 to pn-1 are used to indicate the probability value of the language corresponding to the pixel, and pn is used to indicate the probability value of the pixel being a background pixel. Taking the prediction of Chinese and English as an example, the pixel value of each pixel in the multi-category probability prediction map can be expressed as (p1, p2, p3). If p1 represents the probability that the corresponding pixel in the input image corresponding to the pixel matches the Chinese pixel, p2 represents the probability that the corresponding pixel in the input image corresponding to the pixel matches the English pixel, and p3 represents the probability that the corresponding pixel in the input image corresponding to the pixel matches the background pixel, assuming that the pixel value of a pixel in the multi-category probability prediction map is (0.6, 0.3, 0.1), that is, p1 = 0.6, p2 = 0.3, p3 = 0.1, p1 is the largest, so the language category corresponding to p1 is determined to be the language category of this pixel, then the language category corresponding to this pixel is Chinese.

[0058] When determining the text line position information in the target text image and the probability value of the preset text attribute category matched by each pixel point through a pre-trained deep learning network, the deep learning network needs to be trained first. Figure 3 and Figure 4 The deep learning network structure shown describes the training scheme of the deep learning network.

[0059] In some embodiments of the present application, before determining the text line position information in the target text image and determining the probability value of the preset text attribute category matched by each pixel in the target text image, it also includes: constructing a plurality of training samples; training a deep learning network including a feature extraction module, a first classification module and a second classification module based on the training samples; wherein the sample data of each of the training samples is a text image including at least one text line, the language information matched by each of the text lines and the position information in the text image, and each of the training samples matches at least the following two sample labels: a first label and a second label; the first label is used to indicate the probability of each image in the corresponding sample data; The pixel belongs to the true value of the foreground pixel or the background pixel, and the second label is used to indicate that each pixel in the corresponding sample data is a background pixel or a true value of a preset language pixel; the loss value of the deep learning network includes the loss value of the first classification module and the loss value of the second classification module, and the loss value of the first classification module is calculated based on the cross entropy of the two-category probability prediction map obtained by the first classification module predicting the training sample and the first label of the corresponding training sample; the loss value of the second classification module is calculated based on the cross entropy of the multi-category probability prediction map obtained by the second classification module predicting the training sample and the second label of the corresponding training sample.

[0060] See also Figure 3 The deep learning network structure shown in the figure comprises a trunk network structure (i.e., a feature extraction module 310) and two branch network structures (i.e., a first classification module 320 and an attention module 340 plus a second classification module 330), and the inputs of the two branch network structures are both the outputs of the trunk network structure, and the two branch network structures have their own outputs. Therefore, in the process of constructing the training samples of the deep learning network, for each training sample, its matching sample label includes two parts: the first label corresponding to the prediction result of the first classification module 320, and the second label corresponding to the prediction result of the second classification module 330.

[0061] In some embodiments of the present application, the language information and position information of each text line matched by the sample data of the training sample in the text image are the annotation information indicating the text line position and the corresponding language. In the model training process, the text line position information and the corresponding language information of the text image are used to generate the first label and the second label of the training sample matching corresponding to the text image. Among them, the text line position information and the corresponding language information in the text image are manually annotated information.

[0062] Take the sample data as Figure 2 Taking the text image shown in as an example, the position information of text lines 220, 2101 to 2105 are manually annotated, and the corresponding languages ​​are annotated, such as text line 220 corresponds to Chinese, and text line 2102 corresponds to English. The first label matched by the text image is a binary segmentation map with the same size as the text image. The binary segmentation map separates the text area and the non-text area in the text image. For example, the pixel values ​​of the binary segmentation map are 0 and 1, where 1 indicates that the corresponding pixel in the text image corresponding to the pixel belongs to the text, and 0 indicates that the corresponding pixel in the text image corresponding to the pixel does not belong to the text (such as the background). In order to avoid the situation where some manually annotated text boxes are inaccurate and the boundaries of different text boxes are unclear due to the close distance, the present application adopts the method of shrinking the text area inward in the prior art to make the first label.

[0063] Still taking the sample data as Figure 2Taking the text image shown in as an example, the position information and corresponding language information of text lines 220, 2101 to 2105 are manually annotated. The second label matched by the text image is the multivariate segmentation map of the text image. The pixel value of each pixel in the multivariate segmentation map represents the preset text attribute category matched by the corresponding pixel in the text image. Each pixel in the multivariate segmentation map corresponds one-to-one to a pixel of the text image in the sample data. For a text image with a size of H×W, the size of its multivariate segmentation map is H×W×C, where C is the number of language categories plus one. Taking the multivariate segmentation map as a ternary segmentation map as an example, the size of the input text image is 640x640, and when the languages ​​include Chinese and English, the size of the output multivariate segmentation map is 640x640x3. Each pixel point on the input text image corresponds to a three-dimensional vector at the corresponding position on the multivariate segmentation map. The values ​​of this vector include (1, 0, 0), (0, 1, 0), and (0, 0, 1). The three vectors correspond to three different categories, among which the category where 1 is located represents the language category or background of the pixel point. For example, if a pixel point in the text image is a Chinese character, the pixel value of the pixel point in the ternary segmentation map corresponding to the pixel point is expressed as (1, 0, 0); for another example, if a pixel point in the text image is an English character, the pixel value of the pixel point in the ternary segmentation map corresponding to the pixel point is expressed as (0, 1, 0); for another example, if a pixel point in the text image is the background, the pixel value of the pixel point in the ternary segmentation map corresponding to the pixel point is expressed as (0, 0, 1).

[0064] See also Figure 3The deep learning network shown in the figure performs the following learning process for each training sample during the deep learning network training process: first, after feature extraction and fusion processing by the feature extraction module 310, a first vector representation of the training sample is obtained; then, the first classification module 320 further classifies and maps the first vector representation, and the first classification module 320 outputs a two-category probability prediction map. After the attention module 340 performs weighted processing on the first vector representation based on the two-category probability prediction map, the second classification module 330 performs classification and mapping on the second vector representation obtained after the weighted processing, and outputs a multi-category probability prediction map; then, the cross entropy between the two-category probability prediction map and the first label is calculated, and it is used as the first prediction loss value of the training sample; the cross entropy between the multi-category probability prediction map and the second label is calculated, and it is used as the second prediction loss value of the training sample. After predicting all training samples, the first prediction loss value and the second prediction loss value corresponding to each training sample will be obtained; finally, with the goal of minimizing the first prediction loss value and the second prediction loss value of all training samples (that is, with the goal of minimizing the loss value of the deep learning network (the sum of the first prediction loss value and the second prediction loss value)), the network parameters of the deep learning network are optimized, and the deep learning network is iteratively trained until the first prediction loss value and the second prediction loss value converge, that is, the loss value of the deep learning network converges, and the training process of the deep learning network is completed.

[0065] See also Figure 4 The deep learning network structure shown in the figure comprises a trunk network structure (i.e., a feature extraction module 310) and three branch network structures (i.e., a first classification module 320, an attention module 340 plus a second classification module 330, and a threshold map generation module 350), and the inputs of the three branch network structures are all the outputs of the trunk network structure, and the three branch network structures have their own outputs. Therefore, in the process of constructing the training samples of the deep learning network, for each training sample, its matching sample label includes three parts: a first label corresponding to the prediction result of the first classification module 320, a second label corresponding to the prediction result of the second classification module 330, and a third label corresponding to the prediction result of the threshold map generation module 350.

[0066] The meaning and generation method of the first label and the second label refer to the description in the previous paragraph and will not be repeated here.

[0067] The third label is the threshold map of the text image as sample data. The threshold map can be generated using a method similar to the first label. For example, the text box in the text image is expanded inward and outward by 0.4 times, and then the normalized distance D between the pixel point in the area between the contraction box and the expansion box and its nearest text box boundary is calculated, and 1-D is used as the pixel value of the threshold map at this point. At the same time, the maximum and minimum values ​​of the pixels in the threshold map are set. For example, the maximum value of the pixels in the threshold map can be set to 0.7, and the minimum value of the pixels in the threshold map can be set to 0.3.

[0068] See also Figure 4 The deep learning network shown in the figure performs the following learning process for each training sample during the deep learning network training process: first, after feature extraction and fusion processing by the feature extraction module 310, a first vector representation of the training sample is obtained; then, the first classification module 320 and the threshold map generation module 350 further classify and map the first vector representation respectively, and the first classification module 320 outputs a two-category probability prediction map, and the threshold map generation module 350 outputs a threshold map; at the same time, after the attention module 340 performs weighted processing on the first vector representation based on the two-category probability prediction map, the second classification module 330 further classifies and maps the second vector representation obtained after the weighted processing, and outputs a multi-category probability prediction map; then, the cross entropy between the two-category probability prediction map and the first label is calculated, and it is used as the first prediction loss value of the training sample; the cross entropy between the multi-category probability prediction map and the second label is calculated, and it is used as the second prediction loss value of the training sample; the mean absolute error (i.e., L1loss) between the threshold map output by the threshold map generation module 350 and the third label is calculated, and it is used as the third prediction loss value of the training sample. After predicting all training samples, the first prediction loss value, the second prediction loss value and the third prediction loss value corresponding to each training sample will be obtained; finally, with the goal of minimizing the first prediction loss value, the second prediction loss value and the third prediction loss value of all training samples (that is, with the goal of minimizing the loss value of the deep learning network (the sum of the first prediction loss value, the second prediction loss value and the third prediction loss value)), the network parameters of the deep learning network are optimized, and the deep learning network is iteratively trained until the first prediction loss value, the second prediction loss value and the third prediction loss value converge, that is, the loss value of the deep learning network converges, and the training process of the deep learning network is completed.

[0069] In some embodiments of the present application, the feature extraction module 310 and the first classification module 320 may be trained first, and then, based on the parameters of the feature extraction module 310 obtained through the training, the attention module 340 and the second classification module 330, as well as the threshold map generation module 350 may be further trained, which will not be described in detail in the embodiments of the present application.

[0070] In some embodiments of the present application, the attention module further includes: a spatial attention sub-module and a channel attention sub-module, and based on the two-category probability prediction map, the first vector representation is spatially weighted processed by the attention module to obtain a second vector representation, including any one of the following methods.

[0071] The first method is to use the pixel values ​​of each pixel point of the two-category probability prediction map as weights, and perform spatial weighted processing on the first vector representation through the attention module to obtain a second vector representation. For example, through the spatial attention submodule (i.e., a network module based on the spatial attention mechanism), the pixel values ​​of each pixel point in the two-category probability prediction map output by the first classification module 320 are used as weights, and the pixel values ​​of each pixel point in the two-category probability prediction map are multiplied pixel by pixel with the first vector representation output by the feature extraction module 310, and the result obtained after the multiplication is used as the second vector representation. In the two-category probability prediction map, the pixel values ​​of the pixels in the text line area will be relatively high, and the pixel values ​​of the pixels in the non-text line area (i.e., the background area) will be relatively low. Therefore, when the second vector representation is obtained after the two-category probability prediction map is multiplied with the first vector representation, the features of the text line area in the first vector representation (i.e., the vector representation) will be highlighted, and the features of the non-text line area will be suppressed, so that when the second classification module 330 performs multi-category prediction based on the second vector representation, it is beneficial to improve the output of a more accurate multi-category probability prediction map by the second classification module 330.

[0072] The second method is to use the pixel value of each pixel point in the two-category probability prediction map as a weight, and perform spatial weighted processing on the first vector representation through the attention module; then, the feature map obtained by the spatial weighted processing is input into the attention module, the weight corresponding to each channel is calculated, and then the feature map is channel-weighted with the calculated weight to obtain a second vector representation. For example, first, through the spatial attention submodule (i.e., a network module based on the spatial attention mechanism), the pixel value of each pixel point in the two-category probability prediction map output by the first classification module 320 is used as a weight, and the pixel value of each pixel point in the two-category probability prediction map is multiplied pixel by pixel with the first vector representation output by the feature extraction module 310; then, the feature map obtained after the pixel-by-pixel multiplication is input into the channel attention submodule, and the weight of each channel of the channel attention submodule is calculated; and the feature map obtained after the pixel-by-pixel multiplication is further weighted with the calculated channel weight, and the final result is used as the second vector representation.

[0073] In some embodiments of the present application, the channel attention submodule can adopt the SE-insept ion model structure. Taking the vector representation obtained after the spatial weighted processing as a 640x640x3-dimensional vector as an example, if the calculated channel weight is [0.5, 0.3, 0.2], it means that the vector corresponding to the first channel (such as the first layer feature) is of the highest importance, and the vector corresponding to the third channel (such as the third layer feature) is of the lowest importance. The weight of each channel is multiplied by the vector representation (i.e., feature map) of each channel obtained after the spatial weighted processing to obtain a new vector representation as the second vector representation.

[0074] By further performing weighted processing based on the channel attention mechanism on the vector representation (i.e., feature map) after weighted processing based on the spatial attention mechanism, a second vector representation is obtained, thereby highlighting feature expressions with high weights in the channel direction. When the second classification module 330 performs multi-category probability prediction based on the second vector representation, the accuracy of the multi-category probability prediction map output by the second classification module 330 can be further improved.

[0075] During the online recognition process, the model trained on the target text image will output text lines divided into different languages.

[0076] In some other embodiments of the present application, a complete line of text containing mixed languages ​​may also be marked as one text line, and the text line may be segmented later, but the present application does not limit this.

[0077] The text line language identification method disclosed in the embodiment of the present application uses a deep learning network to simultaneously complete the two tasks of text line detection and language identification, and obtain the position of the text line and the corresponding language. A branch network structure of the deep learning network is used to obtain a two-category probability prediction map to perform text line detection, and another branch network structure is used to obtain a multi-category probability prediction map. The text line detection results and the multi-category probability prediction results are used to classify the language of each text line. The above method improves the accuracy of text line detection and language identification. Moreover, in the training stage, only one training data set needs to be used to train a deep learning network to simultaneously complete the tasks of text line detection and language identification, and the text line position and the corresponding language detection results can be obtained at the same time, which greatly reduces the time cost and maintenance cost of the training process.

[0078] Embodiment 2

[0079] The embodiment of the present application discloses a text line language recognition device, such as Figure 5 As shown, the device comprises:

[0080] The text line position information and pixel classification probability value determination module 510 is used to determine the text line position information in the target text image, and to determine the probability value of a preset text attribute category matched by each pixel in the target text image, wherein the preset text attribute category is used to indicate that the corresponding pixel is a background pixel or a preset language pixel;

[0081] A text line corresponding image region determination module 520, configured to determine the image region corresponding to each text line included in the target text image according to the text line position information;

[0082] The number of pixels in different languages ​​determining module 530 is used to determine the number of pixels in each of the preset languages ​​in each of the image regions according to the probability value of the preset text attribute category matched by each pixel in the image region;

[0083] The text line language determination module 540 is used to determine the language matched by each text line according to the number of pixels of each preset language in the image area corresponding to each text line.

[0084] In some embodiments of the present application, determining text line position information in the target text image includes:

[0085] Performing feature extraction and fusion processing on the target text image through a feature extraction module of a pre-trained deep learning network to obtain a first vector representation of the target text image;

[0086] Performing binary classification mapping on the first vector representation through the first classification module of the deep learning module to determine a binary probability prediction map of the target text image, wherein the pixel value of each pixel point of the binary probability prediction map is used to indicate the probability that the corresponding pixel point in the target text image belongs to a foreground pixel point;

[0087] The text line position information in the target text image is determined according to the two-category probability prediction map.

[0088] In some embodiments of the present application, after the feature extraction module of the pre-trained deep learning network performs feature extraction and fusion processing on the target text image to obtain the first vector representation of the target text image, the method further includes:

[0089] Performing feature mapping on the first vector representation by a threshold map generation module of the deep learning network to determine a threshold map for matching the target text image;

[0090] The determining the text line position information in the target text image according to the two-category probability prediction map includes:

[0091] The threshold map and the two-category probability prediction map are differentiably binarized to obtain a binary segmentation map for matching the target text image, wherein the pixel value of each pixel point in the binary segmentation map is used to indicate whether the corresponding pixel point in the target text image belongs to a foreground pixel point or a background pixel point;

[0092] According to the binary segmentation map, text line position information in the target text image is determined.

[0093] In some embodiments of the present application, after the feature extraction module of the pre-trained deep learning network performs feature extraction and fusion processing on the target text image to obtain the first vector representation of the target text image, it also includes: based on the two-category probability prediction map, spatially weighted processing is performed on the first vector representation by the attention module to obtain a second vector representation; multi-classification mapping is performed on the second vector representation by the second classification module of the deep learning network to determine the multi-class probability prediction map of the target text image; wherein the pixel value of each pixel point in the multi-class probability prediction map is used to indicate the probability value that the corresponding pixel point in the target text image matches a preset text attribute category.

[0094] In some embodiments of the present application, based on the two-category probability prediction map, the first vector representation is spatially weighted processed by the attention module to obtain a weighted vector representation, including any one of the following methods: a first method, taking the pixel value of each pixel point of the two-category probability prediction map as a weight, and performing spatial weighted processing on the first vector representation through the attention module to obtain a second vector representation; a second method, taking the pixel value of each pixel point of the two-category probability prediction map as a weight, and performing spatial weighted processing on the first vector representation through the attention module; thereafter, the feature map obtained by the spatial weighted processing is input into the attention module, the weight corresponding to each channel is calculated, and then the feature map is channel-weighted with the calculated weight to obtain a second vector representation.

[0095] In some embodiments of the present application, determining the language matched by the text line according to the number of pixels of each preset language in the image area includes:

[0096] The language matched by the preset language pixel point corresponding to the maximum value of the number is determined as the language matched by the text line corresponding to the image area.

[0097] In some embodiments of the present application, Figure 6 As shown, the device also includes:

[0098] The training sample construction module 550 is used to construct a plurality of training samples; wherein the sample data of each training sample is a text image including at least one text line, language information matched by each text line and position information in the text image, and each training sample matches at least the following two sample labels: a first label and a second label; the first label is used to indicate the true value of each pixel point in the corresponding sample data belonging to a foreground pixel point or a background pixel point, and the second label is used to indicate the true value of each pixel point in the corresponding sample data being a background pixel point or a preset language pixel point;

[0099] A model training module 560 is used to train a deep learning network based on the training samples, wherein the deep learning network includes a feature extraction module, a first classification module, and a second classification module; wherein the loss value of the deep learning network includes the loss value of the first classification module and the loss value of the second classification module, and the loss value of the first classification module is calculated based on the cross entropy of the two-category probability prediction graph obtained by predicting the training samples by the first classification module and the first label of the corresponding training sample; the loss value of the second classification module is calculated based on the cross entropy of the multi-category probability prediction graph obtained by predicting the training samples by the second classification module and the second label of the corresponding training sample.

[0100] In some embodiments of the present application, the training sample construction module 550 is also used to construct a plurality of training samples; wherein the sample data of each training sample is a text image including at least one text line, language information matched by each text line and position information in the text image, and each training sample matches at least the following three sample labels: a first label, a second label and a third label; the first label is used to indicate the true value of each pixel in the corresponding sample data belonging to a foreground pixel or a background pixel, and the second label is used to indicate the true value of each pixel in the corresponding sample data being a background pixel or a preset language pixel; the third label is a threshold map of the corresponding sample data;

[0101] The model training module 560 is also used to train a deep learning network based on the training samples, and the deep learning network includes a feature extraction module, a first classification module, a second classification module and a threshold map generation module; wherein the loss value of the deep learning network includes the loss value of the first classification module, the loss value of the second classification module, and the loss value of the threshold map generation module, the loss value of the first classification module is calculated based on the cross entropy of the two-category probability prediction map obtained by predicting the training samples by the first classification module and the first label of the corresponding training sample; the loss value of the second classification module is calculated based on the cross entropy of the multi-category probability prediction map obtained by predicting the training samples by the second classification module and the second label of the corresponding training sample; the loss value of the threshold map generation module is calculated based on the mean absolute error between the threshold map obtained by predicting the training samples by the threshold map generation module and the third label of the corresponding training sample.

[0102] The text line language identification device disclosed in the embodiment of the present application is used to implement the text line language identification method described in the first embodiment of the present application. The specific implementation methods of each module of the device will not be repeated here, and reference can be made to the specific implementation methods of the corresponding steps in the method embodiment.

[0103] The text line language recognition device disclosed in the embodiment of the present application determines the text line position information in the target text image, and determines the probability value of the preset text attribute category matched by each pixel point in the target text image, wherein the preset text attribute category is used to indicate that the corresponding pixel point is a background pixel point or a preset language pixel point; according to the text line position information, determines the image area corresponding to each text line included in the target text image; according to the probability value of the preset text attribute category matched by each pixel point in the image area, determines the number of each preset language pixel point in each of the image areas; for the image area corresponding to each text line, determines the language matched by the text line according to the number of each preset language pixel point in the image area, thereby effectively improving the accuracy of the text line language recognition algorithm in the text image and reducing the amount of calculation.

[0104] The text line language recognition device disclosed in the embodiment of the present application independently determines the text line position information in the target text image, and determines the probability value of each pixel point in the target text image matching a preset text attribute category (for example, the text line position information in the target text image and the probability value of each pixel point matching a preset text attribute category are determined by two network branches of a deep learning network, respectively), and fuses the text line position information and the text category attribute to perform language recognition in the text line, thereby avoiding the problem of error accumulation caused by inaccurate determination of the text line position information when first identifying the text line position information in the target text image and then performing language recognition of the specified text line based on the text line position information in the prior art, thereby reducing the accuracy of text line language recognition.

[0105] By using a deep learning network to simultaneously complete the two tasks of text line detection and language recognition, the position of the text line and the corresponding language are obtained at the same time. A branch network structure of the deep learning network is used to obtain a two-category probability prediction map to perform text line detection, and another branch network structure is used to obtain a multi-category probability prediction map. The text line detection results and the multi-category probability prediction results are used to classify the language of each text line. The above device improves the accuracy of text line detection and language recognition. Moreover, in the training stage, only one training data set is needed to train a deep learning network to simultaneously complete the tasks of text line detection and language recognition, and the text line position and the corresponding language detection results are obtained at the same time, which greatly reduces the time cost and maintenance cost during the training process.

[0106] Accordingly, the present application also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the text line language recognition method as described in Embodiment 1 of the present application when executing the computer program. The electronic device may be a PC, a mobile terminal, a personal digital assistant, a tablet computer, etc.

[0107] The present application also discloses a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the text line language recognition method as described in the first embodiment of the present application are implemented.

[0108] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0109] The above is a detailed introduction to a text line language recognition method and device provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0110] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0111] References to "one embodiment," "embodiment," or "one or more embodiments" herein mean that a particular feature, structure, or characteristic described in conjunction with the embodiment is included in at least one embodiment of the present application. In addition, please note that examples of the term "in one embodiment" herein do not necessarily all refer to the same embodiment.

[0112] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0113] In the claims, any reference signs placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application may be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for identifying the language of a text line, characterized in that: include: Determine text line position information in a target text image, and determine a probability value of a preset text attribute category matched by each pixel in the target text image, wherein the preset text attribute category is used to indicate that the corresponding pixel is a background pixel or a preset language pixel; Determining, according to the text line position information, an image area corresponding to each text line included in the target text image; Determining the number of pixels of each preset language in each of the image regions according to the probability value of the preset text attribute category matched by each pixel in the image region; For each of the image regions corresponding to the text lines, determining the language matched by the text lines according to the number of pixels of each of the preset languages ​​in the image region; The determining of text line position information in the target text image includes: Performing feature extraction and fusion processing on the target text image through a feature extraction module of a pre-trained deep learning network to obtain a first vector representation of the target text image; Performing binary classification mapping on the first vector representation through the first classification module of the deep learning network to determine a binary probability prediction map of the target text image, wherein the pixel value of each pixel point of the binary probability prediction map is used to indicate the probability that the corresponding pixel point in the target text image belongs to a foreground pixel point; Determine text line position information in the target text image according to the two-category probability prediction map; The deep learning network further includes: an attention module, after determining the two-category probability prediction map of the target text image, further including: Based on the two-category probability prediction graph, performing spatial weighted processing on the first vector representation through the attention module to obtain a second vector representation; The second vector representation is subjected to multi-classification mapping by the second classification module of the deep learning network to determine a multi-class probability prediction map of the target text image; wherein the pixel value of each pixel point in the multi-class probability prediction map is used to indicate the probability value that the corresponding pixel point in the target text image matches a preset text attribute category.

2. The method according to claim 1, characterized in that After the step of performing feature extraction and fusion processing on the target text image through the feature extraction module of the pre-trained deep learning network to obtain the first vector representation of the target text image, the method further includes: Performing feature mapping on the first vector representation by a threshold map generation module of the deep learning network to determine a threshold map for matching the target text image; The step of determining the text line position information in the target text image according to the two-category probability prediction map comprises: The threshold map and the two-category probability prediction map are differentiably binarized to obtain a binary segmentation map for matching the target text image, wherein the pixel value of each pixel point in the binary segmentation map is used to indicate whether the corresponding pixel point in the target text image belongs to a foreground pixel point or a background pixel point; According to the binary segmentation map, text line position information in the target text image is determined.

3. The method according to claim 1, characterized in that The step of performing spatial weighted processing on the first vector representation by the attention module based on the two-category probability prediction graph to obtain a second vector representation comprises: Using the pixel value of each pixel point of the two-category probability prediction map as a weight, the first vector representation is spatially weighted by the attention module to obtain a second vector representation; or, The first vector representation is spatially weighted by the attention module, with the pixel value of each pixel point of the two-category probability prediction map as the weight; then, the feature map obtained by the spatial weighted processing is input into the attention module, the weight corresponding to each channel is calculated, and then the feature map is channel-weighted by the calculated weight to obtain the second vector representation.

4. The method according to any one of claims 1 to 3, characterized in that: The step of determining the language matched by the text line according to the number of pixels of each preset language in the image area comprises: The language matched by the preset language pixel point corresponding to the maximum value of the number is determined as the language matched by the text line corresponding to the image area.

5. The method according to any one of claims 1 to 3, characterized in that: Before the steps of determining the text line position information in the target text image and determining the probability value of the preset text attribute category matched by each pixel point in the target text image, the method further includes: Constructing a plurality of training samples; wherein the sample data of each training sample is a text image including at least one text line, language information matched by each text line and position information in the text image, and each training sample matches at least the following two sample labels: a first label and a second label; the first label is used to indicate the true value of each pixel in the corresponding sample data belonging to a foreground pixel or a background pixel, and the second label is used to indicate the true value of each pixel in the corresponding sample data being a background pixel or a pixel of a preset language; A deep learning network is trained based on the training samples, and the deep learning network includes a feature extraction module, a first classification module, and a second classification module; wherein the loss value of the deep learning network includes the loss value of the first classification module and the loss value of the second classification module, and the loss value of the first classification module is calculated based on the cross entropy of the two-category probability prediction graph obtained by predicting the training samples by the first classification module and the first label of the corresponding training sample; the loss value of the second classification module is calculated based on the cross entropy of the multi-category probability prediction graph obtained by predicting the training samples by the second classification module and the second label of the corresponding training sample.

6. The method according to any one of claims 1 to 3, characterized in that: Before the steps of determining the text line position information in the target text image and determining the probability value of the preset text attribute category matched by each pixel point in the target text image, the method further includes: Constructing a plurality of training samples; wherein the sample data of each training sample is a text image including at least one text line, language information matched by each text line and position information in the text image, and each training sample matches at least the following three sample labels: a first label, a second label and a third label; the first label is used to indicate the true value of each pixel in the corresponding sample data belonging to a foreground pixel or a background pixel, and the second label is used to indicate the true value of each pixel in the corresponding sample data being a background pixel or a pixel of a preset language; the third label is a threshold map of the corresponding sample data; A deep learning network is trained based on the training samples, and the deep learning network includes a feature extraction module, a first classification module, a second classification module and a threshold map generation module; wherein the loss value of the deep learning network includes the loss value of the first classification module, the loss value of the second classification module, and the loss value of the threshold map generation module, the loss value of the first classification module is calculated based on the cross entropy of the two-category probability prediction map obtained by predicting the training samples by the first classification module and the first label of the corresponding training sample; the loss value of the second classification module is calculated based on the cross entropy of the multi-category probability prediction map obtained by predicting the training samples by the second classification module and the second label of the corresponding training sample; the loss value of the threshold map generation module is calculated based on the mean absolute error between the threshold map obtained by predicting the training samples by the threshold map generation module and the third label of the corresponding training sample.

7. A text line language recognition device, characterized in that: include: A text line position information and pixel classification probability value determination module, used to determine the text line position information in the target text image, and to determine the probability value of a preset text attribute category matched by each pixel in the target text image, wherein the preset text attribute category is used to indicate that the corresponding pixel is a background pixel or a preset language pixel; A text line corresponding image area determination module, used to determine the image area corresponding to each text line included in the target text image according to the text line position information; A module for determining the number of pixels of different languages, used to determine the number of pixels of each preset language in each of the image regions according to the probability value of the preset text attribute category matched by each pixel in the image region; A text line language determination module, configured to determine, for each image area corresponding to each text line, the language matched by the text line according to the number of pixels of each preset language in the image area; The text line position information and pixel classification probability value determination module includes: A feature extraction module, used for performing feature extraction and fusion processing on the target text image to obtain a first vector representation of the target text image; A first classification module is used to perform binary classification mapping on the first vector representation to determine a binary probability prediction map of the target text image; wherein the pixel value of each pixel point in the binary probability prediction map is used to indicate the probability that the corresponding pixel point in the target text image belongs to a foreground pixel point; and determine the text line position information in the target text image according to the binary probability prediction map; an attention module, configured to perform spatial weighting processing on the first vector representation based on the two-category probability prediction map to obtain a second vector representation; The second classification module is used to perform multi-classification mapping on the second vector representation to determine a multi-class probability prediction map of the target text image; wherein the pixel value of each pixel point in the multi-class probability prediction map is used to indicate the probability value of the corresponding pixel point in the target text image matching a preset text attribute category.

8. An electronic device comprising a memory, a processor, and a program code stored in the memory and executable on the processor, wherein: When the processor executes the program code, the text line language recognition method described in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having program code stored thereon, characterized in that: When the program code is executed by a processor, the steps of the text line language recognition method described in any one of claims 1 to 6 are implemented.

Images