A text chunking method, a picture-text translation method and related devices

Abstract

The present application provides a text block method, a picture-text translation method and related equipment, wherein the text block method comprises: obtaining text line information corresponding to a plurality of text lines in a target picture; obtaining text line features corresponding to the plurality of text lines in the target picture according to the target picture and the text line information corresponding to the plurality of text lines in the target picture; and performing block on the plurality of text lines in the target picture according to the text line features corresponding to the plurality of text lines in the target picture. The text block method provided by the present application can realize block on the plurality of text lines in the picture. On the basis of the text block method provided by the present application, the present application further provides a picture-text translation method, which translates in the unit of text block. Since the text lines in one text block have semantic relationship, the translation in the unit of text block can obtain a more accurate, coherent and fluent translation.

Description

Technical Field

[0001] This invention relates to the field of text processing technology, and in particular to a text segmentation method, an image-text translation method, and related equipment. Background Technology

[0002] In some application scenarios, it is necessary to segment the text in an image into text blocks. Text segmentation refers to dividing several lines of text with semantic relationships into a text block.

[0003] For example, in the context of image-text translation, the text in an image needs to be translated. The current solution is to translate the text content of each line of text individually. However, individual lines of text lack complete contextual semantic relationships due to the absence of preceding or following text, which results in an inability to obtain an accurate, coherent, and fluent translation. In order to achieve better translation results, the current approach is to divide the text lines in the image that have semantic relationships into text blocks and translate the text content of each text block.

[0004] However, there is currently no practical text segmentation scheme. Summary of the Invention

[0005] In view of this, the present invention provides a text segmentation method, an image-text translation method, and related equipment for segmenting text lines in an image into blocks. The technical solution is as follows:

[0006] Obtain text line information corresponding to several text lines in the target image, wherein the text line information includes text line position information and text line content;

[0007] Based on the target image and the text line information corresponding to several text lines in the target image, obtain the text line features corresponding to several text lines in the target image.

[0008] Based on the text line features corresponding to the text lines in the target image, the text lines in the target image are divided into blocks.

[0009] Optionally, obtaining the text line features corresponding to the several text lines in the target image based on the target image and the text line information corresponding to the several text lines in the target image includes:

[0010] Based on the target image and the text line information corresponding to several text lines in the target image, the semantic features, visual features, and positional features corresponding to the several text lines in the target image are obtained respectively.

[0011] Optionally, the step of obtaining semantic features, visual features, and positional features corresponding to the several text lines in the target image based on the target image and the text line information corresponding to the several text lines in the target image includes:

[0012] Based on the text content corresponding to several text lines in the target image, obtain the semantic features corresponding to several text lines in the target image.

[0013] Based on the target image and the text line position information corresponding to the text lines in the target image, obtain the visual features corresponding to the text lines in the target image.

[0014] Based on the text line position information corresponding to several text lines in the target image, obtain the position features corresponding to several text lines in the target image.

[0015] Optionally, obtaining the semantic features corresponding to the text lines in the target image based on their respective text line contents includes:

[0016] For each line of text in the target image:

[0017] The text line content corresponding to the text line is segmented according to the characters to obtain the segmentation unit sequence corresponding to the text line;

[0018] Obtain the encoding of each word segmentation unit in the word segmentation unit sequence corresponding to the text line to obtain the encoding sequence corresponding to the text line, wherein the encoding of each word segmentation unit includes several bytes;

[0019] Decompose each code in the encoding sequence corresponding to the text line into bytes to obtain the byte sequence corresponding to the text line.

[0020] Obtain the word vector corresponding to the number represented by each byte in the byte sequence corresponding to the text line, so as to obtain the word vector sequence corresponding to the text line;

[0021] Based on the word vector sequence corresponding to the text line, obtain the semantic features corresponding to the text line.

[0022] Optionally, obtaining the word vector corresponding to the number represented by each byte in the byte sequence corresponding to the text line includes:

[0023] From the pre-built word vector table, obtain the word vector corresponding to the number represented by each byte in the byte sequence corresponding to the text line;

[0024] The word vector table includes word vectors corresponding to the indices of each virtual word in the constructed virtual word table, and the word vector corresponding to the number represented by one byte is the word vector corresponding to the index in the word vector table that is the same as the number represented by that byte.

[0025] Optionally, the step of obtaining the visual features corresponding to the several text lines in the target image based on the target image and the text line position information corresponding to the several text lines in the target image includes:

[0026] Based on the target image, obtain the overall visual features of the target image;

[0027] Based on the text line position information corresponding to several text lines in the target image, the visual features corresponding to several text lines in the target image are obtained from the overall visual features of the target image.

[0028] Optionally, the step of dividing the several text lines in the target image into blocks based on the text line features corresponding to the several text lines in the target image includes:

[0029] Construct a target feature sequence that includes block delimiter features, end-of-image features, and text line features corresponding to several text lines in the target image;

[0030] At each decoding moment, based on the feature with the highest output probability at the previous decoding moment, the feature with the highest output probability at the current decoding moment is determined from the target feature sequence, and the index of the feature with the highest output probability at the current decoding moment is determined as the decoding result at the current decoding moment;

[0031] Based on the decoding results at each decoding time, the block results of several text lines in the target image are determined.

[0032] Optionally, determining the feature with the highest output probability at the current decoding time from the target feature sequence based on the feature with the highest output probability at the previous decoding time includes:

[0033] Based on the feature with the highest output probability at the previous decoding time, determine the state feature vector that can represent the information of the decoded result;

[0034] Calculate the similarity between the state feature vector and each feature in the target feature sequence;

[0035] The similarity between the state feature vector and each feature in the target feature sequence is normalized to obtain the output probability of each feature in the target feature sequence at the current decoding time.

[0036] Based on the output probability of each feature in the target feature sequence at the current decoding time, determine the feature with the highest output probability at the current decoding time from the target feature sequence.

[0037] Optionally, the step of obtaining text line features corresponding to the several text lines in the target image based on the target image and the text line information corresponding to the several text lines in the target image, and dividing the several text lines in the target image into blocks based on the text line features corresponding to the several text lines in the target image, includes:

[0038] Based on the pre-trained text segmentation model, the target image and the text line information corresponding to several text lines in the target image are processed to obtain the segmentation results of several text lines in the target image.

[0039] The text segmentation model is trained using several training samples from multiple languages ​​and sample labels corresponding to those training samples. Each training sample includes a training image and text line information corresponding to several text lines in the training image. The sample label is the actual segmentation result of several text lines in the training image contained in the corresponding training sample.

[0040] A method for translating text into images, including:

[0041] Obtain the target image;

[0042] The text segmentation method described above is used to segment several lines of text in the target image into blocks;

[0043] For each obtained text block, the text lines contained in the text block are connected into a sentence in the order of reading, and the sentence is translated.

[0044] A text segmentation device includes: a text line information acquisition module, a text line feature acquisition module, and a text segmentation module;

[0045] The text line information acquisition module is used to acquire text line information corresponding to several text lines in the target image, wherein the text line information includes text line position information and text line content;

[0046] The text line feature acquisition module is used to acquire text line features corresponding to several text lines in the target image based on the target image and the text line information corresponding to several text lines in the target image.

[0047] The text segmentation module is used to segment several text lines in the target image into blocks based on the text line features corresponding to those lines.

[0048] A text segmentation device includes: a memory and a processor;

[0049] The memory is used to store programs;

[0050] The processor is configured to execute the program to implement each step of the text segmentation method described above.

[0051] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the text segmentation method described in any of the preceding claims.

[0052] The text segmentation method provided by this invention, after obtaining the target image, first acquires the text line information corresponding to several text lines in the target image, then, based on the target image and the text line information corresponding to the several text lines in the target image, acquires the text line features corresponding to the several text lines in the target image, and finally, segments the several text lines in the target image into blocks based on the text line features corresponding to the several text lines in the target image. It is evident that the text segmentation method provided by this invention can segment several text lines in an image into blocks. Based on the text segmentation method provided by this invention, this invention also provides a text-image translation method. This translation method translates text blocks as units. Since the text lines within a text block have semantic relationships, translating by text blocks can obtain a more accurate, coherent, and fluent translation. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0054] Figure 1 This is a schematic diagram of the hardware architecture involved in the present invention;

[0055] Figure 2 A flowchart illustrating the text segmentation method provided in an embodiment of the present invention;

[0056] Figure 3 This is a schematic diagram illustrating an example of obtaining text line information corresponding to several text lines in a target image, as provided in an embodiment of the present invention.

[0057] Figure 4 A schematic diagram illustrating the process of implementing text segmentation based on a model, as provided in an embodiment of the present invention;

[0058] Figure 5This is a schematic diagram illustrating an example of dividing a target image into blocks based on a text segmentation model provided in an embodiment of the present invention;

[0059] Figure 6 This is a flowchart illustrating the process of determining the segmentation result of several text lines in a target image based on the semantic features, visual features, and positional features corresponding to several text lines in the target image, as well as the decoding part of the text segmentation model, provided in an embodiment of the present invention.

[0060] Figure 7 This is a schematic diagram of the decoding part of the text segmentation model provided in an embodiment of the present invention;

[0061] Figure 8 A flowchart illustrating the training of a text segmentation model provided in an embodiment of the present invention;

[0062] Figure 9 This is a schematic diagram of the structure of the text segmentation device provided in an embodiment of the present invention;

[0063] Figure 10 This is a schematic diagram of the structure of a text segmentation device provided in an embodiment of the present invention. Detailed Implementation

[0064] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0065] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] To achieve text segmentation, the inventors conducted research and, through continuous investigation, ultimately proposed a text segmentation method with relatively good performance. Before introducing the text segmentation method provided by this invention, the hardware architecture involved in this invention will be described first.

[0067] In one possible implementation, such as Figure 1 As shown, the hardware architecture involved in this invention may include: electronic device 101 and server 102.

[0068] For example, electronic device 101 can be any electronic product that can interact with a user, such as a PC, laptop, tablet, handheld computer, PAD, etc.

[0069] It should be noted that, Figure 1 This is just one example; there can be many types of electronic devices, not limited to... Figure 1 The laptop in the middle.

[0070] For example, server 102 can be a single server, a server cluster consisting of multiple servers, or a cloud computing server center. Server 102 may include processors, memory, and network interfaces, etc.

[0071] For example, electronic device 101 can establish a connection and communicate with server 102 through a wireless communication network; for example, electronic device 101 can establish a connection and communicate with server 102 through a wired communication network.

[0072] Electronic device 101 can acquire the target image and send the target image to server 102. Server 102 divides several text lines in the target image into blocks according to the text block division method provided by the present invention, and sends the block division result to electronic device 101.

[0073] In another possible implementation, the hardware architecture involved in this invention may include an electronic device. The electronic device is an electronic product with strong data processing capabilities, such as a PC or mobile phone. The electronic device acquires a target image and divides several lines of text in the target image into blocks according to the text segmentation method provided in this invention.

[0074] Those skilled in the art should understand that the above-described electronic devices and servers are merely examples, and other existing or future electronic devices or servers that are applicable to this invention should also be included within the scope of protection of this invention, and are hereby incorporated by reference.

[0075] The text segmentation method provided by the present invention will be described below through the following embodiments.

[0076] First Embodiment

[0077] Please see Figure 2 The diagram illustrates a flowchart of a text segmentation method provided in an embodiment of the present invention, which may include:

[0078] Step S201: Obtain the text line information corresponding to several text lines in the target image.

[0079] The text line information includes the text line location information and the text line content.

[0080] Specifically, the process of obtaining the text line information corresponding to several text lines in the target image may include:

[0081] Step S2011: Perform text line detection on the target image to obtain the position information of each text line in the target image.

[0082] Figure 3 Taking an image containing 6 text lines as an example, a schematic diagram is given for obtaining the text line information corresponding to each of the several text lines in the image. By performing text line detection on an image containing 6 text lines, the position information of the 6 text lines in the target image can be obtained.

[0083] The text line position information corresponding to a text line can represent the position of the text line in the target image. The text line position information corresponding to a text line can be represented as (x,y,w,h), where (x,y) represents the coordinates of the upper left corner of the text line, w represents the width of the text line, and h represents the height of the text line.

[0084] Optionally, a text line detection model (such as PSENet, DBNet, etc.) can be used to perform text line detection on the target image. Of course, this embodiment is not limited to using a text line detection model to perform text line detection on the target image. Any method that can achieve text line detection is applicable to this invention.

[0085] Step S2012: Based on the text line position information corresponding to each text line in the target image, obtain the image of each text line from the target image.

[0086] After obtaining the text line position information corresponding to each text line in the target image, the image of each text line can be cropped from the target image based on the text line position information. For example... Figure 3 As shown, after obtaining the text line position information corresponding to the 6 text lines in the image, the images of the 6 text lines can be cropped from the image based on the text line position information corresponding to the 6 text lines, so as to obtain the images of the 6 text lines.

[0087] Step S2013: Identify the text line content from the image of each text line to obtain the text line content corresponding to each text line.

[0088] like Figure 3 As shown, after obtaining images of 6 text lines, text content recognition can be performed on the image of each text line to obtain the text content corresponding to the 6 text lines in the image.

[0089] Optionally, an OCR recognition model can be used to recognize the text content of each text line, thereby obtaining the text content of each text line.

[0090] Step S202: Based on the target image and the text line information corresponding to several text lines in the target image, obtain the text line features corresponding to several text lines in the target image.

[0091] Specifically, based on the target image and the text line information corresponding to several text lines in the target image, the semantic features, visual features, and positional features corresponding to several text lines in the target image can be obtained.

[0092] More specifically, semantic features can be obtained from the text content of each text line in the target image; visual features can be obtained from the target image and the position information of each text line in the target image; and position features can be obtained from the position information of each text line in the target image.

[0093] Among them, semantic features are features that can represent the semantic information of the corresponding text line, visual features are features that can represent the visual information of the corresponding text line (such as texture, background, color, etc.), and positional features are features that can represent the positional information of the corresponding text line.

[0094] It should be noted that this embodiment does not limit the text line features to include semantic features, visual features and positional features. The text line features may also include any two or any one of the semantic features, visual features and positional features. In addition, the text line features may include other features related to the text line besides the features listed above.

[0095] Step S203: Divide the text lines in the target image into blocks based on the text line features corresponding to the text lines in the target image.

[0096] Specifically, the process of segmenting several text lines in the target image into blocks based on their respective text line features can include: constructing a target feature sequence that includes block delimiter features, end-of-line features, and text line features corresponding to the several text lines in the target image; at each decoding time, determining the feature with the highest output probability at the current decoding time from the target feature sequence based on the feature with the highest output probability at the previous decoding time, and determining the index of the feature with the highest output probability at the current decoding time as the decoding result at the current decoding time; and determining the segmentation result of several text lines in the target image based on the decoding results at each decoding time.

[0097] This embodiment uses the text line features corresponding to several text lines in the target image as a basis to divide the text lines with semantic relationships in the target image into a text block, so as to obtain the block result of several text lines in the target image.

[0098] The text segmentation method provided in this embodiment of the invention, after obtaining a target image, first acquires the text line information corresponding to several text lines in the target image, then acquires the text line features corresponding to the several text lines in the target image based on the target image and the text line information corresponding to the several text lines in the target image, and finally segments the several text lines in the target image into blocks based on the text line features corresponding to the several text lines in the target image. Therefore, the text segmentation method provided in this embodiment of the invention can achieve the segmentation of several text lines in an image.

[0099] Second Embodiment

[0100] The text segmentation method provided in the above embodiments can be implemented based on a model or not. This embodiment takes a model as an example to introduce the process of implementing text segmentation based on a model.

[0101] Please see Figure 4 The diagram illustrates a process for implementing text segmentation based on a model, which may include:

[0102] Step S401: Obtain the text line information corresponding to several text lines in the target image.

[0103] The text line information includes the text line location information and the text line content.

[0104] The specific implementation process of step S401 can be found in the specific implementation process of step S201 in the above embodiment, and will not be repeated here.

[0105] Step S402: Based on the target image, the text line information corresponding to several text lines in the target image, and the pre-trained text segmentation model, obtain the text line features corresponding to several text lines in the target image.

[0106] Optionally, semantic features, visual features, and positional features corresponding to several text lines in the target image can be obtained based on the target image, the text line information corresponding to several text lines in the target image, and the pre-trained text segmentation model.

[0107] In one possible implementation, the text segmentation model may include an encoding part and a decoding part. It can obtain the semantic features, visual features, and positional features corresponding to the several text lines in the target image based on the target image, the text line information corresponding to several text lines in the target image, and the encoding part of the text segmentation model.

[0108] Step S403: Based on the text line features corresponding to several text lines in the target image, use a text segmentation model to segment the several text lines in the target image into blocks.

[0109] This embodiment uses the semantic features, visual features, and positional features corresponding to several text lines in the target image as a basis, and uses a text segmentation model to determine the segmentation results of several text lines in the target image.

[0110] As mentioned above, a text segmentation model can include an encoding part and a decoding part. It can determine the segmentation result of several text lines in a target image based on the semantic features, visual features, and positional features corresponding to several text lines in the target image, as well as the decoding part of the text segmentation model.

[0111] Considering that the text in different images may be in different languages ​​in practical applications, the initial approach was to model separately for each language. For example, a text segmentation model 'a' was trained for language A, and this model was used to segment text lines in language A within the image. Similarly, a text segmentation model 'b' was trained for language B, and this model was used to segment text lines in language B within the image. Finally, a text segmentation model 'c' was trained for language C, and this model was used to segment text lines in language C within the image. Understandably, modeling separately for each language requires labeling a certain number of segments for each language. Furthermore, text segmentation requires considering the contextual semantic relationships of text lines; therefore, labelers must be proficient in the corresponding language, making text segmentation costly.

[0112] In view of the problems with the above approach, the applicant attempted to propose a solution that could reduce the amount of data required for each language and save annotation costs. The applicant conducted in-depth research on this and finally proposed the following: a model that can uniformly solve the text segmentation task of multiple languages ​​can be obtained by training training data of multiple languages, and the text of multiple languages ​​can be segmented based on the model.

[0113] That is, the text segmentation model in this embodiment is trained using several training samples from multiple languages ​​and corresponding sample labels for each training sample. Each training sample includes a training image, text line information (position information and content of the corresponding text lines) for several text lines in the training image, and the sample label is the actual segmentation result of the several text lines in the training image contained in the corresponding training sample. The training objective of the text segmentation model in this embodiment is to make the predicted segmentation result for several text lines in the training image as consistent with the actual segmentation result of several text lines in the training image.

[0114] Because the text segmentation model in this embodiment uses training data from multiple languages, and is trained with the goal of making the predicted segmentation results for several text lines in the training image as consistent as possible with the actual segmentation results for several text lines in the training image, it has the ability to segment text lines in images of multiple languages. This allows the present invention to segment text lines in images of multiple languages ​​based on a single text segmentation model. For example, the text segmentation model trained using training data from languages ​​A, B, and C can segment text lines in images of language A (where the text in the image is in language A), images of language B (where the text in the image is in language B), and images of language C (where the text in the image is in language C).

[0115] This invention provides a text segmentation method. After obtaining a target image, the method first acquires the text line information corresponding to several text lines in the target image. Then, based on the target image, the text line information corresponding to the several text lines in the target image, and a pre-trained text segmentation model, the several text lines in the target image are segmented into blocks. Therefore, this invention provides a text segmentation method that can segment several text lines in an image. Furthermore, since the text segmentation model in this invention uses several training samples from multiple languages ​​and corresponding sample labels, and is trained with the goal of making the predicted segmentation results for several text lines in the training image as consistent with the actual segmentation results for several text lines in the training image, it has the ability to segment text lines in images of multiple languages. In other words, this invention provides a text segmentation method that can segment text lines in images of multiple languages.

[0116] Third Embodiment

[0117] This embodiment describes the process described in the above embodiment of "obtaining semantic features, visual features, and positional features corresponding to several text lines in the target image based on the target image, the text line information corresponding to several text lines in the target image, and the encoding part of the text segmentation model".

[0118] In one possible implementation, such as Figure 5 As shown, the encoding part of the text segmentation model can include a semantic feature acquisition module, a visual feature acquisition module, and a positional feature acquisition module. A flowchart illustrating the process of acquiring the semantic, visual, and positional features corresponding to the text lines in the target image based on the target image, the text line information corresponding to several text lines in the target image, and the encoding part of the text segmentation model can include:

[0119] Step a1: Based on the text content corresponding to several text lines in the target image and the semantic feature acquisition module of the text segmentation model, obtain the semantic features corresponding to several text lines in the target image.

[0120] To obtain the semantic features corresponding to several lines of text in the target image, one possible implementation is to first segment the text content of each line of text according to the constructed dictionary to obtain the segmentation unit sequence corresponding to the text line. Then, from the word vector table composed of word vectors corresponding to each word in the dictionary, the word vector of each segmentation unit in the segmentation unit sequence corresponding to the text line is obtained (for each segmentation unit in the segmentation unit sequence, the word vector corresponding to the segmentation unit is obtained from the word vector table according to the index of the segmentation unit in the dictionary) to obtain the word vector sequence corresponding to the text line. The word vector sequence corresponding to the text line is then input into the semantic feature acquisition module of the text segmentation model to obtain the semantic features corresponding to the text line output by the semantic feature acquisition module.

[0121] Considering that this invention requires segmenting text lines in images of multiple languages, the dictionary needs to encompass all word segmentation units contained in multiple languages. For example, if the multiple languages ​​include English and French, and the word segmentation unit is a word, then the dictionary needs to include all words in English and all words in French. The dictionary encompassing all word segmentation units of multiple languages ​​makes the dictionary very long, and the dictionary needs to be updated every time a new language is added.

[0122] In view of this, this embodiment provides another preferred implementation method, specifically, performing the following for each line of text in the target image:

[0123] Step a11: Segment the text content corresponding to the text line according to the characters to obtain the segmentation unit sequence corresponding to the text line.

[0124] Each segmentation unit in the sequence of segmentation units obtained by segmenting the text line content corresponding to the text line by character is one character.

[0125] For example, for a line of English text, the text content corresponding to that line is segmented by character, that is, by English letter, and the segmentation result is a string of English letters, that is, a sequence of English letters.

[0126] Step a12: Obtain the encoding of each word segmentation unit in the word segmentation unit sequence corresponding to the text line, and obtain the encoding sequence corresponding to the text line.

[0127] Optionally, the Unicode encoding of each word segment in the sequence of word segments corresponding to the text line can be obtained.

[0128] Unicode is an important universal character encoding standard for interaction and display. It covers languages ​​from the United States, Europe, the Middle East, Africa, India, Asia and the Pacific, as well as ancient scripts and technical symbols. Considering that characters in most languages ​​have unique Unicode encodings, this invention, after obtaining the word segmentation unit sequence corresponding to the text line, can obtain the Unicode encoding of each word segmentation unit in the word segmentation unit sequence corresponding to the text line, thereby obtaining the Unicode encoding sequence corresponding to the text line.

[0129] Step a13: Decompose each code in the encoding sequence corresponding to the text line into bytes to obtain the byte sequence corresponding to the text line.

[0130] It should be noted that each code in the encoding sequence corresponding to this text line consists of multiple bytes, with one byte being 8 bits.

[0131] Step a14: Obtain the word vector corresponding to the number represented by each byte in the byte sequence corresponding to the text line, so as to obtain the word vector sequence corresponding to the text line.

[0132] Each byte represents a number between 0 and 255.

[0133] When obtaining the word vector corresponding to the number represented by each byte in the byte sequence corresponding to the text line, the word vector corresponding to the number represented by each byte in the byte sequence corresponding to the text line can be obtained from a pre-built word vector table.

[0134] The word vector table includes word vectors corresponding to the indices of each virtual word in the constructed virtual word table. The word vector corresponding to the number represented by one byte is the word vector corresponding to the index in the word vector table that is the same as the number represented by that byte.

[0135] Optionally, considering that each byte represents a number from 0 to 255, the constructed virtual word list may include 256 virtual words, with indices from 0 to 255. The word vector table includes word vectors corresponding to indices 0 to 255. Optionally, the 256 virtual words in the virtual word list can be the same as their indices, i.e., from 0 to 255.

[0136] For example, if the number represented by one byte is 156, then the word vector corresponding to index 156 is obtained from the word vector table, and the word vector corresponding to index 156 is used as the word vector corresponding to that byte.

[0137] Step a15: Input the word vector sequence corresponding to the text line into the semantic feature acquisition module of the text segmentation model to obtain the semantic features corresponding to the text line output by the semantic feature acquisition module.

[0138] Step a2: Based on the target image, the text line position information corresponding to several text lines in the target image, and the visual feature acquisition module of the text segmentation model, obtain the visual features corresponding to several text lines in the target image.

[0139] Specifically, the target image can first be input into the visual feature acquisition module of the text segmentation model to obtain the overall visual features of the target image output by the visual feature acquisition module. Then, based on the text line position information corresponding to several text lines in the target image, the visual features corresponding to several text lines in the target image can be obtained from the overall visual features of the target image.

[0140] Optionally, the RoiAlign algorithm can be used to extract the visual features corresponding to several text lines in the target image from the overall visual features of the target image based on the text line position information corresponding to several text lines in the target image.

[0141] Step a3: Based on the text line position information corresponding to several text lines in the target image, and the position feature acquisition module of the text segmentation model, obtain the position features corresponding to several text lines in the target image.

[0142] Specifically, for each text line in the target image, the text line location information corresponding to that text line can be input into the location feature acquisition module of the text segmentation model to obtain the location feature corresponding to that text line output by the location feature acquisition module.

[0143] The above embodiment mentions that the text line position information corresponding to a text line can be represented as (x,y,w,h). After the four-dimensional text line position information (x,y,w,h) is input into the position feature acquisition module of the text segmentation model, the position feature acquisition module can map the four-dimensional text line position information (x,y,w,h) into a feature of a specified dimension (such as 128 dimensions) as the position feature corresponding to the text line.

[0144] It should be noted that this embodiment does not limit the execution order of steps a1 to a3. These three steps can be executed in any order or in parallel.

[0145] Fourth embodiment

[0146] This embodiment describes the implementation process of "determining the segmentation result of several text lines in the target image based on the semantic features, visual features, and positional features corresponding to several text lines in the target image, as well as the decoding part of the text segmentation model" in the above embodiment.

[0147] Please see Figure 6This document illustrates a flowchart illustrating the process of determining the segmentation results of several text lines in a target image based on their semantic, visual, and positional features, as well as the decoding part of a text segmentation model. The flowchart may include:

[0148] Step S601: Construct a target feature sequence including block delimiter features, end delimiter features, and text line features corresponding to several text lines in the target image.

[0149] The text line feature is obtained by fusing the semantic features, visual features and positional features corresponding to the corresponding text line. Optionally, the semantic features, visual features and positional features corresponding to a text line can be concatenated into a single feature in a set order, which is used as the text line feature corresponding to that text line.

[0150] Step S602: At each decoding time, based on the feature with the highest output probability at the previous decoding time and the decoding part of the text block model, determine the feature with the highest output probability at the current decoding time from the target feature sequence, and determine the index of the feature with the highest output probability at the current decoding time as the decoding result at the current decoding time.

[0151] Specifically, based on the feature with the highest output probability at the previous decoding time and the decoding part of the text segmentation model, the feature with the highest output probability at the current decoding time is determined from the target feature sequence, including:

[0152] Step S6021: Based on the feature with the highest output probability at the previous decoding time and the decoding part of the text block model, determine the state feature vector that can represent the information of the decoded result.

[0153] Optional, such as Figure 7 As shown, the decoding part of the text segmentation model may include: a state feature vector determination module, which can input the feature with the highest output probability at the previous decoding time step into the state feature vector determination module to obtain a state feature vector that can represent the information of the decoded result. Optionally, the state feature vector determination module can use an LSTM, inputting the feature with the highest output probability at the previous decoding time step into the LSTM. The LSTM processes the latent features and the input features, and outputs a state feature vector that can represent the information of the decoded result.

[0154] Step S6022: Based on the decoding part of the text segmentation model, calculate the similarity between the state feature vector and each feature in the target feature sequence.

[0155] like Figure 7As shown, the decoding part of the text segmentation model can also include an attention calculation module. The state feature vector and the target feature sequence are input into the attention calculation module, which calculates the similarity between the state feature vector and each feature in the target feature sequence.

[0156] Step S6023: Based on the decoding part of the text segmentation model, normalize the similarity between the state feature vector and each feature in the target feature sequence to obtain the output probability of each feature in the feature sequence at the current decoding time.

[0157] like Figure 7 As shown, the decoding part of the text segmentation model can also include a normalization module. The similarity between the state feature vector and each feature in the target feature sequence is input into the normalization module. The normalization module normalizes each similarity to a value between 0 and 1. The normalized value of the similarity between the state feature vector and a feature in the target feature sequence is used as the output probability of that feature at the current decoding time. This allows obtaining the output probability of each feature in the feature sequence at the current decoding time. Optionally, the normalization module can use a softmax layer.

[0158] For example, if the target feature sequence is {f1,f2,f3,f4,f5}, after calculating the similarity s1, s2, s3, s4, and s5 between f1, f2, f3, f4, and f5 and the state feature vector respectively, s1, s2, s3, s4, and s5 can be passed through a softmax layer to obtain the output probabilities of f1, f2, f3, f4, and f5.

[0159] Step S6024: Based on the output probability of each feature in the target feature sequence at the current decoding time, determine the feature with the highest output probability at the current decoding time from the feature sequence.

[0160] After obtaining the output probability of each feature in the target feature sequence at the current decoding time, the feature with the highest output probability at the current decoding time can be obtained. Then, the index of the feature with the highest output probability at the current decoding time can be used as the decoding result at the current decoding time.

[0161] It should be noted that each feature in the target feature sequence has an index. For example, if the target image has 5 lines of text, the target feature sequence includes block delimiter features, end-of-line features, and the text line features corresponding to the 5 lines of text in the target image. The index of the block delimiter feature can be set to 1, the index of the end-of-line feature to 2, the index of the text line feature corresponding to the 1st text line to 3, the index of the text line feature corresponding to the 2nd text line to 4, the index of the text line feature corresponding to the 3rd text line to 5, the index of the text line feature corresponding to the 4th text line to 6, and the index of the text line feature corresponding to the 5th text line to 7. If the feature with the highest output probability at the current decoding time is the text line feature corresponding to the 3rd text line, then the index 5 of the text line feature corresponding to the 3rd text line is taken as the decoding result at the current decoding time.

[0162] Step S603: Based on the decoding results at each decoding time, determine the block results of several text lines in the target image.

[0163] For example, the target image contains 5 text lines. In the target feature sequence, the index of the block delimiter feature is 1, the index of the end-of-line feature is 2, the index of the text line feature corresponding to the 1st text line is 3, the index of the text line feature corresponding to the 2nd text line is 4, the index of the text line feature corresponding to the 3rd text line is 5, the index of the text line feature corresponding to the 4th text line is 6, and the index of the text line feature corresponding to the 5th text line is 7. The decoding results at each decoding time are 3, 4, 5, 1, 6, 7, 2 respectively. According to the decoding results at each decoding time, it can be seen that the text lines with feature indices 3, 4, and 5 correspond to the text lines respectively. The text lines corresponding to the features with feature indices 6 and 7 are divided into one text block. The text lines corresponding to the features with feature indices 3, 4, and 5 are the first, second, and third text lines in the target image, respectively. The text lines corresponding to the features with feature indices 6 and 7 are the fourth and fifth text lines in the target image, respectively. In other words, the final block division result is that the first, second, and third text lines in the target image are divided into one text block, and the fourth and fifth text lines in the target image are divided into another text block.

[0164] Fifth Embodiment

[0165] This embodiment introduces the training process of the text segmentation model.

[0166] Please see Figure 8 The diagram illustrates the training process of a text segmentation model, which may include:

[0167] Step S801: Obtain a training sample from several training samples in multiple languages ​​as the current training sample.

[0168] It should be noted that the training samples for multiple languages ​​are training samples that simultaneously cover multiple languages, and each training sample is a training sample of one or more languages ​​from multiple languages.

[0169] For example, the multiple languages ​​are Chinese, English, and French. In the first possible implementation, the training samples may include Chinese training samples (single language), English training samples (single language), and French training samples (single language). In the second possible implementation, the training samples may include single-language training samples that simultaneously cover Chinese, English, and French, as well as multilingual mixed training samples, such as Chinese training samples (single language), Chinese-English mixed training samples, and Chinese-French mixed training samples. In the third possible implementation, the training samples may include multilingual mixed training samples that simultaneously cover Chinese, English, and French, such as Chinese-English mixed training samples, Chinese-French mixed training samples, and English-French mixed training samples.

[0170] As mentioned in the above embodiments, each training sample includes a training image and text line information corresponding to several text lines in the training image. The training sample of a certain language mentioned above refers to the text in the training image contained in the training sample being text in that language. For example, the text in the training image contained in the Chinese training sample is Chinese text, the text in the training image contained in the English training sample is English text, and the text in the training image contained in the mixed Chinese and English training sample is mixed Chinese and English text.

[0171] Step S802: Based on the current training sample and the encoding part of the text segmentation model, obtain the semantic features, visual features and positional features corresponding to several text lines in the training image contained in the current training sample.

[0172] Specifically, the implementation process of step S802 may include: obtaining semantic features corresponding to several text lines in the training images contained in the current training sample based on the text line content corresponding to several text lines in the training images contained in the current training sample, and the semantic feature acquisition module of the text segmentation model; obtaining visual features corresponding to several text lines in the training images contained in the current training sample based on the training images contained in the current training sample, the text line position information corresponding to several text lines in the training images contained in the current training sample, and the visual feature acquisition module of the text segmentation model; and obtaining position features corresponding to several text lines in the training images contained in the current training sample based on the text line position information corresponding to several text lines in the training images contained in the current training sample, and the position feature acquisition module of the text segmentation model.

[0173] To facilitate the training of the semantic feature acquisition module for unified multilingual modeling, this invention adopts a byte-based modeling approach. Specifically, when acquiring the visual features corresponding to several text lines in the training images contained in the current training sample, for each text line in the training images contained in the current training sample: the text line content corresponding to the text line is segmented by character to obtain a sequence of segmentation units corresponding to that text line; the encoding (e.g., Unicode encoding) of each segmentation unit in the sequence of segmentation units corresponding to that text line is obtained to obtain an encoding sequence corresponding to that text line; each encoding in the encoding sequence corresponding to that text line is decomposed by byte to obtain a byte sequence corresponding to that text line; and the word vector corresponding to the number represented by each byte in the byte sequence corresponding to that text line is obtained from the word vector table (which may include the word vectors corresponding to the indices of the 256 virtual words in the constructed virtual vocabulary) to obtain the word vector sequence corresponding to that text line.

[0174] Optionally, the semantic feature acquisition module in the initial text segmentation model can adopt a language model pre-trained on multilingual corpora by setting a prediction task. For example, to train a text segmentation model applicable to languages ​​A, B, and C, a language model pre-trained on large-scale corpora of languages ​​A, B, and C can be used.

[0175] Optionally, the prediction task can be set as predicting whether two sentences have a contextual relationship. During pre-training, the semantic features corresponding to the two sentences can be obtained based on the language model. Then, the probability that the two sentences have a contextual relationship can be predicted based on the semantic features corresponding to the two sentences. The prediction loss is determined based on the predicted probability and the annotation information of the two sentences (used to indicate whether the two sentences have a contextual relationship). The parameters of the language model are updated according to the prediction loss.

[0176] It should be noted that the specific implementation process of step S802 is similar to the specific implementation process of "obtaining the semantic features, visual features and positional features corresponding to several text lines in the target image based on the target image, the text line information corresponding to several text lines in the target image, and the encoding part of the text block model". For a more specific implementation process of step S802, please refer to the relevant parts of the above embodiments. This embodiment will not repeat it here.

[0177] Step S803: Construct a training feature sequence including block delimiter features, end delimiter features, and text line features corresponding to several text lines in the training image contained in the current training sample.

[0178] Among them, the text line feature is obtained by fusing the semantic features, visual features and positional features corresponding to the corresponding text line. Specifically, the semantic features, visual features and positional features corresponding to a text line can be concatenated into a feature in a set order, which is used as the text line feature corresponding to that text line.

[0179] Step S804: Based on the decoding part of the text segmentation model, predict the output probability of each feature of the training feature sequence at each decoding time.

[0180] Specifically, in the decoding part of the text segmentation model, the process of predicting the output probability of each feature in the training feature sequence at each decoding time step can include:

[0181] Step S8041: At each decoding time, based on the feature with the highest output probability at the previous decoding time and the decoding part of the text block model, determine the state feature vector that can represent the information of the decoded result.

[0182] Specifically, the feature with the highest output probability at the previous decoding time can be input into the state feature vector determination module (such as LSTM) of the text block model to obtain the state feature vector output by the state feature vector determination module that can represent the information of the decoded result.

[0183] Step S8042: Based on the decoding part of the text segmentation model, calculate the similarity between the state feature vector and each feature in the training feature sequence.

[0184] Specifically, the state feature vector and the training feature sequence can be input into the attention calculation module of the text segmentation model. The attention calculation module calculates the similarity between the state feature vector and each feature in the training feature sequence.

[0185] Step S8043: Based on the decoding part of the text segmentation model, normalize the similarity between the state feature vector and each feature in the training feature sequence to obtain the output probability of each feature in the training feature sequence at the current decoding time.

[0186] Specifically, the similarity between the state feature vector and each feature in the training feature sequence is input into the normalization module (such as the softmax layer) of the text segmentation model. The normalization module normalizes each similarity to a value between 0 and 1. The normalized value of the similarity between the state feature vector and a feature in the training feature sequence is used as the output probability of that feature at the current decoding time. In this way, the output probability of each feature in the training feature sequence at the current decoding time can be obtained.

[0187] Step S805: Determine the prediction loss based on the output probability of each feature in the training feature sequence at each decoding time and the sample label corresponding to the current training sample.

[0188] Specifically, the cross-entropy loss can be calculated based on the output probability of each feature in the training feature sequence at each decoding time and the sample label corresponding to the current training sample. The calculation method of the cross-entropy loss is an existing technology, which will not be elaborated here in this embodiment.

[0189] Step S806: Update the parameters of the text segmentation model based on the prediction loss.

[0190] The text segmentation model is trained multiple times according to the above steps S801 to S806 until the preset number of training times is reached or the model converges.

[0191] The text segmentation model trained through the above process is capable of segmenting text lines in images of multiple languages.

[0192] This invention enables the training of a unified model capable of solving text segmentation tasks across multiple languages ​​using training data from multiple languages. Compared to training a model separately for each language, this reduces annotation costs. For example, if a good text segmentation model for a single language requires 20,000 training data points, then theoretically, using unified modeling for two languages ​​might only require 10,000 training data points per language to train a good model. Furthermore, the "multiple languages" mentioned in this invention can refer to multiple languages ​​within the same language family or multiple languages ​​within different language families. That is, this invention can train a unified model capable of solving text segmentation tasks across multiple languages ​​within the same language family using training data from multiple languages ​​within different language families, and it can also train a unified model capable of solving text segmentation tasks across multiple languages ​​within different language families using training data from multiple languages ​​within different language families.

[0193] Sixth Embodiment

[0194] Based on the text segmentation method provided in the above embodiments, this embodiment provides a text-image translation method, which may include:

[0195] Step b1: Obtain the target image.

[0196] The target image is an image containing several lines of text.

[0197] Step b2: Using the text segmentation method provided in the above embodiment, segment several text lines in the target image into blocks.

[0198] That is, the text line information corresponding to several text lines in the target image is obtained. The text line information includes text line position information and text line content. Based on the target image, the text line information corresponding to several text lines in the target image, and a pre-trained text segmentation model, the several text lines in the target image are segmented into blocks. The text segmentation model is trained using several training samples in multiple languages ​​and sample labels corresponding to the training samples. Each training sample includes a training image, the text line information corresponding to several text lines in the training image, and the sample label is the actual segmentation result of the several text lines in the training image contained in the corresponding training sample.

[0199] For a more detailed implementation of the text segmentation method and the training process of the text segmentation model, please refer to the above embodiments, which will not be repeated here.

[0200] Step b3: For each obtained text block, connect the text lines contained in the text block into a sentence in the order of reading, and translate the sentence.

[0201] The text translation method provided in this invention translates text blocks as units. Since the text lines in a text block have semantic relationships, translating text blocks as units can obtain more accurate, coherent and fluent translations.

[0202] Seventh Embodiment

[0203] This invention also provides a text segmentation device. The text segmentation device provided in this invention will be described below. The text segmentation device described below can be referred to in correspondence with the text segmentation method described above.

[0204] Please see Figure 9 The diagram shows a structural schematic of a text segmentation device provided in an embodiment of the present invention, which may include: a text line information acquisition module 901, a text line feature acquisition module 902, and a text segmentation module 903.

[0205] The text line information acquisition module 901 is used to acquire text line information corresponding to several text lines in the target image, wherein the text line information includes text line position information and text line content.

[0206] The text line feature acquisition module 902 is used to acquire text line features corresponding to several text lines in the target image based on the target image and the text line information corresponding to several text lines in the target image.

[0207] The text segmentation module 903 is used to segment the text lines in the target image into blocks based on the text line features corresponding to the text lines in the target image.

[0208] Optionally, when the text line feature acquisition module 902 acquires the text line features corresponding to several text lines in the target image based on the target image and the text line information corresponding to several text lines in the target image, it is specifically used for:

[0209] Based on the target image and the text line information corresponding to several text lines in the target image, the semantic features, visual features, and positional features corresponding to the several text lines in the target image are obtained respectively.

[0210] Optionally, when the text line feature acquisition module 902 acquires the semantic features, visual features, and positional features corresponding to the several text lines in the target image based on the target image and the text line information corresponding to the several text lines in the target image, it is specifically used for:

[0211] Based on the text content corresponding to several text lines in the target image, obtain the semantic features corresponding to several text lines in the target image.

[0212] Based on the target image and the text line position information corresponding to the text lines in the target image, obtain the visual features corresponding to the text lines in the target image.

[0213] Based on the text line position information corresponding to several text lines in the target image, obtain the position features corresponding to several text lines in the target image.

[0214] Optionally, when the text line feature acquisition module 902 acquires the semantic features corresponding to the text lines in the target image based on the text line content corresponding to the text lines in the target image, it is specifically used for:

[0215] For each line of text in the target image:

[0216] The text line content corresponding to the text line is segmented according to the characters to obtain the segmentation unit sequence corresponding to the text line;

[0217] Obtain the encoding of each word segmentation unit in the word segmentation unit sequence corresponding to the text line to obtain the encoding sequence corresponding to the text line, wherein the encoding of each word segmentation unit includes several bytes;

[0218] Decompose each code in the encoding sequence corresponding to the text line into bytes to obtain the byte sequence corresponding to the text line.

[0219] Obtain the word vector corresponding to the number represented by each byte in the byte sequence corresponding to the text line, so as to obtain the word vector sequence corresponding to the text line;

[0220] Based on the word vector sequence corresponding to the text line, obtain the semantic features corresponding to the text line.

[0221] Optionally, when the text line feature acquisition module 902 acquires the word vector corresponding to the number represented by each byte in the byte sequence corresponding to the text line, it is specifically used for:

[0222] From the pre-built word vector table, obtain the word vector corresponding to the number represented by each byte in the byte sequence corresponding to the text line;

[0223] The word vector table includes word vectors corresponding to the indices of each virtual word in the constructed virtual word table, and the word vector corresponding to the number represented by one byte is the word vector corresponding to the index in the word vector table that is the same as the number represented by that byte.

[0224] Optionally, when the text line feature acquisition module 902 acquires the visual features corresponding to the several text lines in the target image based on the target image and the text line position information corresponding to the several text lines in the target image, it is specifically used for:

[0225] Based on the target image, obtain the overall visual features of the target image;

[0226] Based on the text line position information corresponding to several text lines in the target image, the visual features corresponding to several text lines in the target image are obtained from the overall visual features of the target image.

[0227] Optionally, when the text segmentation module 903 segments the text lines in the target image according to the text line features corresponding to the text lines in the target image, it is specifically used for:

[0228] Construct a target feature sequence that includes block delimiter features, end-of-image features, and text line features corresponding to several text lines in the target image;

[0229] At each decoding moment, based on the feature with the highest output probability at the previous decoding moment, the feature with the highest output probability at the current decoding moment is determined from the target feature sequence, and the index of the feature with the highest output probability at the current decoding moment is determined as the decoding result at the current decoding moment;

[0230] Based on the decoding results at each decoding time, the block results of several text lines in the target image are determined.

[0231] Optionally, when the text segmentation module 903 determines the feature with the highest output probability at the current decoding time from the target feature sequence based on the feature with the highest output probability at the previous decoding time, it is specifically used for:

[0232] Based on the feature with the highest output probability at the previous decoding time, determine the state feature vector that can represent the information of the decoded result;

[0233] Calculate the similarity between the state feature vector and each feature in the target feature sequence;

[0234] The similarity between the state feature vector and each feature in the target feature sequence is normalized to obtain the output probability of each feature in the target feature sequence at the current decoding time.

[0235] Based on the output probability of each feature in the target feature sequence at the current decoding time, determine the feature with the highest output probability at the current decoding time from the target feature sequence.

[0236] Optionally, the text line feature acquisition module 902, when acquiring the text line features corresponding to several text lines in the target image based on the target image and the text line information corresponding to several text lines in the target image, is specifically used for:

[0237] Based on the target image, the text line information corresponding to several text lines in the target image, and the pre-trained text segmentation model, the text line features corresponding to several text lines in the target image are obtained.

[0238] The text segmentation module 903, when segmenting several text lines in the target image according to their respective text line features, is specifically used for:

[0239] Based on the text line features corresponding to several text lines in the target image, the text line segmentation model is used to segment the several text lines in the target image into blocks.

[0240] The text segmentation model is trained using several training samples from multiple languages ​​and sample labels corresponding to those training samples. Each training sample includes a training image and text line information corresponding to several text lines in the training image. The sample label is the actual segmentation result of several text lines in the training image contained in the corresponding training sample.

[0241] This invention provides a text segmentation device that, after obtaining a target image, first acquires text line information corresponding to several text lines in the target image. Then, based on the target image, the text line information corresponding to the several text lines in the target image, and a pre-trained text segmentation model, it segments the several text lines in the target image into blocks. Therefore, this invention provides a text segmentation device capable of segmenting several text lines in an image. Furthermore, since the text segmentation model in this invention uses several training samples from multiple languages ​​and corresponding sample labels for each training sample, and is trained with the goal of making the predicted segmentation results for several text lines in the training image as consistent with the actual segmentation results for several text lines in the training image, it has the ability to segment text lines in images of multiple languages. In other words, this invention provides a text segmentation device capable of segmenting text lines in images of multiple languages.

[0242] Eighth embodiment

[0243] This invention also provides a text segmentation device; please refer to [link to relevant documentation]. Figure 10 The diagram shows the structure of the text segmentation device, which may include: at least one processor 1001, at least one communication interface 1002, at least one memory 1003 and at least one communication bus 1004.

[0244] In this embodiment of the invention, the number of processor 1001, communication interface 1002, memory 1003 and communication bus 1004 is at least one, and processor 1001, communication interface 1002 and memory 1003 communicate with each other through communication bus 1004.

[0245] The processor 1001 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0246] The memory 1003 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;

[0247] The memory stores a program, which the processor can call. The program is used for:

[0248] Obtain text line information corresponding to several text lines in the target image, wherein the text line information includes text line position information and text line content;

[0249] Based on the target image and the text line information corresponding to several text lines in the target image, obtain the text line features corresponding to several text lines in the target image.

[0250] Based on the text line features corresponding to the text lines in the target image, the text lines in the target image are divided into blocks.

[0251] The text segmentation model is trained using several training samples from multiple languages ​​and sample labels corresponding to those training samples. Each training sample includes a training image and text line information corresponding to several text lines in the training image. The sample label is the actual segmentation result of several text lines in the training image contained in the corresponding training sample.

[0252] Optionally, the refined and extended functions of the program can be found in the description above.

[0253] Ninth Embodiment

[0254] This invention also provides a readable storage medium that stores a program suitable for execution by a processor, the program being used for:

[0255] Obtain text line information corresponding to several text lines in the target image, wherein the text line information includes text line position information and text line content;

[0256] Based on the target image and the text line information corresponding to several text lines in the target image, obtain the text line features corresponding to several text lines in the target image.

[0257] Based on the text line features corresponding to the text lines in the target image, the text lines in the target image are divided into blocks.

[0258] Optionally, the refined and extended functions of the program can be found in the description above.

[0259] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0260] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0261] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method of text chunking, characterized by, include: Obtain text line information corresponding to several text lines in the target image, wherein the text line information includes text line position information and text line content; Based on the target image and the text line information corresponding to several text lines in the target image, obtain the text line features corresponding to several text lines in the target image. Based on the text line features corresponding to the text lines in the target image, the text lines in the target image are divided into blocks; The step of obtaining text line features corresponding to several text lines in the target image based on the target image and the text line information corresponding to several text lines in the target image; and dividing the several text lines in the target image into blocks based on the text line features corresponding to several text lines in the target image, includes: Based on the pre-trained text segmentation model, the target image and the text line information corresponding to several text lines in the target image are processed to obtain the segmentation results of several text lines in the target image. The text segmentation model is trained using several training samples from multiple languages ​​and sample labels corresponding to those training samples. Each training sample includes a training image and text line information corresponding to several text lines in the training image. The sample label is the actual segmentation result of several text lines in the training image contained in the corresponding training sample.

2. The text chunking method of claim 1, wherein, The step of obtaining text line features corresponding to several text lines in the target image based on the target image and the text line information corresponding to several text lines in the target image includes: Based on the target image and the text line information corresponding to several text lines in the target image, the semantic features, visual features, and positional features corresponding to the several text lines in the target image are obtained respectively.

3. The text chunking method of claim 2, wherein, The step of obtaining semantic features, visual features, and positional features corresponding to the several text lines in the target image based on the target image and the text line information corresponding to the several text lines in the target image includes: Based on the text content corresponding to several text lines in the target image, obtain the semantic features corresponding to several text lines in the target image. Based on the target image and the text line position information corresponding to the text lines in the target image, obtain the visual features corresponding to the text lines in the target image. Based on the text line position information corresponding to several text lines in the target image, obtain the position features corresponding to several text lines in the target image.

4. The text segmentation method according to claim 3, characterized in that, The step of obtaining semantic features corresponding to several text lines in the target image based on their respective text line contents includes: For each line of text in the target image: The text line content corresponding to the text line is segmented according to the characters to obtain the segmentation unit sequence corresponding to the text line; Obtain the encoding of each word segmentation unit in the word segmentation unit sequence corresponding to the text line to obtain the encoding sequence corresponding to the text line, wherein the encoding of each word segmentation unit includes several bytes; Decompose each code in the encoding sequence corresponding to the text line into bytes to obtain the byte sequence corresponding to the text line. Obtain the word vector corresponding to the number represented by each byte in the byte sequence corresponding to the text line, so as to obtain the word vector sequence corresponding to the text line; Based on the word vector sequence corresponding to the text line, obtain the semantic features corresponding to the text line.

5. The text segmentation method according to claim 4, characterized in that, The step of obtaining the word vector corresponding to the number represented by each byte in the byte sequence corresponding to the text line includes: From the pre-built word vector table, obtain the word vector corresponding to the number represented by each byte in the byte sequence corresponding to the text line; The word vector table includes word vectors corresponding to the indices of each virtual word in the constructed virtual word table, and the word vector corresponding to the number represented by one byte is the word vector corresponding to the index in the word vector table that is the same as the number represented by that byte.

6. The text segmentation method according to claim 3, characterized in that, The step of obtaining the visual features corresponding to the several text lines in the target image based on the target image and the text line position information corresponding to the several text lines in the target image includes: Based on the target image, obtain the overall visual features of the target image; Based on the text line position information corresponding to several text lines in the target image, the visual features corresponding to several text lines in the target image are obtained from the overall visual features of the target image.

7. The text segmentation method according to claim 1, characterized in that, The step of dividing the target image into blocks based on the text line features corresponding to the text lines in the target image includes: Construct a target feature sequence that includes block delimiter features, end-of-image features, and text line features corresponding to several text lines in the target image; At each decoding moment, based on the feature with the highest output probability at the previous decoding moment, the feature with the highest output probability at the current decoding moment is determined from the target feature sequence, and the index of the feature with the highest output probability at the current decoding moment is determined as the decoding result at the current decoding moment; Based on the decoding results at each decoding time, the block results of several text lines in the target image are determined.

8. The text segmentation method according to claim 7, characterized in that, The step of determining the feature with the highest output probability at the current decoding time from the target feature sequence based on the feature with the highest output probability at the previous decoding time includes: Based on the feature with the highest output probability at the previous decoding time, determine the state feature vector that can represent the information of the decoded result; Calculate the similarity between the state feature vector and each feature in the target feature sequence; The similarity between the state feature vector and each feature in the target feature sequence is normalized to obtain the output probability of each feature in the target feature sequence at the current decoding time. Based on the output probability of each feature in the target feature sequence at the current decoding time, determine the feature with the highest output probability at the current decoding time from the target feature sequence.

9. A method for translating text into images, characterized in that, include: Obtain the target image; The text segmentation method as described in any one of claims 1 to 8 is used to segment several text lines in the target image into blocks; For each obtained text block, the text lines contained in the text block are connected into a sentence in the order of reading, and the sentence is translated.

10. A text segmentation device, characterized in that, It includes: a text line information acquisition module, a text line feature acquisition module, and a text segmentation module; The text line information acquisition module is used to acquire text line information corresponding to several text lines in the target image, wherein the text line information includes text line position information and text line content; The text line feature acquisition module is used to acquire text line features corresponding to several text lines in the target image based on the target image and the text line information corresponding to several text lines in the target image. The text segmentation module is used to segment the text lines in the target image into blocks based on the text line features corresponding to the text lines in the target image. The text line feature acquisition module is specifically used to process the target image and the text line information corresponding to several text lines in the target image based on the pre-trained text segmentation model, so as to obtain the text line features corresponding to several text lines in the target image. The text segmentation module is specifically used to use the text segmentation model to segment several text lines in the target image into blocks based on the text line features corresponding to the several text lines in the target image. The text segmentation model is trained using several training samples from multiple languages ​​and sample labels corresponding to those training samples. Each training sample includes a training image and text line information corresponding to several text lines in the training image. The sample label is the actual segmentation result of several text lines in the training image contained in the corresponding training sample.

11. A text segmentation device, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is configured to execute the program to implement the various steps of the text segmentation method as described in any one of claims 1 to 8.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the various steps of the text segmentation method as described in any one of claims 1 to 8.

Images