A text splicing method and device, electronic equipment and storage medium

Abstract

The application provides a text splicing method and device, electronic equipment and storage medium. The method comprises the following steps: determining each text block in a target image; determining the number mark of each text block and marking the number mark at the preset position of the corresponding text block to obtain a marked target image; inputting the marked target image into a number sorting model to obtain the number mark sorting output by the number sorting model; the number sorting model is obtained by training an initial number sorting model based on sample images and corresponding sorting labels; the sample image is an image added with a sample number mark; the sorting label is obtained by sorting the sample number mark based on the order of the sample text; and each text block is spliced based on the number mark sorting to obtain spliced text. The number mark can enhance the image information, and then the strengthened image features can be extracted, which is beneficial to the accurate prediction of the number mark sorting by the model, and the spliced text with high accuracy can be obtained based on the sorting.

Description

Technical Field

[0001] This invention relates to the field of information processing technology, and in particular to a text splicing method, apparatus, electronic device, and storage medium. Background Technology

[0002] In practical applications, many business scenarios require the use of concatenated text. For example, in image-to-text transfer scenarios, it is necessary to first identify all text in the image, then concatenate the identified text to obtain the complete text corresponding to the image, and finally store this complete text in the storage medium of the electronic device to achieve image-to-text transfer. Another example is in the structured extraction of image information scenarios, where it is necessary to extract target text from the image according to business requirements. In this case, it is also necessary to concatenate the identified text in the image to obtain the complete text corresponding to the image, and then extract the target text based on this complete text.

[0003] For text splicing, current technologies typically rely on pre-defined rules, such as arranging text according to its relative position within an image, following a top-to-bottom or left-to-right order. However, this method is limited by the fixed nature of these rules, leading to incorrect text order. For instance, if the reading order of some text in an image is not top-to-bottom or left-to-right, the resulting spliced ​​text may be out of order. Similarly, if the entire text in some images is not read from top to bottom or left to right, the resulting spliced ​​text may also be incorrect. Therefore, existing methods for text splicing generally have low accuracy. Summary of the Invention

[0004] This invention provides a text splicing method, apparatus, electronic device, and storage medium to address the shortcomings of low splicing accuracy in existing technologies and to improve the accuracy of text splicing.

[0005] This invention provides a text concatenation method, comprising:

[0006] Identify the text blocks in the target image;

[0007] For each text block in the target image, a number marker is determined for each text block, and the number marker is marked at a preset position of the corresponding text block to obtain the marked target image;

[0008] The labeled target image is input into the numbering sorting model to obtain the numbering label sorting output by the numbering sorting model; the numbering sorting model is obtained by training an initial numbering sorting model based on sample images and corresponding sorting labels, the sample image is an image after adding sample number labels to sample text in the sample image, and the sorting label is a label obtained by sorting the sample number labels based on the order of the sample text;

[0009] The text blocks are concatenated based on the numbering markers to obtain the concatenated text.

[0010] According to a text concatenation method provided by the present invention, determining the numbering marker of the text block includes:

[0011] Based on the position information of the text block in the target image, determine the height of the text block in the target image;

[0012] Determine the height of the numbered marker that matches the height of the text block;

[0013] The numbering marker is randomly generated based on its height.

[0014] According to a text concatenation method provided by the present invention, determining the numbering marker of the text block includes:

[0015] Randomly generate the labels corresponding to the text blocks;

[0016] The label and at least one character in the text block are identified as the numbering marker of the text block.

[0017] According to a text concatenation method provided by the present invention, determining the label and at least one character in the text block as the numbering marker of the text block includes:

[0018] If at least one character includes all characters in the text block, the selected text block is obtained by surrounding the text block with the smallest bounding box.

[0019] The label and the selected text block are designated as the numbering markers for the text block.

[0020] According to a text concatenation method provided by the present invention, determining the numbering marker of the text block includes:

[0021] Based on the pixel values ​​of each pixel in the target image, determine the pixel color of each pixel;

[0022] The color of the pixel with the most pixels is determined as the reference color of the target image;

[0023] The target color is determined based on the reference color, and the color difference between the target color and the reference color is greater than a preset color difference;

[0024] The text block is numbered based on the target color.

[0025] According to a text concatenation method provided by the present invention, determining the numbering marker of the text block includes:

[0026] Randomly generate the labels corresponding to the text blocks;

[0027] Extract the background image at the location of the label;

[0028] The background image and the label are used as the numbering markers for the text block.

[0029] According to a text concatenation method provided by the present invention, the method further includes:

[0030] Receive a text concatenation request, wherein the text concatenation request includes information to be extracted;

[0031] Based on the text concatenation request, target information corresponding to the information to be extracted is extracted from the concatenated text.

[0032] The present invention also provides a text splicing device, comprising:

[0033] The first determining module is used to determine each text block in the target image;

[0034] The second determining module is used to determine the number of each text block in the target image, and mark the number at a preset position of the corresponding text block to obtain the marked target image;

[0035] The processing module is used to input the labeled target image into the numbering sorting model to obtain the numbering label sorting output by the numbering sorting model; the numbering sorting model is obtained by training an initial numbering sorting model based on sample images and corresponding sorting labels, the sample image is an image after adding sample number labels to sample text in the sample image, and the sorting label is a label obtained by sorting the sample number labels based on the order of the sample text;

[0036] The splicing module is used to splice the text blocks according to the sorting of the numbered tags to obtain spliced ​​text.

[0037] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the text splicing methods described above.

[0038] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the text splicing method as described above.

[0039] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the text splicing methods described above.

[0040] This invention provides a text splicing method, apparatus, electronic device, and storage medium. The method determines the text blocks to be spliced ​​by identifying them in a target image. For each text block in the target image, a number is assigned, and these numbers are marked at preset positions to obtain a labeled target image. This allows for targeted enhancement of image information for each text block, making them more visually distinctive and facilitating subsequent computer vision processing for identification and feature extraction. The labeled target image is then input into a numbering and sorting model to obtain a numbering and sorting output from the model. This model can then be used to perform calculations on the labeled target image. In computer vision processing, this model can extract more representative image features from labeled target images. The numbering and sorting model is trained on an initial numbering and sorting model based on sample images and corresponding sorting labels. The sample images are images with sample text added with sample numbers, and the sorting labels are labels obtained by sorting the sample numbers based on the order of the sample text. Therefore, based on the trained numbering and sorting model and the more representative image features extracted by the model, the correct sorting order of each number mark can be accurately predicted. Based on the number mark sorting, the text blocks corresponding to each number mark can be concatenated in the correct order to obtain the correctly sorted concatenated text, thus improving the accuracy of text concatenation. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0042] Figure 1 This is a flowchart illustrating the text splicing method provided in an embodiment of the present invention;

[0043] Figure 2 This is one of the schematic diagrams of the labeled target image provided in the embodiments of the present invention;

[0044] Figure 3 This is the second schematic diagram of the labeled target image provided in the embodiments of the present invention;

[0045] Figure 4 This is a schematic diagram of the information extraction process provided in an embodiment of the present invention;

[0046] Figure 5 This is a schematic diagram of the text splicing device provided in an embodiment of the present invention;

[0047] Figure 6 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0049] It should be noted that the serial numbers assigned to the objects described in this invention, such as "first" and "second", are only used to distinguish the objects being described and do not have any sequential or technical meaning.

[0050] In various scenarios involving image information applications, many business requirements necessitate the recognition and correct stitching of text within images before subsequent image information applications can proceed. Therefore, high accuracy in text stitching is a prerequisite for the effective implementation of these applications.

[0051] Existing text concatenation methods primarily rely on pre-defined rules. Therefore, the effectiveness of these rules largely determines the accuracy of the text concatenation. These pre-defined rules can be algorithms designed based on prior knowledge. Regardless of their simplicity or complexity, they are fixed rules based on prior knowledge. In practical applications, different scenarios handle images with varying sorting formats, and the sorting of text content varies greatly. Using fixed pre-defined rules for text sorting and concatenation often results in a certain amount of incorrectly concatenated text, leading to low accuracy.

[0052] To address the aforementioned problems, this invention provides a text concatenation method. The method involves: identifying text blocks in a target image; assigning a number to each text block and marking it at a preset position to obtain a marked target image; inputting the marked target image into a numbering sorting model to obtain a sorted numbering label output by the model; the numbering sorting model is trained on an initial numbering sorting model based on sample images and corresponding sorting labels; the sample images are images with sample text marked with sample numbers, and the sorting labels are labels obtained by sorting the sample number marks based on the order of the sample text; and concatenating the text blocks based on the numbering label sort to obtain the concatenated text. This method abandons the idea of ​​pre-defined rule-based text concatenation, implementing the text concatenation task through computer vision processing. With the help of the trained numbering sorting model, it can perform highly accurate text concatenation on various types of target images. Training the initial numbering sorting model based on sample images and corresponding sorting labels allows the model to accumulate sorting experience with number tags during the learning process. When faced with various types of target images, the model can sort the number tags in the target image with a high accuracy based on the accumulated sorting experience, thereby outputting a highly accurate number tag sort and further obtaining accurate spliced ​​text.

[0053] Meanwhile, the method of this invention incorporates image information enhancement concepts in both the model training and model application stages. During model training, sample number markers are added to the sample text in the sample images to enhance the information of the sample text. This can be understood as the sample number markers enhancing and highlighting the information of each sample text, making it easier for the initial numbering and sorting model to extract image features with stronger representational power during learning. This reduces the difficulty of model learning while improving the model's ability to accurately sort. Correspondingly, during model application, after determining each text block in the target image, number markers are marked at preset positions of each text block, resulting in an image with enhanced image information, i.e., the marked target image. Based on this marked target image and the trained numbering and sorting model, the number marker order can be predicted relatively accurately. Based on the correct number marker order, the corresponding text blocks can be sequentially concatenated to obtain concatenated text with high accuracy. The following is combined with... Figures 1 to 4 The text splicing method provided in the embodiments of the present invention will be described.

[0054] Figure 1This is a flowchart illustrating the text splicing method provided in this embodiment of the invention. The text splicing method provided in this embodiment is applicable to any scenario where text is spliced ​​in an image. The executing entity of this method can be an electronic device such as a smartphone, computer, server, server cluster, or a specially designed text splicing device, or a text splicing device installed in such an electronic device. This text splicing device can be implemented through software, hardware, or a combination of both. Figure 1 As shown, the text splicing method includes steps 110 to 140.

[0055] Step 110: Identify the text blocks in the target image.

[0056] Specifically, the target image is the image from which text splicing is to be performed. Any image-text recognition method can be used to determine the text blocks within the target image. For example, text blocks can be determined based on an image recognition model or an image-text conversion algorithm. A text block can be a single character or a set of two or more characters that are close in position.

[0057] For example, using an Optical Character Recognition (OCR) engine to perform text recognition on a target image can identify individual text blocks within the image. OCR engines can be, for example, open-source tools like PaddleOCR and EasyOCR.

[0058] Step 120: For each text block in the target image, determine the number of the text block and mark the number at the preset position of the corresponding text block to obtain the marked target image.

[0059] Specifically, a numbering mark can be understood as a mark that serves as a unique identifier, such as a mark that includes at least one of the following: a number, a word, a letter, or an identifier. The numbering mark for a text block can be determined, for example, by randomly or sequentially generating the numbering mark corresponding to each text block using a mark generation algorithm; or it can be generated based on numbering mark criteria, etc.

[0060] The preset position of a text block can be any position around the text block, such as the left, top, right, or bottom of the text block, etc., and this embodiment of the invention does not limit this. After determining and marking the number of each text block in the target image, the marked target image is obtained.

[0061] Figure 2 This is one of the schematic diagrams of the labeled target image provided in the embodiments of the present invention, such as... Figure 2As shown, after identifying each text block in the target image, a numbered label can be determined and placed on the left side of the corresponding text block.

[0062] For example, generating numbered markers sequentially using a marker generation algorithm could involve using an OCR engine to determine each text block in the target image, assigning a numerical number to each text block output by the OCR engine, using this numerical number as the corresponding text block's marker, and then marking this marker at a preset position in the target image corresponding to the text block. This method can yield results such as... Figure 2 The labeled target image is shown.

[0063] For example, the random generation of numbered tags by the tag generation algorithm can be achieved by setting up a tag generation program, which randomly generates a tag that serves as a unique identifier for each text block, such as a random number or symbol, and then determining each tag as the numbered tag for each text block.

[0064] For example, generating numbering tags based on numbering criteria could involve setting numbering criteria and generating numbering tags corresponding to each text block based on those criteria. For instance, the numbering criteria could involve sequentially numbering the text blocks according to their row and column numbers, and then determining the sequentially numbered text blocks as their numbering tags.

[0065] Step 130: Input the labeled target image into the number sorting model to obtain the number label sorting output by the number sorting model; the number sorting model is obtained by training the initial number sorting model based on the sample image and the corresponding sorting label. The sample image is the image after adding sample number labels to the sample text in the sample image, and the sorting label is the label obtained by sorting the sample number labels based on the order of the sample text.

[0066] Specifically, the numbering and labeling sorting can be a sequence obtained by sorting the numbering and labels of each text block in the target image. This sequence corresponds to the correct sorting sequence of each text block in the target image, or it can be understood as the correct reading order of each text block in the target image.

[0067] The initial numbering and sorting model can be an initial neural network model, which can include at least one of the following neural networks: Convolutional Neural Network (CNN), Recurrent Neural Networks (RNN), Long Short-Term Memory (LSTM) neural network, and Deep Neural Networks (DNN). Based on the initial numbering and sorting model, sample images, and the corresponding sorting labels for the sample images, a numbering and sorting model can be trained.

[0068] For example, the numbering and sorting model can be a neural network model obtained after supervised training of an initial numbering and sorting model based on sample images and corresponding sorting labels. For images containing text, each text block is identified, and a corresponding number label is added to each text block to obtain the sample image. Each text block is the sample text, and the added number label is the sample number label. Sorting the number labels according to the correct sorting order yields the ground truth labels for training, i.e., the sorting labels corresponding to the sample images. It should be understood that using the same method for determining and labeling number labels during both the model training and application phases can improve the accuracy of the numbering label sorting output by the numbering and sorting model during application. If different methods for determining and labeling number labels are used during the model training and application phases, it increases the difficulty of model prediction and affects the accuracy of the numbering label sorting output by the numbering and sorting model.

[0069] By using a numbered sorting model to output numbered labels, the difficulty and complexity of directly concatenating text blocks in a target image can be reduced. During training, the model can be trained on sample images of various layouts to improve its predictive ability and enhance its stability and robustness in application. Furthermore, predicting numbered labels based on labeled target images eliminates the need for pre-defined text concatenation rules and layout analysis, reducing the additional risk of accuracy degradation associated with these steps. Numbered label prediction based on image features facilitates model training and convergence, improving the efficiency of training models suitable for specific application scenarios.

[0070] Step 140: Sorting each text block based on its numbering markers and then concatenating them to obtain the concatenated text.

[0071] Specifically, after obtaining the numbered sorting model output by the numbered sorting model, each text block can be retrieved based on the correspondence between the numbered sorting model and the text block, and sorted according to the order of the numbered sorting model to obtain a highly accurate spliced ​​text.

[0072] The text concatenation method provided in this invention determines the text blocks to be concatenated in a target image. For each text block in the target image, a number is assigned and marked at a preset position to obtain a marked target image. This method selectively adds image information to each text block, making each text block more visually distinctive, which is beneficial for subsequent computer vision processing for identification and feature extraction. The marked target image is then input into a numbering and sorting model to obtain a numbering and sorting output from the model. This model can then be used to perform computer vision processing on the marked target image. This model can extract more representative image features from the labeled target image. The numbering and sorting model is trained on the initial numbering and sorting model based on sample images and corresponding sorting labels. The sample images are images with sample numbers added to the sample text in the sample images, and the sorting labels are labels obtained by sorting the sample numbers based on the order of the sample text. Therefore, based on the trained numbering and sorting model and the more representative image features extracted by the model, the correct sorting order of each number mark can be accurately predicted. Based on the number mark sorting, the text blocks corresponding to each number mark can be concatenated in the correct order to obtain the correctly sorted concatenated text, which improves the accuracy of text concatenation.

[0073] For example, in order to enable the model to more accurately identify and extract image features, the font height of the numbering markers can be made to match the height of the text block when determining the numbering markers. This can reduce the difficulty of model inference and prediction, thereby improving the accuracy of determining the order of the numbering markers.

[0074] In one embodiment, determining the numbering marker of a text block can be achieved in the following way: determining the height of the text block in the target image based on the position information of the text block in the target image; determining the height of the numbering marker that matches the height of the text block; and randomly generating a numbering marker based on the height of the numbering marker.

[0075] For example, the positional information of the text block in the target image can be information that characterizes the position of the text block. For instance, the positional information may include the coordinates of each vertex of the text block, the coordinates of the center point of the text block, and the length and height of the text block.

[0076] When determining text blocks based on image recognition models, a task to determine the location information of text blocks can be added to the image recognition model. This allows the location information of each text block in the target image to be obtained while recognizing the content of each text block. Alternatively, when performing image-to-text conversion based on an OCR engine, the location information of the text blocks in the target image can be obtained through the OCR engine.

[0077] To determine the height of a text block in a target image based on its location, the height can be calculated by subtracting the coordinates of its vertices. Alternatively, if the location information includes the height, the height can be directly obtained, thus determining the height of the text block in the target image.

[0078] Determine the height of the numbered marker that matches the height of the text block. This can be done by setting the height of the numbered marker to be the same as the height of the text block, or by setting the font size of the numbered marker to be close to the height of the text block, thereby enabling adaptive adjustment of the font size or height of the numbered marker based on the height of the text block.

[0079] Randomly generating number markers based on their height can be understood as follows: after determining the height of the number markers that match the height of the text blocks, randomly generating random numbers that fit the height of each text block, such as random numerical numbers. It's worth noting that during the initial training phase of the number sorting model, if the text blocks in the image are labeled according to the mathematical order of the numbers themselves (from top to bottom and left to right), the trained number sorting model may output an overfitted number sort. This can be understood as follows: when labeling sample text in a sample image based on the mathematical order of the numbers, there will be a pattern in the mathematical order of the number markers. When training the initial number sorting model with a large number of similar sample images, the model may prematurely converge, causing the trained number sorting model to sort the number markers according to the mathematical order of the numbers, resulting in an overfitted number sort and thus reduced accuracy. To address this phenomenon, randomly generated number markers can overcome the overfitting issue in number sorting. Using the same random generation method to determine the numbering labels for each text block during both model training and application can improve the model's predictive ability and increase the accuracy of the output numbering label sorting.

[0080] During both the training and application phases of the model, either adaptive height-based numbering or random number generation can be used to determine the numbering of each text block. It should be noted that within the same target image or sample image, each randomly generated number remains a unique identifier.

[0081] In this embodiment, the height of the text block in the target image is determined based on its position information; the height of the corresponding number marker is determined; and a number marker is randomly generated based on the height of the number marker. On one hand, by matching the height of the text block with the number marker, it avoids situations where a taller text block corresponds to a shorter number marker, or vice versa, thereby reducing the gap between the height of the text block and the height of the corresponding number marker and preventing ambiguity in the correspondence between the number marker and the text block, which increases the difficulty of model prediction. On the other hand, generating number markers corresponding to each text block based on random number generation improves the model's predictive ability, resulting in a more accurate number marker ranking during model application.

[0082] For example, the text characters in each text block are usually not the same. Therefore, when determining the numbering mark, this feature can be used to increase the difference in image features of each text block, improve the ability of the numbering sorting model to recognize the numbering mark and predict the numbering mark sorting, and further improve the accuracy of the obtained numbering mark sorting.

[0083] In one embodiment, determining the numbering mark of a text block can be achieved by: randomly generating a label corresponding to the text block; and determining the label and at least one character in the text block as the numbering mark of the text block.

[0084] Specifically, by using a random number generation method, a corresponding number can be assigned to each text block. This number can be in numerical form. The number can be marked at a preset position in the corresponding text block. This number, together with at least one character in the corresponding text block, can form special image information. The number and at least one character in the text block are used as the identification mark for the text block.

[0085] Figure 3 This is the second schematic diagram of the labeled target image provided in the embodiments of the present invention, such as... Figure 3As shown, after determining each text block in the target image, a random numbering method is used to generate corresponding numerical labels for each text block, and each numerical label is marked on the left side of the corresponding text block. For example, if the first two characters in the text block corresponding to the numerical label 6 are "listed", then "6 listed" is determined as the number label corresponding to this text block; if the first two characters in the text block corresponding to the numerical label 8 are "quality guarantee", then "8 quality guarantee" is determined as the number label corresponding to this text block; if the first two characters in the text block corresponding to the numerical label 9 are "general", then "9 general" is determined as the number label corresponding to this text block; if the first two characters in the text block corresponding to the numerical label 7 are "English", then "7 English" is determined as the number label corresponding to this text block; if the first two characters in the text block corresponding to the numerical label 2 are "Chinese", then "2 Chinese" is determined as the number label corresponding to this text block. After inputting the labeled target image into the number sorting model, an example of the sorted number labels output by the number sorting model is "6 listed; 8 quality guarantee... 9 general; 7 English; 2 Chinese;...".

[0086] In this embodiment, when determining the number label of the text block, the label corresponding to the text block can be randomly generated; at least one character in the label and the text block is determined as the number label of the text block. Based on this, the labeled number label and some characters in the corresponding text block can be used together as the prediction target of the model, further improving the ability and stability of the number sorting model to predict the sorted number labels.

[0087] In one implementation, in order to further improve the prediction ability of the number sorting model, when at least one character in the label and the text block is determined as the number label of the text block, it can be that when at least one character includes all characters in the text block, the text block is surrounded by the minimum bounding box to obtain the boxed text block; the label and the boxed text block are determined as the number label of the text block.

[0088] Specifically, when determining the number label, a random numbering method can be used to generate corresponding numerical labels for each text block, mark each numerical label on the left side of the corresponding text block, and surround the corresponding text block with the minimum bounding box to obtain the boxed text block.

[0089] As Figure 2 shown, for each text block, the corresponding text block is surrounded by its own minimum bounding box to obtain the boxed text block, and a random numerical label is generated for each text block, and each numerical label is marked on the left side of each text block. The label and the boxed text block are jointly determined as the number label of the text block.

[0090] It should be understood that using this method to determine the sample number labels of the sample images when training the initial numbering and sorting model makes it easier for the model to extract the image features corresponding to each text block during training, thus reducing the difficulty of model training. When applying the trained numbering and sorting model for prediction, determining the number labels corresponding to each text block in the target image using this method allows the numbering and sorting model to extract the image features of each text block in the target image more accurately, thereby obtaining a more accurate number label sorting.

[0091] In this embodiment, when determining at least one character in the label and text block as the numbering marker of the text block, it can be done by enclosing the text block with the smallest bounding box, provided that at least one character includes all characters in the text block; then, the label and the selected text block are determined as the numbering markers of the text block. Based on this, using the label and the entire selected text block as the numbering marker can make the image feature differences between each text block greater. Utilizing the differences in characters within the text block can enhance the unique identification function of the text block, that is, to verify each text block. Therefore, the robustness of the numbering sorting model in outputting the numbering markers can be improved.

[0092] In some application scenarios, the color of an image can affect the recognition of numbered markers. For example, if the background of the target image is black, and the color of the numbered markers is fixed to black, the numbered markers on the target image may not be accurately recognized. In this case, the numbered markers cannot enhance the image information of each text block. To avoid this situation, the color of the numbered markers can be adaptively determined to mitigate this risk.

[0093] In one embodiment, determining the numbering mark of the text block can specifically involve: determining the pixel color of each pixel based on the pixel value of each pixel in the target image; determining the pixel color of the pixel with the most pixels as the reference color of the target image; determining the target color based on the reference color, wherein the color difference between the target color and the reference color is greater than a preset color difference; and determining the numbering mark of the text block based on the target color.

[0094] Specifically, the pixel values ​​of each pixel in the target image can be, for example, the RGB values ​​of the pixel values, namely the values ​​of the R channel, G channel, and B channel; or the grayscale values ​​of the pixel values, etc.

[0095] For example, while determining each text block in the target image, the pixel value of each pixel in the target image can be analyzed and determined by an image pixel value analysis algorithm to obtain the pixel value of each pixel; if the input target image includes pixel value information, the pixel value of each pixel can also be obtained directly.

[0096] By analyzing the pixel values ​​of each pixel, the color of each pixel can be determined. For example, based on the value range of each RGB channel, the pixel colors can be pre-divided into a predetermined number of color categories, such as black, white, red, yellow, blue, green, and gray. By analyzing and comparing the pixel values ​​of each pixel in the target image with the predetermined RGB value ranges for each color category, the pixel color of each pixel can be determined. The number of pixels for each color is then counted, and the pixel color corresponding to the maximum number of pixels is determined as the reference color of the target image.

[0097] Further, a target color is determined based on a reference color, where the color difference between the target color and the reference color is greater than a preset color difference. This step aims to identify a color with a significant contrast to the target image color as the target color, used for assigning a number to it. Specifically, after determining the reference color, at least one target color can be determined based on the reference color and the preset color difference. The preset color difference can be a set of preset color difference values; for example, the color difference values ​​between preset black, white, red, yellow, blue, green, and gray pixels can be calculated, and the average value is calculated based on these values. This average color difference value is then determined as the preset color difference.

[0098] For example, if the reference color of the target image is determined to be black, and the color difference between black and white is greater than a preset color difference, then white can be determined as the target color. When determining the numbering markers of text blocks, the color of each numbering marker is set to white, and the white numbering markers are marked in the target image to facilitate the identification of the numbering markers in the target image and the prediction of the numbering marker order.

[0099] In this embodiment, when determining the numbering markers of text blocks, the pixel color of each pixel is determined based on its pixel value in the target image; the pixel color with the largest number of pixels is determined as the reference color of the target image; the target color is determined based on the reference color, and the color difference between the target color and the reference color is greater than a preset color difference; the numbering markers of the text blocks are then determined based on the target color. Therefore, by adaptively determining the color of the numbering markers, the color of the numbering markers can be avoided from being the same as or similar to the color of the target image, thus reducing the prediction difficulty during model training and application, and improving the accuracy of the output numbering marker sorting.

[0100] For example, when determining the numbering markers, the font of the numbering markers can be adaptively determined to make the numbering markers clearer and more obvious in the target image, which is beneficial to the recognition and prediction of the numbering sorting model.

[0101] For example, while identifying text blocks in a target image, a font analysis algorithm can be used to determine the font of each text block, and the font of the text block can be designated as the target font. When determining the numbered markers, the target font is set to the font of the numbered markers. Alternatively, while identifying text blocks in the target image, the character type of each text block can be determined, such as detecting whether the text in the text block contains Chinese characters, English characters, or German characters. Based on the preset correspondence between character type and font, the target font corresponding to the text block can be determined, and the target font can be designated as the font of the numbered markers. After determining the target font, the numbered markers of the target font can be marked at preset positions in each text block of the target image.

[0102] In this embodiment, when determining the numbering mark, an adaptive font for the numbering mark can be determined to achieve the purpose of clearly and obviously marking the numbering mark, which is beneficial to computer vision processing, thereby reducing the difficulty of the numbering sorting model to make predictions and improving the accuracy of predictions.

[0103] In practical applications, in order to further enhance the unique identification function of the numbering mark and improve its visibility, when determining the numbering mark, the background image at the location of the mark can be combined, and both the background image and the mark can be determined as the numbering mark of the text block.

[0104] In one embodiment, determining the numbering mark of a text block can specifically involve: randomly generating a label corresponding to the text block; extracting the background image at the location of the label; and using the background image and the label as the numbering mark of the text block.

[0105] Specifically, when determining the numbering markers, a random number generation method can be used to generate corresponding numerical markers for each text block. Each numerical marker is then placed at a preset position within its corresponding text block. The background image at the marker's location is extracted; this background image can be an image of the target image itself at that location. For example, if the target image includes a background pattern, the background image is extracted at the numerical marker's location, using the center point of the numerical marker and a preset area. The numerical marker containing the background image is then used as the numbering marker for the text block.

[0106] In this embodiment, by randomly generating labels corresponding to text blocks, extracting the background image at the location of the label, and determining the background image and label as the numbering mark of the text block, the uniqueness and distinctiveness of the numbering mark can be further enhanced. This method can be used in both model training and model inference stages to enhance the image features of each text block, thereby improving the model's prediction ability and obtaining a more accurate numbering mark order.

[0107] It should be understood that the technical features in the above embodiments can be combined and applied arbitrarily without conflict, in order to improve the accuracy of text splicing.

[0108] In practical applications, many business scenarios require structured extraction of key text information from target images. Structured extraction can be understood as extracting the necessary information within a specific business context. For example, in business scenarios such as financial reimbursement, document recognition, intelligent claims processing, or medical record extraction, it is necessary to selectively extract relevant information from images such as financial statements, documents, insurance policies, or medical records. For plain text images and rich document images that include various information formats such as text, graphics, or tables, the following two types of methods can be used for structured information extraction.

[0109] The first type of method mainly includes two steps. The first step is to detect and recognize the text in the target image using a text recognition tool to obtain all the text contained in the target image. The second step is to use a text information extraction model to perform model prediction processing on all the text obtained in the first step in order to obtain the target text information that needs to be extracted.

[0110] In this type of approach, the first step is typically text recognition based on an OCR model. The OCR model and the information extraction model are independent and decoupled. The general OCR model's recognition is based on text blocks; that is, after detecting and recognizing each text block in the target image separately, the results of the text blocks need to be correctly concatenated to obtain a passage that conforms to the actual reading order. Errors in the concatenation process introduce a large amount of text noise into the second-stage information extraction, posing a significant challenge to the robustness of the subsequent extraction model.

[0111] Regarding the text concatenation order, these methods often employ pre-defined rules based on text position information, or incorporate layout analysis models that combine text position and content features to achieve concatenation. Using pre-defined rules to concatenate text blocks faces the challenge of being unsuitable for diverse layouts; while layout analysis models are susceptible to image feature interference, require significant annotation work, and further need to be matched with OCR text blocks, undoubtedly introducing new errors and noise. Therefore, these methods generally result in a high error rate in the extracted information.

[0112] The second type of method is based on models that can directly extract information from images. The target image is input into the model, and the model outputs the target text information. While this method bypasses the steps of text recognition and text concatenation, its accuracy depends entirely on the model's image recognition and prediction capabilities. In cases where the layout is relatively fixed and the target field is short, such as ID cards or train tickets, the model can easily fit a template for extraction. However, this approach has poor scalability for scenarios with variable layouts and long field content, and it struggles to achieve high extraction accuracy in open-format scenarios.

[0113] Based on the above problems, the text splicing method provided in this embodiment of the invention can solve the problem of low text splicing accuracy. After improving the accuracy of text splicing, the accuracy of information extraction can be improved based on the more accurate spliced ​​text.

[0114] In one embodiment, the text splicing method further includes: receiving a text splicing request, the text splicing request including information to be extracted; and extracting target information corresponding to the information to be extracted from the spliced ​​text based on the text splicing request.

[0115] Specifically, a concatenation request can be any form of request that triggers the start of text concatenation, and this text concatenation request includes the information to be extracted. The information to be extracted can be understood as the information that needs to be extracted during structured extraction.

[0116] For example, given multiple target images for insurance clauses, the text information corresponding to the "Contract Formation and Effectiveness" clause in each target image is extracted; this text information is the target information. Similarly, given multiple target images for drug instruction manuals, the text information corresponding to the "Specifications" section in each target image is extracted; this text information is the target information. Figure 3 In this context, "200ml" represents the target information to be extracted.

[0117] By using the text splicing methods provided in the above embodiments to splice text blocks in a target image, highly accurate spliced ​​text can be obtained. Based on the text splicing request, target information corresponding to the information to be extracted can be extracted from the spliced ​​text.

[0118] For example, based on the task requirements of the application scenario, an initial information extraction model is trained to obtain an information extraction model capable of extracting target information from concatenated text. Inputting the target image into a numbering and sorting model yields a numbered list output by the model. Based on this numbered list, the corresponding text blocks are retrieved, allowing the text blocks to be arranged sequentially according to their numbered lists, resulting in the concatenated text. Inputting this concatenated text into the trained information extraction model enables the extraction of the desired target information.

[0119] In this embodiment, a text concatenation request is received, which includes information to be extracted. Based on the text concatenation request, target information corresponding to the information to be extracted can be extracted from the concatenated text. Therefore, by marking the target image with numbers, the step of sequentially concatenating text blocks to obtain the complete concatenated text is transformed into a two-step task: first predicting the correct text block concatenation order, and then performing the concatenation. Although the numbering increases the number of steps, the accuracy of the sorting is improved by using numbering markers, which in turn improves the correctness of the concatenated text, thus ensuring the accuracy of target information extraction. The method provided in this embodiment is easy to learn and is not affected by document format or content length, effectively improving the accuracy of information extraction.

[0120] Below, in conjunction with Figure 4 The process of extracting the above information will be described in detail through a specific embodiment. Figure 4 This is a schematic diagram of the information extraction process provided in an embodiment of the present invention, such as... Figure 4 As shown, information extraction may include steps 410 to 440.

[0121] Step 410, Image preprocessing.

[0122] Specifically, the target image is preprocessed. This preprocessing may include orientation correction and noise reduction of the target image.

[0123] For example, the orientation of the target image can be detected, and if the orientation of the target image does not meet the information extraction requirements, it can be corrected to ensure that the orientation of the target image or the text within the target image is correct. Another example is the removal of interfering factors such as watermarks, creases, blurred areas, or occlusions in the target image to achieve a noise reduction effect.

[0124] Step 420, text detection and recognition.

[0125] Specifically, text blocks in the target image are detected and recognized to determine the text and symbols in each text block; the position information of the text block in the target image is determined, such as the coordinates of the center point of the text block, the width and height of the text block, and the coordinates of each vertex of the smallest bounding box surrounding the text block.

[0126] Step 430: Sort and concatenate text blocks.

[0127] Specifically, the text blocks in the target image can be spliced ​​together using any of the text splicing methods in the above embodiments to obtain spliced ​​text.

[0128] Step 440, Information Extraction.

[0129] Specifically, after obtaining a relatively accurate concatenated text, various information extraction methods can be applied to extract the target information corresponding to the information to be extracted from the concatenated text. For example, the target information can be extracted through an information extraction model.

[0130] For example, when the target image is a paper or other image with many pages, variable layout, and a large number of words, the above steps can quickly and accurately extract the target information. The number of numbering tags required by the solution in this embodiment is only related to the number of text blocks, and only the numbering tags of each text block need to be marked, without marking all the text. When the target image is an image with a relatively fixed format, such as an invoice, this solution can still focus on key target fields such as amount, company, tax number, etc., and extract the target information corresponding to the key target fields in a structured manner, which has strong applicability.

[0131] In this embodiment, through image preprocessing, text detection and recognition, text block sorting and concatenation, and information extraction, highly accurate structured information extraction of target information from the target image can be achieved. Furthermore, because the labeling process is simple, the task complexity and the cost of preparing training data are low. It is unaffected by the amount of text in the sample or target image, avoids the additional risks associated with layout analysis, and is more conducive to model training and convergence, thus improving the overall performance in document image information extraction tasks.

[0132] The text splicing device provided in the embodiments of the present invention will be described below. The text splicing device described below can be referred to in correspondence with the text splicing method described above.

[0133] Figure 5 This is a structural schematic diagram of the text splicing device provided in an embodiment of the present invention, with reference to... Figure 5 As shown, the text splicing device 500 includes:

[0134] The first determining module 510 is used to determine each text block in the target image;

[0135] The second determining module 520 is used to determine the numbering mark of each text block in the target image, and mark the numbering mark at the preset position of the corresponding text block to obtain the marked target image;

[0136] The processing module 530 is used to input the labeled target image into the numbering sorting model to obtain the numbering label sorting output by the numbering sorting model. The numbering sorting model is obtained by training the initial numbering sorting model based on the sample image and the corresponding sorting label. The sample image is the image after adding sample number labels to the sample text in the sample image. The sorting label is the label obtained by sorting the sample number labels based on the order of the sample text.

[0137] The splicing module 540 is used to splice text blocks based on numbered tags to obtain spliced ​​text.

[0138] In one example embodiment, the second determining module 520 is specifically used for:

[0139] Based on the position information of the text block in the target image, determine the height of the text block in the target image;

[0140] Determine the height of the numbered marker that matches the height of the text block;

[0141] Numbering markers are randomly generated based on their height.

[0142] In one example embodiment, the second determining module 520 is specifically used for:

[0143] Randomly generate labels corresponding to text blocks;

[0144] The label and at least one character in the text block are identified as the numbering marker for the text block.

[0145] In one example embodiment, the second determining module 520 is specifically used for:

[0146] If at least one character includes all characters in the text block, the selected text block is obtained by enclosing the text block with the smallest bounding box;

[0147] Define the labels and selected text blocks as numbered markers for the text blocks.

[0148] In one example embodiment, the second determining module 520 is specifically used for:

[0149] Determine the pixel color of each pixel based on the pixel value of each pixel in the target image;

[0150] The color of the pixel with the most pixels is determined as the reference color of the target image;

[0151] The target color is determined based on the reference color, and the color difference between the target color and the reference color is greater than the preset color difference;

[0152] The text block number is determined based on the target color.

[0153] In one example embodiment, the second determining module 520 is specifically used for:

[0154] Randomly generate labels corresponding to text blocks;

[0155] Extract the background image at the location of the label;

[0156] Use the background image and labels to identify the numbering markers for the text blocks.

[0157] In one example embodiment, the text splicing device 500 further includes a receiving module and an extraction module;

[0158] The receiving module is used to receive text concatenation requests, which include the information to be extracted.

[0159] The extraction module is used to extract the target information corresponding to the information to be extracted from the concatenated text based on the text concatenation request.

[0160] The apparatus of this embodiment can be used to execute the method of any embodiment in the text splicing method side embodiment. Its specific implementation process and technical effects are similar to those in the text splicing method side embodiment. For details, please refer to the detailed description in the text splicing method side embodiment, which will not be repeated here.

[0161] Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiment of the present invention, such as... Figure 6 As shown, the electronic device may include a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a text splicing method, which includes: determining each text block in the target image; determining the numbering mark of each text block in the target image and marking the numbering mark at a preset position of the corresponding text block to obtain a marked target image; inputting the marked target image into a numbering sorting model to obtain the numbering mark sorting output by the numbering sorting model; the numbering sorting model is obtained by training an initial numbering sorting model based on sample images and corresponding sorting labels, where the sample image is an image with sample text added with sample numbering marks, and the sorting label is a label obtained by sorting the sample numbering marks based on the order of the sample text; splicing each text block based on the numbering mark sorting to obtain spliced ​​text.

[0162] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0163] On the other hand, embodiments of the present invention also provide a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the text splicing method provided by the above methods. The method includes: determining each text block in a target image; determining a number marker for each text block in the target image and marking the number marker at a preset position of the corresponding text block to obtain a marked target image; inputting the marked target image into a numbering sorting model to obtain a numbering marker sorting output by the numbering sorting model; the numbering sorting model is obtained by training an initial numbering sorting model based on sample images and corresponding sorting labels, the sample images are images after adding sample number markers to sample text in the sample images, and the sorting labels are labels obtained by sorting the sample number markers based on the order of the sample text; splicing each text block based on the numbering marker sorting to obtain spliced ​​text.

[0164] In another aspect, embodiments of the present invention also provide a computer program product, the computer program product including a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the text splicing method provided by the above methods. The method includes: determining each text block in a target image; determining a number marker for each text block in the target image and marking the number marker at a preset position of the corresponding text block to obtain a marked target image; inputting the marked target image into a numbering sorting model to obtain a numbering marker sorting output by the numbering sorting model; the numbering sorting model is obtained by training an initial numbering sorting model based on sample images and corresponding sorting labels, the sample image is an image after adding sample number markers to sample text in the sample image, and the sorting label is a label obtained by sorting the sample number markers based on the order of the sample text; splicing each text block based on the number marker sorting to obtain spliced ​​text.

[0165] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0166] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0167] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A text concatenation method, characterized in that, include: Identify the text blocks in the target image; For each text block in the target image, a number marker is determined for each text block, and the number marker is marked at a preset position for the corresponding text block to obtain the marked target image; the number markers for each text block are different. The labeled target image is input into the numbering sorting model to obtain the numbering label sorting output by the numbering sorting model; the numbering sorting model is obtained by training an initial numbering sorting model based on sample images and corresponding sorting labels, the sample image is an image after adding sample number labels to sample text in the sample image, and the sorting label is a label obtained by sorting the sample number labels based on the order of the sample text; The text blocks are concatenated based on the numbering markers to obtain the concatenated text.

2. The text splicing method according to claim 1, characterized in that, The determination of the numbering marker for the text block includes: Based on the position information of the text block in the target image, determine the height of the text block in the target image; Determine the height of the numbered marker that matches the height of the text block; The numbering marker is randomly generated based on its height.

3. The text splicing method according to claim 1, characterized in that, The determination of the numbering marker for the text block includes: Randomly generate the labels corresponding to the text blocks; The label and at least one character in the text block are identified as the numbering marker of the text block.

4. The text splicing method according to claim 3, characterized in that, The step of determining the label and at least one character in the text block as the numbering marker of the text block includes: If at least one character includes all characters in the text block, the selected text block is obtained by surrounding the text block with the smallest bounding box. The label and the selected text block are designated as the numbering markers for the text block.

5. The text splicing method according to claim 1, characterized in that, The determination of the numbering marker for the text block includes: Based on the pixel values ​​of each pixel in the target image, determine the pixel color of each pixel; The color of the pixel with the most pixels is determined as the reference color of the target image; The target color is determined based on the reference color, and the color difference between the target color and the reference color is greater than a preset color difference; The text block is numbered based on the target color.

6. The text splicing method according to claim 1, characterized in that, The determination of the numbering marker for the text block includes: Randomly generate the labels corresponding to the text blocks; Extract the background image at the location of the label; The background image and the label are used as the numbering markers for the text block.

7. The text splicing method according to any one of claims 1-6, characterized in that, The method further includes: Receive a text concatenation request, wherein the text concatenation request includes information to be extracted; Based on the text concatenation request, target information corresponding to the information to be extracted is extracted from the concatenated text.

8. A text splicing device, characterized in that, include: The first determining module is used to determine each text block in the target image; The second determining module is used to determine the number of each text block in the target image, and mark the number at a preset position of the corresponding text block to obtain the marked target image; the number of each text block is different. The processing module is used to input the labeled target image into the numbering sorting model to obtain the numbering label sorting output by the numbering sorting model; the numbering sorting model is obtained by training an initial numbering sorting model based on sample images and corresponding sorting labels, the sample image is an image after adding sample number labels to sample text in the sample image, and the sorting label is a label obtained by sorting the sample number labels based on the order of the sample text; The splicing module is used to splice the text blocks according to the sorting of the numbered tags to obtain spliced ​​text.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the text splicing method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the text splicing method as described in any one of claims 1 to 7.

Images