Image text segmentation method, device, electronic device and readable storage medium

Abstract

The present invention relates to the field of artificial intelligence technology, and provides an image text segmentation method, device, electronic device and readable storage medium. The method comprises: obtaining an image to be segmented, performing feature extraction on the image to be segmented, obtaining text edge features, text skeleton features and multi-level visual features; using multi-level visual features to perform feature screening on text edge features and text skeleton features respectively, obtaining target edge features and target skeleton features; performing feature conversion on multi-level visual features, obtaining target visual features; obtaining a target query for guiding image text segmentation, and according to the target query, fusing and decoding the target edge features, the target skeleton features and the target visual features, obtaining a target decoding result; determining the text segmentation result according to the target decoding result and the target visual features. The present invention can perform text segmentation from an image more accurately, and improve the accuracy of text segmentation.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an image text segmentation method, device, electronic device and readable storage medium. Background Art

[0002] The core goal of image text segmentation technology is to accurately extract text from an image, thereby providing a clearer and more accurate basis for subsequent text recognition and understanding. Specifically, image text segmentation technology identifies which areas contain text information by analyzing pixels and features in an image. Current technical means have been able to use text-related supervisory information to improve the accuracy of text segmentation to a certain extent and separate these areas from the background. However, existing methods still have the problem of inaccurate text segmentation. Summary of the invention

[0003] In view of this, embodiments of the present invention provide an image text segmentation method, device, electronic device and readable storage medium to solve the problem of inaccurate text segmentation in the prior art.

[0004] A first aspect of an embodiment of the present invention provides an image text segmentation method, the method comprising:

[0005] Acquire an image to be segmented, perform feature extraction on the image to be segmented, and obtain text edge features, text skeleton features, and multi-level visual features; use the multi-level visual features to perform feature screening on the text edge features and text skeleton features, respectively, to obtain target edge features and target skeleton features; perform feature conversion on the multi-level visual features to obtain target visual features; obtain a target query for guiding image text segmentation, and based on the target query, fuse and decode the target edge features, target skeleton features, and target visual features to obtain a target decoding result; determine the text segmentation result based on the target decoding result and the target visual features.

[0006] A second aspect of an embodiment of the present invention provides an image text segmentation device, the device comprising:

[0007] The feature extraction module is configured to obtain an image to be segmented, perform feature extraction on the image to be segmented, and obtain text edge features, text skeleton features, and multi-level visual features; the feature screening module is configured to use multi-level visual features to perform feature screening on text edge features and text skeleton features respectively, and obtain target edge features and target skeleton features; the feature conversion module is configured to perform feature conversion on multi-level visual features to obtain target visual features; the fusion decoding module is configured to obtain a target query for guiding image text segmentation, and according to the target query, perform fusion decoding on the target edge features, target skeleton features, and target visual features to obtain a target decoding result; the segmentation module is configured to determine the text segmentation result according to the target decoding result and the target visual features.

[0008] According to a third aspect of an embodiment of the present invention, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0009] According to a fourth aspect of an embodiment of the present invention, a readable storage medium is provided, wherein the readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0010] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0011] The image to be segmented is obtained, and features are extracted from the image to be segmented to obtain text edge features, text skeleton features and multi-level visual features. By extracting text edge features and text skeleton features, text edge perception and text skeleton perception can be introduced in text segmentation, and multi-level visual features are used to perform feature screening on text edge features and text skeleton features respectively to enhance text edge perception and text skeleton perception, so that the obtained target edge features and target skeleton features can better reflect the text information in the image. At the same time, the edge and skeleton of the text are focused on during the segmentation process to accurately identify and segment the text area; the multi-level visual features are further converted to obtain target visual features; a target query for guiding image text segmentation is obtained, and according to the target query, the target edge features, target skeleton features and target visual features are fused and decoded to obtain a target decoding result; the text segmentation result is determined according to the target decoding result and the target visual feature, so that the fused decoding result containing multiple guiding features can be used to more accurately perform text segmentation from the image, thereby improving the accuracy of text segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0013] Figure 1 It is a flowchart of an image text segmentation method provided by an embodiment of the present invention;

[0014] Figure 2 It is a structural schematic diagram of an image text segmentation device provided by an embodiment of the present invention;

[0015] Figure 3 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0016] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0017] In the field of image text processing, the image text segmentation task is extremely critical, and its core demand is to accurately segment the text area from the image. This operation is an important cornerstone for subsequent effective text editing and complete text removal, and is directly related to the quality and efficiency of the entire text processing process.

[0018] At present, the existing text segmentation technology has been continuously developing and evolving. The existing methods have improved their performance to a certain extent with the help of various supervisory signals closely related to the text. However, these technologies generally have an obvious shortcoming, that is, they often fail to fully pay attention to the decisive role played by the text edge and the text skeleton in the segmentation process. The text edge can outline the outline boundary of the text area and provide key clues for accurately defining the text range; while the text skeleton reflects the morphological characteristics of the text from the internal structure level, and both are indispensable for achieving high-precision text segmentation. For example, although the traditional edge detection algorithm can distinguish the text edge well, it cannot distinguish between the text area and the non-text area, resulting in the edges detected in the non-text area may interfere with the performance of the text segmentation model. In addition, the existing text segmentation method still has certain limitations when facing the segmentation of the text edge area, especially when the text edge is blurred or the background is complex, the results obtained are even more unsatisfactory. Based on this, the present invention provides an image text segmentation method, which can more accurately perform text segmentation from an image and improve the accuracy of text segmentation.

[0019] An image text segmentation method, device, electronic device and readable storage medium according to an embodiment of the present invention will be described in detail below with reference to the accompanying drawings.

[0020] Figure 1 FIG. 1 is a flow chart of an image text segmentation method provided by an embodiment of the present invention. Figure 1 As shown, the method includes:

[0021] S101, obtaining an image to be segmented, performing feature extraction on the image to be segmented, and obtaining text edge features, text skeleton features, and multi-level visual features;

[0022] S102, using multi-level visual features to perform feature screening on the text edge features and the text skeleton features respectively, to obtain target edge features and target skeleton features;

[0023] S103, performing feature conversion on the multi-level visual features to obtain target visual features;

[0024] S104, obtaining a target query for guiding image text segmentation, and fusing and decoding target edge features, target skeleton features, and target visual features according to the target query to obtain a target decoding result;

[0025] S105, determining a text segmentation result according to the target decoding result and the target visual feature.

[0026] Specifically, the image text segmentation method of this embodiment can be executed by a client or a server, or by a client and a server together. The following is an example of the execution subject being a client. The image to be segmented is the original image input when performing the image text segmentation task. After obtaining the image to be segmented, the image to be segmented can be preprocessed, for example, using Gaussian filtering, median filtering, bilateral filtering, mean filtering and other smoothing techniques to preprocess the image to be segmented to reduce noise. Further, feature extraction is performed on the preprocessed image to be segmented. Specifically, first, edge detection algorithms such as Canny edge detection algorithm, Prewitt edge detection algorithm or Roberts edge detection algorithm can be used to extract features from the image to be segmented to obtain text edge features. Secondly, feature extraction can be performed using text skeleton extraction algorithms such as Hilditch algorithm or Zhang-Suen algorithm to obtain text skeleton features. Then, a visual encoder can be used to extract visual features from the image to be segmented. The visual encoder can use a deep learning model such as ResNet as the backbone network of the visual encoder to extract multi-level visual features in the image to be segmented.

[0027] It can be understood that the text edge feature can clearly depict the boundary between the text and the background. In the image, the edge of the text contains the outline information of the text, such as the starting and ending positions of the strokes, the degree of curvature, etc. This is very important for accurately locating the position of the text in the image. By extracting the text edge features, it can help the subsequent segmentation algorithm to better separate the text from the complex background or other interfering elements. The text skeleton feature reflects the internal structure of the text. It can be regarded as the "central axis" of the text strokes, which is a simplified and abstract representation of the text shape. Extracting the text skeleton feature helps to understand the basic shape and structure of the text. For example, in handwritten text recognition or some text segmentation with large font deformation, the skeleton feature can provide the connectivity and topological structure information of the text strokes. Multi-level visual features contain information extracted from different levels and angles of the image. It can include low-level pixel-level features, such as basic visual information such as color and brightness, and can also include high-level semantic features, such as texture, shape combination and other more abstract information.

[0028] In some examples, multi-level visual features are used to perform feature screening on text edge features and text skeleton features respectively to obtain target edge features and target skeleton features. Specifically, by using multi-level visual features to screen target edge features, false edges in text edge features that may be caused by noise or non-text factors (such as decorative lines in images, pseudo edges generated by image compression, etc.) can be removed, which can provide more reliable boundary information for text segmentation and help improve the accuracy of text segmentation. For target skeleton features, the screening process can eliminate false skeleton branches or redundant information that do not conform to the actual structure of the text. The target skeleton features can better reflect the real internal structure of the text and provide more accurate shape guidance for subsequent text segmentation, thereby improving the quality and accuracy of text segmentation.

[0029] In some examples, feature transformation is performed on multi-level visual features to obtain target visual features; multi-level visual features represent feature information at different levels of the image to be segmented, and these features may differ in format, dimension, etc. Through feature transformation, these features of different formats and dimensions can be unified into a suitable form, which is convenient for subsequent text prediction.

[0030] In some examples, a target query for guiding image text segmentation is obtained, and according to the target query, the target edge features, target skeleton features and target visual features are fused and decoded to obtain a target decoding result; specifically, the target query is a learnable query, which can learn the common positions of texts in different scenarios, the typical features of texts and the association between texts and other elements of the image. Through such a learning process, the learnable query is like having a "knowledge base" for image text segmentation, which can provide accurate guidance for subsequent segmentation operations. Further, under the guidance of the target query, the target edge features, target skeleton features and target visual features are fused and decoded. The target query here is like a "conductor". According to the text rules it has learned, it decides how to fuse the boundary information of the target edge features, the structural information of the target skeleton features and the comprehensive description information of the target visual features, so that the final target decoding result can reflect the real situation and accurate position of the text in the image to the greatest extent, laying the foundation for further obtaining accurate text segmentation results. Finally, according to the target decoding results and the target visual features, a more accurate text segmentation result can be obtained by utilizing the target decoding results and the target visual features. How to obtain the text segmentation result according to the target decoding results and the target visual features will be described in detail in subsequent embodiments and will not be elaborated here.

[0031] According to the technical solution provided by the embodiment of the present invention, an image to be segmented is obtained, and feature extraction is performed on the image to be segmented to obtain text edge features, text skeleton features and multi-level visual features. By extracting text edge features and text skeleton features, text edge perception and text skeleton perception can be introduced in text segmentation, and multi-level visual features are used to perform feature screening on text edge features and text skeleton features respectively to enhance text edge perception and text skeleton perception, so that the obtained target edge features and target skeleton features can better reflect the text information in the image. At the same time, the edge and skeleton of the text are focused on during the segmentation process to accurately identify and segment the text area; the multi-level visual features are further feature converted to obtain target visual features; a target query for guiding image text segmentation is obtained, and according to the target query, the target edge features, target skeleton features and target visual features are fused and decoded to obtain a target decoding result; the text segmentation result is determined according to the target decoding result and the target visual feature, so that the fused decoding result containing multiple guiding features can be used to more accurately perform text segmentation from the image, thereby improving the accuracy of text segmentation.

[0032] In some embodiments, multi-level visual features are used to perform feature screening on text edge features and text skeleton features respectively to obtain target edge features and target skeleton features, including: splicing the multi-level visual features and using the target convolutional network as a text detection head; inputting the spliced ​​multi-level visual features into the target convolutional network, using the target convolutional network to determine the text area of ​​the multi-level visual features to obtain a text area detection frame; and screening the text edge features and text skeleton features respectively according to the text area detection frame to obtain target edge features and target skeleton features.

[0033] Specifically, in the process of using multi-level visual features to perform feature screening on text edge features and text skeleton features respectively, since multi-level visual features refer to features extracted at different levels, including low-level edges and textures to high-level objects and scene information, these multi-level visual features are spliced ​​and fused into a unified feature representation. Then, this feature representation is input into the target convolutional network, and the target convolutional network of this embodiment preferably uses a 1x1 convolutional layer. The 1x1 convolutional layer is a special convolutional layer, and its convolution kernel size is 1x1, so it does not change the spatial size of the input feature. However, the number of feature channels can be changed by adjusting the number of convolution kernels, thereby achieving dimensionality reduction or dimensionality increase of the feature.

[0034] It can be understood that in this embodiment, the 1x1 convolution layer acts as a text detection head, further processing and transforming the spliced ​​features to predict the text area and obtain the text area detection frame (denoted as Mask M). Mask M is a binary matrix of the same size as the input image, where pixels with a value of 1 represent text areas and pixels with a value of 0 represent non-text areas. The 1x1 convolution layer can accurately predict Mask M, thereby achieving the task of text detection.

[0035] It can be understood that the text edge feature is the feature extracted previously to depict the boundary between the text and the background. However, in the process of feature extraction, there may be some false edges caused by noise or non-text factors. Now with the text area detection frame, we can only focus on the part within the detection frame. By filtering the text edge features through the text area detection frame, we can exclude the interference edges that are not related to the text, and only retain the valid edge features that truly belong to the text in the detection frame. In this way, we get the target edge feature, which more accurately represents the true boundary of the text in the text area. Similarly, the text area detection frame also has the same principle for filtering the text skeleton features.

[0036] In this way, multi-level visual features are used to effectively screen text edge features and text skeleton features, and more accurate target edge features and target skeleton features are obtained, providing a more reliable foundation for subsequent tasks such as image text segmentation.

[0037] In some embodiments, the text edge features and the text skeleton features are respectively screened according to the text area detection frame to obtain target edge features and target skeleton features, including: element-by-element multiplication of the text edge features and the text area detection frame to obtain target edge features; element-by-element multiplication of the text skeleton features and the text area detection frame to obtain target skeleton features.

[0038] Specifically, since the text area detection box Mask M is a matrix corresponding to the image size, the element values ​​in the matrix usually have two cases: the position within the text area, the element value is 1 (or other specific values ​​indicating "belonging to the text area"); the position outside the text area, the element value is 0 (or a specific value indicating "not belonging to the text area"). It can be thought of as a "mask" that accurately covers the text area of ​​the image and distinguishes the text area from the non-text area.

[0039] When the text edge feature is multiplied element by element with the text region detection frame Mask M, the "screening" effect of Mask M is actually being used. For each element in the text edge feature, it is multiplied with the element at the corresponding position in Mask M. Since the value of Mask M is 1 (or the corresponding representation value) in the text region and 0 (or the corresponding representation value) outside the text region, after multiplication: the text edge feature elements in the text region remain unchanged because they are multiplied with the elements with a value of 1 (or the corresponding representation value) in Mask M. These retained elements constitute the effective edge features that truly belong to the text in the text region. The text edge feature elements outside the text region are multiplied with the elements with a value of 0 (or the corresponding representation value) in Mask M, and the result is 0, which is equivalent to "filtering" out these edge features that are outside the text region and may be interference. After the above element-by-element multiplication operation, the result obtained is the target edge feature. Compared with the original text edge feature, the target edge feature more accurately includes only the effective edge features of the text in the text area, and removes those false edge features outside the text area that may be caused by background, noise and other factors, thereby providing more accurate text boundary information for subsequent tasks such as text segmentation.

[0040] It can be understood that the operation and principle of multiplying the text skeleton feature by the text area detection box element by element to obtain the target skeleton feature is similar to the operation of obtaining the target edge feature, which will not be elaborated here.

[0041] In some embodiments, according to a target query, target edge features, target skeleton features, and target visual features are fused and decoded to obtain a target decoding result, including: taking the target query as a first parameter, and taking the target edge features, target skeleton features, and target visual features as second parameters; inputting the first parameter and the second parameter into a target decoder, the target decoder including a plurality of consecutively arranged decoding submodules; according to the first parameter, using the first decoding submodule to fusion-decode the second parameter to obtain a first decoding result; using the first decoding result as a new first parameter, and according to the new first parameter, using the second decoding submodule to fusion-decode the second parameter to obtain a second decoding result; repeating the fusion decoding operation of each decoding submodule until the target decoding result output by the last decoding submodule is obtained.

[0042] Specifically, the first parameter (target query) and the second parameter (target edge feature, target skeleton feature, target visual feature) are input to the target decoder together. In this embodiment, the target decoder preferably adopts a decoder of a Transformer structure, and the decoder of the Transformer structure includes a plurality of continuously arranged Transformer decoding submodules, because the self-attention mechanism in the Transformer decoder can effectively process the input long sequence data, and when processing a complex second parameter sequence including target edge features, target skeleton features and target visual features, it can capture the relationship between different positions in the sequence, and secondly, the self-attention mechanism can automatically assign different weights to different features in the second parameter according to the input first parameter. For example, when the target query pays more attention to the skeleton structure of the target, the Transformer decoder can assign a higher weight to the target skeleton feature through the self-attention mechanism, so as to more effectively fuse these features to generate a decoding result. This ability to dynamically adjust feature weights according to task requirements makes the Transformer decoder more flexible and efficient when fusing multiple types of features (such as edge, skeleton and visual features).

[0043] Furthermore, according to the first parameter, the first decoding submodule is used to perform fusion decoding on the second parameter to obtain a first decoding result, that is, the first parameter (target query) and the second parameter are input into the first decoding submodule for fusion decoding to obtain the first decoding result. Then the first decoding result is used as a new first parameter and input into the second decoding submodule together with the second parameter for fusion decoding, and so on, the fusion decoding operation of each decoding submodule is repeatedly performed until the predetermined number of fusion decoding steps is reached to obtain the final target fusion result.

[0044] In this way, the final target decoding result is the final result obtained after all decoding sub-modules in the entire target decoder are processed in sequence and information is continuously integrated, and it can meet the relevant requirements of the target query originally set.

[0045] In some embodiments, each decoding submodule includes a self-attention layer, a pixel-aware attention layer, an edge-aware attention layer, a skeleton-aware attention layer and a feedforward network layer. The fusion decoding operation of each decoding submodule includes: using the self-attention layer to process the first parameter, and using the pixel-aware attention layer to fuse the target visual feature with the processed first parameter to obtain a first fusion feature; using the edge-aware attention layer to fuse the target edge feature with the first fusion feature to obtain a second fusion feature; using the skeleton-aware attention layer to fuse the target skeleton feature with the second fusion feature to obtain a third fusion result; using the feedforward network layer to perform a nonlinear transformation on the third fusion result to obtain a decoding result corresponding to each decoding submodule.

[0046] Specifically, the self-attention layer uses the self-attention mechanism to process the first parameter. The self-attention mechanism can capture the relationship between different parts within the first parameter, and automatically assign different weights according to the relevance of each part. The result of the self-attention layer processing is the processed first parameter. The pixel-aware attention layer focuses on fusing the target visual features with the first parameter processed by the self-attention layer. Through this fusion operation, the first fused feature is obtained. This feature contains both the key information about the target query carried by the first parameter after processing by the self-attention layer, and the visual details provided by the target visual features, so that subsequent processing can be based on a more comprehensive and integrated visual information foundation.

[0047] Furthermore, the edge-aware attention layer then fuses the target edge feature with the first fused feature obtained previously, and after this step of fusion, a second fused feature is generated. The feature at this time further integrates the contour information of the target edge feature and the content contained in the first fused feature, making it richer and more comprehensive, and more conducive to the subsequent accurate understanding and processing of the target.

[0048] Furthermore, the skeleton-aware attention layer is responsible for fusing the target skeleton features with the second fusion features. After completing this fusion, the third fusion result is obtained. This result has processed the first parameter in multiple steps and successively integrated multiple features such as target vision, edge, skeleton, etc. It is the result of comprehensive information from multiple aspects and provides a complete information basis for the final decoding result.

[0049] Furthermore, the feedforward network layer performs a nonlinear transformation on the third fusion result obtained above. The feedforward network layer usually includes multiple neuron layers, and nonlinear transformation is achieved by means of activation functions and the like. The third fusion result can be further processed to explore the hidden relationships and features therein and convert it into a form more suitable as a decoding result. After the nonlinear transformation of the feedforward network layer, the decoding result corresponding to each decoding submodule is finally obtained.

[0050] In some embodiments, the text segmentation result is determined based on the target decoding result and the target visual feature, including: performing pixel prediction on the target decoding result to obtain the predicted classification result of the pixel point; and performing element-by-element multiplication of the predicted classification result of the pixel point with the target visual feature to obtain the segmented text result.

[0051] Specifically, in the process of determining the text segmentation result, the target decoding result is first predicted by pixel point prediction to predict the probability of each pixel point belonging to text or background. Specifically, the target decoding result can be input into a Mask head of a multi-layer perceptron structure to predict the classification result of the pixel point, and then the predicted classification result is multiplied element by element with the target visual feature to strengthen the features of the text area while suppressing the features of the background area. In this way, a more accurate text segmentation result can be obtained. Finally, based on the predicted classification results of the pixel points, the outline of the text area can be constructed, thereby achieving effective segmentation of the text in the image. This not only improves the accuracy of text segmentation, but also shows better robustness when dealing with complex backgrounds and texts of different font sizes.

[0052] In some embodiments, feature conversion is performed on multi-level visual features to obtain target visual features, including: upsampling the multi-level visual features to convert low-resolution visual features in the multi-level visual features into high-resolution visual features; and using the upsampled multi-level visual features as target visual features.

[0053] Specifically, in the process of feature conversion, the multi-level visual features are first upsampled to enhance the resolution of the feature map so that details that may be ignored at lower resolutions can be captured and utilized. Upsampling can be achieved through a variety of methods, such as bilinear interpolation, bicubic interpolation, or through a transposed convolution layer in a convolutional neural network. Through upsampling, low-resolution visual features are converted into high-resolution visual features, while gradually restoring the multi-level visual features to the size of the image to be segmented, which helps the subsequent pixel prediction process to achieve accurate pixel-level prediction, because high-resolution feature maps can provide richer spatial information.

[0054] It should be understood that the order of execution of the steps in the above embodiment does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the process of the embodiment of the present invention.

[0055] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present invention, which will not be described in detail here.

[0056] The following are embodiments of the device of the present invention, which can be used to implement the method embodiments of the present invention. For details not disclosed in the device embodiments of the present invention, please refer to the method embodiments of the present invention.

[0057] Figure 2 FIG. 1 is a schematic diagram of the structure of an image text segmentation device provided by an embodiment of the present invention. Figure 2 As shown, the device comprises:

[0058] The feature extraction module 201 is configured to obtain the image to be segmented, perform feature extraction on the image to be segmented, and obtain text edge features, text skeleton features, and multi-level visual features;

[0059] The feature screening module 202 is configured to use multi-level visual features to screen the text edge features and the text skeleton features respectively, and obtain the target edge features and the target skeleton features;

[0060] A feature conversion module 203 is configured to perform feature conversion on the multi-level visual features to obtain target visual features;

[0061] The fusion decoding module 204 is configured to obtain a target query for guiding image text segmentation, and fuse and decode the target edge features, target skeleton features, and target visual features according to the target query to obtain a target decoding result;

[0062] The segmentation module 205 is configured to determine a text segmentation result according to the target decoding result and the target visual feature.

[0063] In some embodiments, the feature screening module 202 is further configured to splice the multi-level visual features and use the target convolutional network as a text detection head; input the spliced ​​multi-level visual features into the target convolutional network, and use the target convolutional network to determine the text area of ​​the multi-level visual features to obtain a text area detection frame; and screen the text edge features and the text skeleton features according to the text area detection frame to obtain target edge features and target skeleton features.

[0064] In some embodiments, the feature screening module 202 is further configured to perform element-wise multiplication of the text edge feature and the text area detection frame to obtain the target edge feature; and perform element-wise multiplication of the text skeleton feature and the text area detection frame to obtain the target skeleton feature.

[0065] In some embodiments, the fusion decoding module 204 is also configured to use the target query as the first parameter, and the target edge feature, the target skeleton feature and the target visual feature as the second parameter; input the first parameter and the second parameter into the target decoder, and the target decoder includes a plurality of decoding sub-modules arranged in succession; according to the first parameter, use the first decoding sub-module to perform fusion decoding on the second parameter to obtain a first decoding result; use the first decoding result as the new first parameter, and according to the new first parameter, use the second decoding sub-module to perform fusion decoding on the second parameter to obtain a second decoding result; repeat the fusion decoding operation of each decoding sub-module until the target decoding result output by the last decoding sub-module is obtained.

[0066] In some embodiments, the fusion decoding module 204 is also configured to process the first parameter using a self-attention layer, and to fuse the target visual feature with the processed first parameter using a pixel-aware attention layer to obtain a first fusion feature; to fuse the target edge feature with the first fusion feature using an edge-aware attention layer to obtain a second fusion feature; to fuse the target skeleton feature with the second fusion feature using a skeleton-aware attention layer to obtain a third fusion result; and to perform a nonlinear transformation on the third fusion result using a feedforward network layer to obtain a decoding result corresponding to each decoding sub-module.

[0067] In some embodiments, the segmentation module 205 is further configured to perform pixel prediction on the target decoding result to obtain a predicted classification result of the pixel; and multiply the predicted classification result of the pixel by the target visual feature element by element to obtain a segmented text result.

[0068] In some embodiments, the feature conversion module 203 is further configured to upsample the multi-level visual features to convert low-resolution visual features in the multi-level visual features into high-resolution visual features; and use the upsampled multi-level visual features as target visual features.

[0069] According to the device provided by the embodiment of the present invention, an image to be segmented is obtained, and feature extraction is performed on the image to be segmented to obtain text edge features, text skeleton features and multi-level visual features. By extracting text edge features and text skeleton features, text edge perception and text skeleton perception can be introduced in text segmentation, and multi-level visual features are used to perform feature screening on text edge features and text skeleton features respectively to enhance text edge perception and text skeleton perception, so that the obtained target edge features and target skeleton features can better reflect the text information in the image, and at the same time, the edge and skeleton of the text are focused on during the segmentation process to accurately identify and segment the text area; the multi-level visual features are further converted to obtain target visual features; a target query for guiding image text segmentation is obtained, and according to the target query, the target edge features, target skeleton features and target visual features are fused and decoded to obtain a target decoding result; the text segmentation result is determined according to the target decoding result and the target visual feature, so that the fused decoding result containing multiple guiding features can be used to more accurately perform text segmentation from the image, thereby improving the accuracy of text segmentation.

[0070] Figure 3 Schematic diagram of an electronic device 3 provided in an embodiment of the present invention. Figure 3As shown, the electronic device 3 of this embodiment includes: a processor 301, a memory 302, and a computer program 303 stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program 303, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 301 executes the computer program 303, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0071] The electronic device 3 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 3 may include, but is not limited to, a processor 301 and a memory 302. Those skilled in the art will appreciate that Figure 3 The electronic device 3 is merely an example and does not limit the electronic device 3 , and may include more or less components than those shown in the figure, or different components.

[0072] The processor 301 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0073] The memory 302 may be an internal storage unit of the electronic device 3, for example, a hard disk or memory of the electronic device 3. The memory 302 may also be an external storage device of the electronic device 3, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 3. The memory 302 may also include both an internal storage unit of the electronic device 3 and an external storage device. The memory 302 is used to store computer programs and other programs and data required by the electronic device.

[0074] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0075] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, and the computer program code may be in source code form, object code form, executable file or some intermediate form. The readable storage medium may include: any entity or device that can carry computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0076] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A method for image text segmentation, characterized in that: include: Acquire an image to be segmented, perform feature extraction on the image to be segmented, and obtain text edge features, text skeleton features, and multi-level visual features; Using the multi-level visual features, feature screening is performed on the text edge features and the text skeleton features to obtain target edge features and target skeleton features; Performing feature conversion on the multi-level visual features to obtain target visual features; Obtaining a target query for guiding image text segmentation, and fusing and decoding the target edge features, the target skeleton features, and the target visual features according to the target query to obtain a target decoding result; Determining a text segmentation result according to the target decoding result and the target visual feature; Using the multi-level visual features to perform feature screening on the text edge features and the text skeleton features respectively to obtain target edge features and target skeleton features, including: Concatenating the multi-level visual features and using the target convolutional network as a text detection head; Input the spliced ​​multi-level visual features into the target convolutional network, use the target convolutional network to determine the text area of ​​the multi-level visual features, and obtain a text area detection frame; the target convolutional network is a 1*1 convolutional layer; Multiplying the text edge feature by the text area detection frame element by element to obtain the target edge feature; Multiplying the character skeleton feature by the text area detection frame element by element to obtain the target skeleton feature; According to the target query, the target edge feature, the target skeleton feature and the target visual feature are fused and decoded to obtain a target decoding result, including: Using the target query as a first parameter, and using the target edge feature, the target skeleton feature, and the target visual feature as a second parameter; Inputting the first parameter and the second parameter into a target decoder, wherein the target decoder comprises a plurality of decoding submodules arranged in series; According to the first parameter, using a first decoding submodule to perform fusion decoding on the second parameter to obtain a first decoding result; Using the first decoding result as a new first parameter, and using a second decoding submodule to perform fusion decoding on the second parameter according to the new first parameter to obtain a second decoding result; Repeat the fusion decoding operation of each decoding submodule until the target decoding result output by the last decoding submodule is obtained; The step of performing feature conversion on the multi-level visual features to obtain target visual features includes: Upsampling the multi-level visual features to convert low-resolution visual features in the multi-level visual features into high-resolution visual features; The upsampled multi-level visual features are used as the target visual features.

2. The method according to claim 1, characterized in that Each of the decoding submodules includes a self-attention layer, a pixel-aware attention layer, an edge-aware attention layer, a skeleton-aware attention layer, and a feedforward network layer. The fusion decoding operation of each decoding submodule includes: Using the self-attention layer to process the first parameter, and using the pixel-aware attention layer to fuse the target visual feature with the processed first parameter to obtain a first fused feature; Using the edge-aware attention layer to fuse the target edge feature with the first fused feature to obtain a second fused feature; Using the skeleton-aware attention layer to fuse the target skeleton feature with the second fusion feature to obtain a third fusion result; The third fusion result is nonlinearly transformed by using the feedforward network layer to obtain a decoding result corresponding to each decoding submodule.

3. The method according to claim 1, characterized in that Determining the text segmentation result according to the target decoding result and the target visual feature includes: Perform pixel prediction on the target decoding result to obtain a predicted classification result of the pixel; The predicted classification result of the pixel point is multiplied element by element by the target visual feature to obtain the segmented text result.

4. An image text segmentation device, characterized in that: include: A feature extraction module is configured to obtain an image to be segmented, perform feature extraction on the image to be segmented, and obtain text edge features, text skeleton features, and multi-level visual features; A feature screening module is configured to use the multi-level visual features to perform feature screening on the text edge features and the text skeleton features respectively to obtain target edge features and target skeleton features; Using the multi-level visual features to perform feature screening on the text edge features and the text skeleton features respectively to obtain target edge features and target skeleton features, including: Concatenating the multi-level visual features and using the target convolutional network as a text detection head; Input the spliced ​​multi-level visual features into the target convolutional network, use the target convolutional network to determine the text area of ​​the multi-level visual features, and obtain a text area detection frame; the target convolutional network is a 1*1 convolutional layer; Multiplying the text edge feature by the text area detection frame element by element to obtain the target edge feature; Multiplying the character skeleton feature by the text area detection frame element by element to obtain the target skeleton feature; The feature conversion module is configured to perform feature conversion on the multi-level visual features to obtain target visual features; performing feature conversion on the multi-level visual features to obtain target visual features, including: upsampling the multi-level visual features to convert low-resolution visual features in the multi-level visual features into high-resolution visual features; and using the upsampled multi-level visual features as the target visual features; A fusion decoding module is configured to obtain a target query for guiding image text segmentation, and according to the target query, fuse and decode the target edge feature, the target skeleton feature and the target visual feature to obtain a target decoding result; according to the target query, fuse and decode the target edge feature, the target skeleton feature and the target visual feature to obtain a target decoding result, including: taking the target query as a first parameter, and taking the target edge feature, the target skeleton feature and the target visual feature as a second parameter; inputting the first parameter and the second parameter into a target decoder, the target decoder including a plurality of decoding submodules arranged in succession; according to the first parameter, using the first decoding submodule to fuse and decode the second parameter to obtain a first decoding result; using the first decoding result as a new first parameter, and according to the new first parameter, using the second decoding submodule to fuse and decode the second parameter to obtain a second decoding result; repeatedly performing the fusion decoding operation of each decoding submodule until the target decoding result output by the last decoding submodule is obtained; The segmentation module is configured to determine a text segmentation result according to the target decoding result and the target visual feature.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 3 are implemented.

6. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.

Images