A text detection method, device and recognition system

Abstract

The present application relates to the field of artificial intelligence technology, and provides a text detection method, device and recognition system. The method comprises: obtaining an image to be detected with text; determining each text sub-region in the image to be detected, and determining the character image block corresponding to each character in the image to be detected; obtaining a topological feature map of the text sub-region according to the image feature matrix and adjacency matrix of each text sub-region respectively; wherein the image feature matrix is ​​used to describe the image features of each character image block in the text sub-region, the adjacency matrix is ​​used to describe the correlation between each character image block in the text sub-region, and the topological feature map is used to describe the graph features of the text sub-region; determining the similarity between each two topological feature maps, dividing two text sub-regions whose similarity meets preset conditions into the same category; marking all characters in the text sub-region belonging to the same category as the same detection frame.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a text detection method, device and recognition system. Background Art

[0002] Text detection can be used to locate text in an image. Natural scene text detection is a type of text detection and is widely used in various fields, such as image retrieval, video analysis, and autonomous driving.

[0003] At present, natural scene text detection is mainly achieved through segmentation methods, which determine the pixels in the image that belong to text, and then detect the corresponding text area based on the distribution of text pixels. However, the distance between two text lines in natural scenes is uncertain, and the segmentation method often cannot distinguish text lines that are close to each other well, resulting in poor accuracy of text detection. Summary of the invention

[0004] The embodiments of the present application provide a text detection method, apparatus, recognition system and equipment for improving the accuracy of text detection.

[0005] In a first aspect, a text detection method is provided, comprising:

[0006] Obtain an image to be detected with text;

[0007] Determine each text sub-region in the image to be detected, and determine a character image block corresponding to each character in the image to be detected;

[0008] According to the image feature matrix and adjacency matrix of each text sub-region, a topological feature map of the text sub-region is obtained; wherein the image feature matrix is ​​used to describe the image features of each character image block in the text sub-region, the adjacency matrix is ​​used to describe the correlation between each character image block in the text sub-region, and the topological feature map is used to describe the graph features of the text sub-region;

[0009] Determine the similarity between each two topological feature graphs, and classify two text sub-regions whose similarity meets preset conditions into the same category;

[0010] All characters in the text sub-region belonging to the same category are marked as the same detection box.

[0011] In a second aspect, a text detection device is provided, the device comprising:

[0012] A transceiver module, used for acquiring an image to be detected with text;

[0013] A detection and segmentation module, used to determine each text sub-region in the image to be detected, and to determine a character image block corresponding to each character in the image to be detected;

[0014] A classification module, for obtaining a topological feature map of a text sub-region according to an image feature matrix and an adjacency matrix of each text sub-region, wherein the image feature matrix is ​​used to describe the image features of each character image block in the text sub-region, the adjacency matrix is ​​used to describe the correlation between each character image block in the text sub-region, and the topological feature map is used to describe the graph features of the text sub-region; and determining the similarity between each two topological feature maps, and classifying two text sub-regions whose similarity meets a preset condition into the same category;

[0015] The labeling module is used to label all characters in the text sub-region belonging to the same category as the same detection box.

[0016] In a possible design, the classification module is specifically used for:

[0017] For each text sub-region, obtaining an image feature vector of each character image block in the text sub-region;

[0018] The corresponding image feature vectors in the text sub-region are combined to obtain the image feature matrix of the text sub-region, and the adjacency matrix of the text sub-region is determined;

[0019] The adjacency matrix and image feature matrix are input into the trained graph convolutional network model to obtain the topological feature map of the text sub-region.

[0020] In a possible design, the classification module is specifically used for:

[0021] Determine a similarity vector between image feature vectors corresponding to every two character image blocks in the text sub-region, and combine the determined multiple similarity vectors to obtain a similarity matrix;

[0022] The similarity matrix is ​​normalized to obtain an adjacency matrix.

[0023] In one possible design, the graph convolutional network model includes a plurality of transformation units connected in sequence, each transformation unit includes a normalization layer and a nonlinear layer connected in sequence, and the output of each transformation unit is expressed as follows:

[0024] Z = (ReLU (LayerNorm (GXW)) + X

[0025] Among them, X represents the output of the previous transformation unit, G represents the adjacency matrix constructed using X, W is the weight matrix of the transformation unit, LayerNorm represents the normalization layer, and ReLU represents the nonlinear layer.

[0026] In a possible design, the annotation module is specifically used for:

[0027] Determine the center point of each character image block in all character image blocks under each category, and expand the character image blocks with the average height of a preset number of character image blocks adjacent to each center point as the radius to obtain an initial detection frame corresponding to the character image block under each category;

[0028] Perform edge smoothing on each initial detection frame to obtain a detection frame for each category.

[0029] In a possible design, the character image block is obtained by a character image block prediction model, and the multiple text sub-regions are obtained by an image segmentation model;

[0030] The character image block prediction model, the image segmentation model and the graph convolutional network model are jointly trained through the following steps:

[0031] Acquire a sample data set; wherein the sample data set includes a plurality of sample images to be detected, a plurality of sample images to be detected annotated with character image blocks, a plurality of sample images to be detected annotated with text sub-regions, and a plurality of sample images to be detected annotated with detection frames;

[0032] Using the sample data set to train a character image block prediction model, an image segmentation model, and a graph convolutional network model until the total loss meets a preset condition, thereby obtaining a trained character image block prediction model, an image segmentation model, and a graph convolutional network model;

[0033] Among them, the total loss is obtained by weighting the classification loss of the character image block, the coordinate regression loss of the character image block, the segmentation loss of the text sub-region and the classification loss of the text sub-region. The classification loss of the character image block is used to represent the loss between the predicted character image block classification and the real character image block classification, the coordinate regression loss is used to represent the loss between the predicted character image block coordinates and the real character image block coordinates, the segmentation loss is used to represent the loss between the predicted text sub-region and the real text sub-region, and the classification loss of the text sub-region is used to represent the loss between the predicted classification results of each text sub-region and the real classification results of each text sub-region.

[0034] In a third aspect, a text recognition system is provided, comprising the text detection device and the text recognition device as described in the second aspect, wherein:

[0035] The text recognition device recognizes the character strings in each detection frame according to the detection results marked with the detection frame output by the text detection device.

[0036] In a fourth aspect, a text detection device is provided, comprising:

[0037] at least one processor, and

[0038] a memory communicatively coupled to the at least one processor;

[0039] The memory stores instructions that can be executed by the at least one processor, and the at least one processor implements any method as described in the first aspect by executing the instructions stored in the memory.

[0040] According to a fifth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions. When the computer instructions are executed on a computer, the computer executes any method described in the first aspect.

[0041] Since the embodiment of the present application adopts the above technical solution, it has at least the following technical effects:

[0042] In the embodiment of the present application, the embodiment of the present application obtains a topological feature map of each text sub-region based on the image feature matrix of multiple character image blocks in each text sub-region and the adjacency matrix through a graph convolutional network model. The obtained topological feature map represents the structural relationship between the character image blocks in the text sub-region, so that each text sub-region can be represented more accurately, so that the classification of each text sub-region determined based on the topological feature map is also more accurate, thereby avoiding classifying text sub-regions that are close but belong to different classes into one class, and avoiding classifying text sub-regions that are far away but belong to one class into different classes, thereby improving the accuracy of text detection. In the embodiment of the present application, the character image block is used as the basic component unit of the text sub-region to determine the topological feature map of the text sub-region, so that feature extraction can be performed on text sub-regions of any shape to meet the detection requirements of various texts. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is an example diagram of a process of implementing text detection using a segmentation method in the prior art;

[0044] Figure 2 A structural schematic diagram of a text detection device provided in an embodiment of the present application;

[0045] Figure 3 A schematic diagram of an application scenario of a text detection method provided in an embodiment of the present application;

[0046] Figure 4 A schematic diagram of an application scenario of another text detection method provided in an embodiment of the present application;

[0047] Figure 5A schematic diagram of another text detection method provided in an embodiment of the present application;

[0048] Figure 6 An example diagram of a process for obtaining a text sub-region provided in an embodiment of the present application;

[0049] Figure 7 A schematic diagram of a processing process of a transformation unit provided in an embodiment of the present application;

[0050] Figure 8 An example diagram of a training model process provided in an embodiment of the present application;

[0051] Fig. 9 An interactive schematic diagram of a text detection method provided in an embodiment of the present application;

[0052] Fig.10 A structural schematic diagram of a text detection device provided in an embodiment of the present application;

[0053] Fig.11 A structural schematic diagram of a text detection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to better understand the technical solution provided by the embodiments of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0055] It should be noted that the “multiple” in the embodiments of the present application refers to two or more than two.

[0056] In order to help those skilled in the art better understand the embodiments of the present application, the professional terms involved in the present application are explained below.

[0057] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0058] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0059] Optical Character Recognition (OCR) technology: an electronic device (such as a scanner or digital camera) checks the characters printed on paper and translates the characters into computer text through certain technical means. With the development of artificial intelligence, more and more artificial intelligence technologies are combined with OCR technology to improve the accuracy and efficiency of OCR recognition.

[0060] Natural scene text detection: Natural scene text detection can be understood as an application of OCR technology, which is used to detect text in natural images.

[0061] Text: generally refers to content formed by characters, such as English fragments, Chinese fragments, or English mixed fragments.

[0062] Character: A glyph-like or glyph-like unit or symbol, such as Chinese characters, letters, numbers, operation symbols, punctuation marks and other symbols.

[0063] Text sub-region: refers to the sub-region obtained after segmenting the text region in the image. Each sub-region includes characters, and the shape of each sub-region may be regular or irregular.

[0064] Character image block: refers to the image area corresponding to a character, which includes the image information of the character image block and the geometric information corresponding to the character image block, such as the size of the character image block and the coordinates of the boundary of the character image block. The character image block can be a regular area or an irregular area. For example, if a single character box is marked by a detector, the area corresponding to the single character box is the character image block.

[0065] Graph features: including point information and edge information. The graph features in the embodiments of the present application are used to represent the structural relationship between the various character image blocks in the text sub-region. The structural relationship here can be understood as the feature information of each character image block (equivalent to the point information) and the feature information of the neighboring character image blocks of the character image block (equivalent to the edge information).

[0066] The design concept of the embodiments of the present application is introduced below.

[0067] The following first introduces the method of using segmentation to achieve text detection in the prior art.

[0068] Taking semantic segmentation as an example, we determine the pixels belonging to text in natural scene text, extract the boundaries of similar pixels, and thus obtain the bounding boxes of each text region. However, the distance between text lines in natural scene text is uncertain, so this segmentation method tends to segment two text lines that are close to each other into one region, while a text line that is far away is segmented into two regions, resulting in low detection accuracy.

[0069] For example, see Figure 1 , which represents the method of realizing text detection by segmentation. Figure 1 a represents a natural scene text image. Figure 1 The distance between "Para" and "dise" in a is far, so in the segmentation process, "Para" will be divided into one text region and "dise" will be divided into another text region. The segmentation results are as follows: Figure 1 As shown in b, "Para" corresponds to the first detection box 101, and "dise" corresponds to the second detection box 102. If text recognition is subsequently performed based on the detection result, "Para" and "dise" will be recognized separately, resulting in poor accuracy of the final recognition result.

[0070] In view of this, the inventor of the present application designed a text detection method, which first obtains multiple text sub-regions in the image to be detected and multiple character image blocks in the image to be detected. Because the multiple text sub-regions and the multiple character image blocks are all derived from the image to be detected, in fact each character image block will belong to the corresponding text sub-region.

[0071] At this time, each text sub-region is regarded as a graph, and each character image block in each text sub-region is regarded as a node of the text sub-region. The idea of ​​graph convolutional neural network (GCN) is used to obtain the image features of multiple character image blocks in a text sub-region and the adjacency matrix between the multiple character image blocks. The image feature matrix and the adjacency matrix of the multiple character image blocks in a text sub-region are input into the trained GCN model to obtain a topological feature map of a text sub-region. By analogy, the topological feature map of each text sub-region in multiple text sub-regions can be obtained.

[0072] Based on the determined topological feature map, the similarity between any two topological feature maps is determined. If the similarity meets the preset conditions, it means that the distribution structure of the character image blocks and the characteristics of the character image blocks in the two text sub-regions are similar. Therefore, the two text sub-regions whose similarity meets the preset conditions are classified into one category, thereby obtaining a more accurate detection result.

[0073] After introducing the main design ideas of the embodiments of the present application, the application scenarios involved in the embodiments of the present application are illustrated below.

[0074] The text detection method in the embodiment of the present application can be implemented by a text detection device, please refer to Figure 2 , which is a structural example diagram of a text detection device 200, wherein the text detection device 200 includes one or more input devices 201, one or more processors 202, one or more memories 203 and one or more output devices 204.

[0075] The input device 201 is used to provide an input interface to obtain an image to be detected with text input by an external device / user. After obtaining the image to be detected, the input device 201 sends the image to be detected to the processor 202. The processor 202 uses the program instructions stored in the memory 203 to perform text detection on the image to be detected and obtain a text detection result. The text detection result is output through the output device 204.

[0076] The input device 201 may include but is not limited to one or more of a physical keyboard, a function key, a trackball, a mouse, a touch screen, a joystick, etc. The processor 202 may be a central processing unit (CPU), or a digital processing unit, etc. The memory 203 may be a volatile memory, such as a random-access memory (RAM); the memory 203 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), or the memory 203 may be any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 203 may be a combination of the above memories. The output device 204 may be a display, a speaker, a printer, etc.

[0077] In a possible embodiment, the text detection device 200 can be a user terminal device or a server device. The user terminal device can be a mobile terminal, a fixed terminal or a portable terminal, such as a mobile phone, a site, a unit, a device, a multimedia computer, a multimedia tablet, an Internet node, a communicator, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an electronic book device, a gaming device or any combination thereof, including accessories and peripherals of these devices or any combination thereof. It is also foreseeable that the text detection device 200 can support any type of interface (such as a wearable device) for the user, etc. The server device can be a server, a large computing device, etc. provided by various services. The server can be one or more servers. The server can also be a physical server or a virtual server, etc.

[0078] Please refer to Figure 3 , which represents an example diagram of an application scenario of a text detection device, and can also be represented as a structural diagram of a text recognition system. The recognition system 300 includes a text detection device 200 and a text recognition device 310. After obtaining the text detection result, the text detection device 200 can output the text detection result to the text recognition device 310, and the text recognition device 310 recognizes the specific text and displays it.

[0079] Furthermore, the text recognition device 310 can also convert the text, such as translate it, so as to obtain the result desired by the user.

[0080] Please refer to Figure 4 , represents an example diagram of an application scenario of a text detection device, the application scenario includes a server 420, a terminal device 410, and a client 411 installed in the terminal device 410. The client 411 is used to represent a client with a text processing function, such as a translation client, a web search client, etc. The client 411 communicates with the server 420 via a network. There are many types of networks, and this application does not limit the communication method. The server 420 in this application scenario is equivalent to being provided with the text recognition device 200 described above, or being provided with the text detection device 200 and the text recognition device 310 discussed above. The terminal device 410 in this application is, for example, a mobile phone, a personal computer, a camera, a vehicle-mounted terminal, etc. The specific implementation method of the server 420 can refer to the content discussed above, which will not be repeated here.

[0081] The user selects an image to be detected through the client 411 in the terminal device 410, the client 411 obtains the image to be detected, the server 420 receives the image to be detected, obtains the text detection result by performing the above-mentioned processing on the image to be detected, and returns the text detection result to the client 411. The client 411 generally refers to a client 411 that can realize image style conversion, such as a camera app, an image processing app, etc., and this application does not limit the specific type of the client 411.

[0082] Alternatively, after obtaining the text detection result, the server 420 may perform text recognition on the text detection result and feed the text recognition result back to the client 411 , or further process the text recognition result and feed it back to the client 411 .

[0083] The principle of the text detection method in the embodiment of the present application is introduced below.

[0084] Please refer to Figure 5 , which is a schematic diagram showing the principle of the text detection method. The text detection process mainly includes S510: determining the character image block part, S520: determining the text sub-region part, S530: determining the classification part and S540: determining the detection frame part. Figure 5 , and introduce each part.

[0085] S510, determining the character image block portion:

[0086] The character image block determination part is mainly used to determine multiple character image blocks in the image to be detected, that is, to mark each character in the image to be detected. The text detection device 200 can use a character image block prediction model (also called a single-character frame prediction model) to determine multiple character image blocks in the image to be detected. The character image block prediction model can be essentially understood as a detector, which marks a single-character frame for the detected characters. The shape of the single-character frame can be arbitrary, and rectangular frames are more common.

[0087] In a possible embodiment, the character image block prediction model may be implemented by using a Region Proposal Network (RPN) model, for example.

[0088] Specifically, the RPN model will preset anchor frames of one or more sizes, perform sliding window processing on the image to be detected, generate the probability that each target candidate frame corresponds to a single character, and determine the target candidate frame whose probability value belonging to a single character is greater than a preset threshold as a character image block.

[0089] For example, the network model of RPN includes a convolutional layer, a Relu layer, a Reg layer, and a Cls layer connected in sequence. Figure 6, which is a schematic diagram of the RPN model processing process. The convolution layer performs convolution processing on the image to be detected to obtain a 1*1*256-dimensional feature map 610. The feature map 610 is processed by sliding window 620 to generate K possible target candidate boxes 630. The Reg layer predicts the position offset of the K target candidate boxes. Here, a rectangular candidate box is taken as an example. Therefore, each candidate box corresponds to 4 position offsets, thereby obtaining 4K position offsets. The Cls layer predicts whether the K target candidate boxes 630 include a single character, because each target candidate box includes two probabilities of results, one probability of belonging to a single character, and the other probability of not belonging to a single character. Therefore, 2K probability values ​​can be obtained, and the corresponding character image block is determined based on the 2K probability values.

[0090] For example, please refer to Figure 5 , the image to be detected is Figure 5 As shown in a, after determining the character image block in the image to be detected, the following Figure 5 The image shown in b, Figure 5 Each character in the image in b is marked by a candidate box, which is equivalent to obtaining each character image block.

[0091] S520, determining a text sub-region portion:

[0092] The part of determining the text sub-region is mainly used to determine the text region in the image to be detected and divide it. The division method can use the idea of ​​image segmentation to process the image to be detected. The specific method of determining the text sub-region is illustrated below.

[0093] In a possible embodiment, multiple text sub-regions can be obtained through an image segmentation model. The image segmentation model can, for example, determine the classification of each pixel in the image to be detected, where the classification includes two classifications: belonging to a character or not belonging to a character. After determining the classification of each pixel in the image to be detected, all pixel sets belonging to the character can be determined, and then all pixel sets belonging to the character are divided to obtain multiple text sub-regions. The image segmentation model can, for example, adopt an instance segmentation model, and the segmentation model can be, for example, an encoding-decoding model.

[0094] For example, please refer to Figure 5 , the image to be detected is Figure 5 As shown in a, the image to be detected includes multiple lines of text. After determining the text sub-region in the image to be detected, the following can be obtained: Figure 5 The image shown in c, Figure 5 The image shown in c includes various text sub-regions.

[0095] S530, determining the classification part:

[0096] Multiple character image blocks and multiple text sub-regions are all derived from the image to be detected. Therefore, each character image block must belong to a certain text sub-region. Therefore, all character image blocks included in each text sub-region can be determined, and these character image blocks can represent the text sub-region.

[0097] In a possible embodiment, for each character image block among multiple character image blocks, the ratio of the overlapping area of ​​each character image block and a text sub-region to the area of ​​each character image block is determined; if the ratio is greater than a preset value, it is determined that each character image block belongs to a text sub-region.

[0098] Specifically, after obtaining multiple character image blocks and multiple text sub-regions, the overlapping area of ​​each character image block and each text sub-region can be determined. If the ratio of the overlapping area to the area of ​​the character image block is greater than a preset value, it means that there is a large area of ​​duplication between the character image block and the text sub-region, so it is determined that the character image block actually belongs to the text sub-region. For example, a specific determination method is as follows:

[0099] area(Cn)∩area(Rm) / area(Cn)>0.8 (1)

[0100] Among them, Cn represents a character image block and Rm represents a text sub-region.

[0101] After determining the relationship between each character image block and the text sub-region, the features of the text sub-region can be extracted. Since the text sub-region is usually irregular, it is difficult to accurately extract the features of these text sub-regions using traditional convolutional neural networks. Therefore, this application uses the GCN model to extract the character image blocks in the text sub-region for modeling and feature extraction, thereby achieving the purpose of classifying the text sub-region.

[0102] Specifically, the essence of the GCN model is to learn the features of all character image blocks in each text sub-region and the correlation between each character image block, so as to determine the topological feature map of the text sub-region. This involves how to express the features of all character image blocks in the text sub-region and the correlation between each character image block in the text sub-region, which is explained by an example below.

[0103] Express the features of all character image blocks in the text sub-region:

[0104] The image feature matrix in the text sub-region may be used to represent the features of all character image blocks in the text sub-region.

[0105] Specifically, since all the character image blocks included in the text sub-region have been determined in the previous text, feature extraction is directly performed on each character image block to extract the image feature vector of each character image block in all the character image blocks in each text sub-region. For example, the image feature vector of a character image block can be expressed as: X1 = {x1, x2..., xn}. There are many ways to extract features, such as the region of interest pooling (ROIpooling) method. After obtaining the image feature vector of each character image block in all the character image blocks in the text sub-region, the image feature vectors of all the character image blocks can be combined to obtain an image feature matrix. The combination method can, for example, be to use the image feature vector of a character image block as a row of the matrix, and so on, so as to obtain an image feature matrix. For example, the image feature matrix in a text sub-region can be expressed as

[0106] The correlation between each character image block:

[0107] An adjacency matrix may be used to express the correlation between the character image blocks in the text sub-region.

[0108] Specifically, the text detection device 200 determines the similarity vector of the image feature vectors corresponding to every two character image blocks in each text sub-region, combines the determined multiple similarity vectors, obtains a similarity matrix, performs normalization processing, and obtains an adjacency matrix. The normalization processing process can be implemented by a normalization function (softmax).

[0109] For example, cosine similarity is used to determine the similarity of image feature vectors corresponding to every two character image blocks, multiple similarities are obtained, the multiple similarities are combined into a similarity matrix, and the similarity matrix is ​​normalized to obtain an adjacency matrix.

[0110] Or, for example, the similarity vector between every two character image blocks is calculated using the following formula:

[0111] F(X i ,X j )=φ(X i ) T φ'(X j ) (2)

[0112] Among them, X i Represents the image feature vector of a character image block, X j represents the image feature vector of another character image block, and φ and φ' represent two different nonlinear transformations.

[0113] For example, φ can be implemented by one or more fully connected layers, and φ' can also be implemented by one or more fully connected layers, using the fully connected layers to map the image feature vectors of the two characters into a nonlinear space.

[0114] If the similarity matrix is ​​not normalized, each character tends to have a larger eigenvalue in subsequent processing. Therefore, the similarity matrix can be normalized to obtain a more standard adjacency matrix.

[0115] G = Softmax(F) (3)

[0116] After obtaining the adjacency matrix and the image feature matrix, the adjacency matrix and the image feature matrix can be input into the GCN model.

[0117] As an embodiment, the GCN model includes at least two transformation units connected in sequence, each transformation unit includes one or more layers, and each transformation unit shares the same calculation method.

[0118] In a possible embodiment, the output of each transformation unit can be expressed as follows:

[0119] Z=(ReLU(LayerNorm(GXW))+X (4)

[0120] Among them, X represents the output of the previous transformation unit, G represents the adjacency matrix constructed using the output X of the previous transformation unit, W is the weight matrix of the transformation unit, LayerNorm represents the normalization layer, ReLU represents the nonlinear layer, and Z represents the output of the transformation unit.

[0121] Please refer to Figure 7 , which represents the schematic diagram of the transfer process in the GCN model. X of the first transformation unit in the GCN model is the image feature matrix discussed above, and G of the first transformation unit is the adjacency matrix discussed above. The first transformation unit uses the weight matrix W to perform convolution processing on GX, and then performs normalization processing, and then performs nonlinear transformation processing, and then adds the image feature matrix based on the nonlinear transformation processing result to obtain the output result Z of the first transformation unit. Of course, in fact, after GXW is subjected to nonlinear transformation, it can be used as the output result. However, by adding the result after nonlinear transformation to the image feature matrix, the image characteristics of each character image block can be retained.

[0122] The output result of the first transformation unit is Z, which is actually also a matrix. The adjacency matrix of the second transformation unit can be constructed based on Z. The following is an example of how to construct the adjacency matrix of the second transformation unit.

[0123] For example, the output of the first transformation unit At this time, Z is used as the input X of the second transformation unit. K1...Km are used as variables in formula (2) to determine the similarity vector between every two vectors in Z. Similarly, a similarity matrix is ​​obtained. Then, the similarity matrix is ​​normalized based on formula (3) to obtain the adjacency matrix of the second transformation unit. Then, the second transformation unit processes the adjacency matrix and Z to obtain the output result of the second transformation unit. Similarly, the output of the last layer in the GCN model is obtained to obtain the topological feature map R of the text sub-region. G 1.

[0124] Since the GCN model processes the features of each character image block and the relationships between each character image block, the final topological feature map actually includes the image features of each character image block in the text sub-region and the relationships between each character image block.

[0125] The above article discusses the processing process for one text sub-region, and the processing process for other text sub-regions can refer to the content discussed above, which will not be repeated here. After processing all text sub-regions, a topological feature map of each text sub-region can be obtained.

[0126] After obtaining the topological feature maps of each text sub-region, the similarity of the topological feature maps of each two text sub-regions can be determined, and two text sub-regions whose similarity meets a preset condition can be determined as the same category. The preset condition can be that the similarity is greater than or equal to a preset threshold, and so on, which text sub-regions belong to the same category can be obtained. The classification can be, for example, whether these text sub-regions belong to a text line, etc.

[0127] As an embodiment, determining the similarity of the topological feature graphs of each two text sub-regions may be determining the cosine similarity of the topological feature graphs of each two text sub-regions.

[0128] After obtaining each topological feature map, the text detection device 200 determines the classification of each text sub-region, which is equivalent to determining the classification of the character image blocks in each text sub-region.

[0129] As an embodiment, different identifiers may be marked for text sub-regions of different categories to distinguish text sub-regions of different categories. Different identifiers may be different colors, for example.

[0130] For example, please refer to Figure 5 ,right Figure 5 The image shown in b and Figure 5 After processing the image shown in c, we can get Figure 5In the image shown in e, different colors are used to distinguish different categories, which is equivalent to determining the classification of each character image block. Therefore, the following can be obtained at the same time: Figure 5 In the image shown in (d), the labeled boxes of different colors in (d) indicate character image blocks of different categories.

[0131] S540, determining the detection frame:

[0132] After determining the classification of each character image block, the text detection device 200 can group the text sub-regions belonging to the same classification together, thereby obtaining a detection frame under the corresponding classification.

[0133] As an embodiment, character image blocks belonging to the same category can be annotated together with a detection frame. As discussed above, the classification of each text sub-region has been determined, which is equivalent to obtaining the classification corresponding to each character image block, and then the character image blocks belonging to the same category are annotated in a detection frame, thereby achieving accurate classification of all character image blocks.

[0134] As an embodiment, the center of each character image block in the character image blocks belonging to the same category can be determined, and the center can be expanded with the average value of the height of the character image blocks adjacent to a preset number of the center point as the radius, or the center can be expanded with the maximum value of the height of the character image blocks adjacent to a preset number of the center point as the radius, to obtain the initial detection frame corresponding to the category. The preset number is, for example, 3.

[0135] Specifically, the center of each character image block in the character image blocks of the same category is determined as the skeleton line, and the average or maximum value of the heights of a preset number of character image blocks around the center of each character image block is used as the radius for expansion, so as to obtain the initial detection frame corresponding to the category. For example, the expansion can be performed along the normal direction of the skeleton line.

[0136] In an embodiment of the present application, a preset number of character image blocks are used as processing units. In this way, the radius corresponding to each processing unit is actually not exactly the same, so that the size of the obtained detection frame is not completely consistent and can adapt to text detection of any shape or size.

[0137] Furthermore, since the radius corresponding to each preset number of character image blocks is different, the shape of the marked initial detection frame may be a polygon, so the initial detection frame can be edge-smoothed to obtain a detection frame with smoother lines. The edge smoothing process uses, for example, a thin plate spline (TPS) interpolation method. TPS uses the minimum curvature surface to control the fitting control points, and uses the control points to transform the coordinate points in the initial detection frame.

[0138] The previous article introduced the processing of each part. The following is an example of the training process of the model involved in each part.

[0139] The character image block preset model, image segmentation model and graph convolutional network model can be trained separately. If the three models are trained separately, the loss of each model only needs to meet the loss of the model training process. The character image block preset model, image segmentation model and graph convolutional network model can also be trained together. Figure 8 The example diagram of the training process in FIG. 1 illustrates the method of joint training.

[0140] Part I:

[0141] A sample data set 810 is obtained.

[0142] The sample data set 810 includes a plurality of sample images to be detected, a plurality of sample images to be detected annotated with character image blocks, a plurality of sample images to be detected annotated with text sub-regions, and a plurality of sample images to be detected annotated with detection boxes. The sample data set 810 may be manually annotated by a user, or the sample data set 810 may be obtained from a network resource.

[0143] Part II:

[0144] The sample data set 810 is used to train the character image block preset model, the image segmentation model and the graph convolutional network model until the total loss meets the preset conditions, and the model training is completed.

[0145] Specifically, the total loss can be obtained by weighting the classification loss of the character image block, the coordinate regression loss of the character image block, the segmentation loss of the text sub-region, and the classification loss of the text sub-region. The preset condition can be that the total loss is less than or equal to a preset value, or each of the classification loss of the character image block, the coordinate regression loss of the character image block, the segmentation loss of the text sub-region, and the classification loss of the text sub-region is less than or equal to the corresponding preset value.

[0146] Among them, the coordinate regression loss of the character image block is used to represent the loss between the coordinates of the predicted character image block and the real coordinates of the character image block. The predicted coordinates of the character image block can be understood as the training result 820 output by the preset model of the character image block, and the real coordinates of the character image block can be understood as the coordinates of the character image block corresponding to the sample image to be detected which is marked with the character image block.

[0147] The segmentation loss of the text sub-region is used to represent the loss between the predicted text sub-region and the real text sub-region. The predicted text sub-region can be understood as the output result 830 of the image segmentation model, and the real text sub-region can be understood as the text sub-region corresponding to the sample to be detected image of the labeled text sub-region.

[0148] The classification loss of the text sub-region is used to represent the loss between the predicted classification results within each text sub-region and the actual classification results of each text sub-region. The predicted classification results within each text sub-region can be understood as the classification results 840 of text sub-regions belonging to the same category determined based on the output topological feature map of the graph convolutional network model. The actual classification results corresponding to each text sub-region can be understood as the classification corresponding to each text sub-region in the sample to be detected marked with a detection box.

[0149] The classification loss of the character image block is used to represent the loss between the predicted category to which the character image block belongs and the actual category of the character image block. The predicted category to which the character image block belongs can be understood as the classification result 840 of each character image block belonging to the same category determined based on the output topological feature map of the graph convolutional network model. The actual category of the character image block can be understood as the category corresponding to each character image block in the sample image to be detected marked with a detection box.

[0150] As an embodiment, the classification loss of the text sub-region can be represented by a cross entropy loss function, and the classification loss of this sub-region can adopt a contrastive loss, that is, the predicted classification of each text sub-region is compared with the classification of the actual text sub-region.

[0151] In a possible embodiment, model parameters in the graph convolutional network model can use empirical values ​​without prior training.

[0152] In a possible embodiment, the above-mentioned model training process may be performed by other devices, and after the training of other devices is completed, the text detection apparatus 200 directly uses it.

[0153] The text detection in the embodiment of the present application performs well in terms of recall, precision and F-measure indicators. The F-measure indicator can be understood as the weighted harmonic average of precision and recall. It has been verified that the precision of the text detection method in the embodiment of the present application can reach 89.7, the recall can reach 85.3, and the F-measure indicator can reach 87.9.

[0154] Based on the text detection principle discussed above, the text detection method involved in the present application is illustrated below. In the embodiment of the present application, the text detection device 200 is illustrated by taking the server 420 as an example.

[0155] Please refer to Fig. 9 , represents a text detection method in an embodiment of the present application.

[0156] S901, the terminal device 410 obtains an image to be detected in response to an input operation of a user.

[0157] Specifically, the user can perform interface input operations on the client 411 of the terminal device 410, such as clicking operations, voice operations, or gesture operations, etc., and select to input the corresponding image to be detected, and the terminal device 410 is equivalent to obtaining the image to be detected.

[0158] As an embodiment, the user can select an image pre-stored in the terminal device 410 as the image to be detected, for example, the user takes a picture of an image with text and uploads it to obtain the translation results of these texts, etc., or the terminal device 410 takes a picture of the current real scene as the image to be detected under the preset conditions, such as the user issues a shooting operation instruction, or the environmental parameters of the terminal device 410 do not meet the preset range values, such as the terminal device 410 is a vehicle-mounted terminal, and the terminal device 410 takes a picture of the signpost in the previous text when the ambient brightness is low, so that the user or the vehicle can identify the road ahead. For example, when the concentration of harmful gases in the car is too high or the temperature is too high, the terminal device 410 can automatically take a picture of the real scene as the image to be detected, so as to facilitate the later inspection of traffic accidents, etc.

[0159] S902 , the terminal device 410 sends the image to be detected to the server 420 .

[0160] Specifically, after the terminal device 410 obtains the image to be detected, it sends the image to be detected to the server 420 through the network.

[0161] S903, the server 420 determines a plurality of text sub-regions in the image to be detected, and determines a plurality of character image blocks in the image to be detected.

[0162] Specifically, the server 420 may use the image segmentation model discussed above to obtain multiple text sub-regions. The server 420 may use the character image block prediction model discussed above to obtain multiple character image blocks.

[0163] In a possible embodiment, the server 420 can perform feature extraction on the image to be detected, and upsample the feature extraction results to obtain a preprocessed feature map, and then input the preprocessed feature map into the image segmentation model and the character image block prediction model respectively, so as to obtain multiple text sub-regions and multiple character image blocks.

[0164] As an embodiment, the server 420 may perform feature extraction on the image to be detected through a 50-layer residual network (Res50) to obtain a feature extraction result. The server 420 then performs multiple upsampling processes on the feature extraction result through a feature pyramid network, for example, three times, to obtain a preprocessed feature map. The size of the preprocessed feature map can be controlled to be 1 / 4 of the size of the image to be detected.

[0165] S904, the server 420 inputs the image feature matrix and adjacency matrix of the text sub-region into the graph convolutional network model to obtain a topological feature graph.

[0166] In a possible embodiment, an image feature vector of each character image block among all character image blocks within each text sub-region is obtained; the feature vectors of all character image blocks within each text sub-region are combined to obtain an image feature matrix, and an adjacency matrix of each text sub-region is determined; wherein the image feature matrix of the character image block is composed of the image feature vectors of all character image blocks included in each text sub-region; the adjacency matrix and the image feature matrix are input into a trained graph convolutional network model to obtain a topological feature map of each text sub-region.

[0167] As an embodiment, a similarity vector of image feature vectors corresponding to every two character image blocks in each text sub-region is determined, and multiple similarity vectors determined are combined to obtain a similarity matrix; the similarity matrix is ​​normalized to obtain an adjacency matrix.

[0168] The method of obtaining the image feature matrix and the adjacency matrix can also refer to the content discussed above, which will not be repeated here. After obtaining the image feature matrix and the adjacency matrix, the server 420 can input the image feature matrix and the adjacency matrix of each text sub-region into the graph convolutional network model, thereby obtaining the topological feature map of the text sub-region, thereby obtaining the topological feature map corresponding to each text sub-region.

[0169] As an embodiment, the graph convolutional network model can be trained in advance by the server 420, or can be trained in advance by other devices.

[0170] S905: The server 420 determines the classification of each text sub-region according to each topological feature graph.

[0171] Specifically, the server 420 determines the similarity between each topological feature graph, and can classify two text sub-regions whose similarity is greater than or equal to a preset threshold into the same category, and so on, thereby determining the classification results corresponding to each text sub-region. The server 420 can determine the cosine similarity between each topological feature graph, and classify the text sub-regions whose cosine similarity is greater than or equal to a preset threshold into one category.

[0172] S906: The server 420 marks the text sub-regions belonging to the same category in the same detection box.

[0173] Specifically, the server 420 marks the text sub-region belonging to a category with a detection frame, so that the corresponding detection frames in the image to be detected can be obtained.

[0174] As an embodiment, the server 420 may directly send the detection box annotation result to the terminal device 410 .

[0175] S907, the server 420 recognizes the text in the detection box.

[0176] Specifically, the server 420 may identify the characters in each detection frame according to the detection result, thereby identifying the text in the image to be detected.

[0177] As an embodiment, the server 420 may also translate the text to obtain a text translation result.

[0178] S908 , the server 420 sends the recognized text to the terminal device 410 .

[0179] After acquiring the text or the text translation result, the terminal device 410 displays the text or the text translation result.

[0180] As an embodiment, S901-S902, S906, and S907 are three optional parts.

[0181] Based on the same inventive concept, the present application embodiment provides a text detection device, please refer to Fig.10 , the text detection device 200 comprises:

[0182] The transceiver module 1001 is used to obtain an image to be detected with text;

[0183] The detection and segmentation module 1002 is used to determine each text sub-region in the image to be detected, and determine the character image block corresponding to each character in the image to be detected;

[0184] The classification module 1003 is used to obtain a topological feature map of the text sub-region according to the image feature matrix and the adjacency matrix of each text sub-region, wherein the image feature matrix is ​​used to describe the image features of each character image block in the text sub-region, the adjacency matrix is ​​used to describe the correlation between each character image block in the text sub-region, and the topological feature map is used to describe the graph features of the text sub-region; and determine the similarity between each two topological feature maps, and classify two text sub-regions whose similarity meets a preset condition into the same category;

[0185] The marking module 1004 is used to mark all characters in the text sub-region belonging to the same category as the same detection box.

[0186] In a possible embodiment, the detection and segmentation module 1002 is specifically used for:

[0187] Perform feature extraction on the image to be detected, and upsample the feature extraction result to obtain a preprocessed feature map;

[0188] The pixels belonging to the text in the preprocessing feature map are divided into regions to obtain various text sub-regions of the image to be detected, and each character in the preprocessing feature map is detected to determine the character image block corresponding to each character in the image to be detected.

[0189] In a possible embodiment, the detection and segmentation module 1002 is further configured to:

[0190] Before obtaining the topological feature map of each text sub-region based on the image feature matrix of each text sub-region and the adjacency matrix of each text sub-region, for each character image block, determine that the text sub-region whose ratio of the first area to the second area is greater than a preset value is the text sub-region to which the character image block belongs; wherein the first area is the area of ​​the overlapping area between the character image block and the text sub-region, and the second area is the area of ​​the character image block.

[0191] In a possible design, the classification module 1003 is specifically used for:

[0192] For each text sub-region, obtaining an image feature vector of each character image block in the text sub-region;

[0193] The corresponding image feature vectors in the text sub-region are combined to obtain the image feature matrix of the text sub-region, and the adjacency matrix of the text sub-region is determined;

[0194] The adjacency matrix and image feature matrix are input into the trained graph convolutional network model to obtain the topological feature map of the text sub-region.

[0195] In a possible design, the classification module 1003 is specifically used for:

[0196] Determine a similarity vector between image feature vectors corresponding to every two character image blocks in the text sub-region, and combine the determined multiple similarity vectors to obtain a similarity matrix;

[0197] The similarity matrix is ​​normalized to obtain the adjacency matrix.

[0198] In one possible design, the graph convolutional network model includes a plurality of transformation units connected in sequence, each transformation unit includes a normalization layer and a nonlinear layer connected in sequence, and the output of each transformation unit is expressed as follows:

[0199] Z = (ReLU (LayerNorm (GXW)) + X

[0200] Among them, X represents the output of the previous transformation unit, G represents the adjacency matrix constructed using X, W is the weight matrix of the transformation unit, LayerNorm represents the normalization layer, and ReLU represents the nonlinear layer.

[0201] In a possible design, the marking module 1004 is specifically used for:

[0202] Determine the center point of each character image block in all character image blocks under each category, and expand the character image blocks with the average height of a preset number of character image blocks adjacent to each center point as the radius to obtain an initial detection frame corresponding to the character image block under each category;

[0203] Perform edge smoothing on each initial detection frame to obtain a detection frame for each category.

[0204] In one possible design, the character image patch is obtained through a character image patch prediction model, and the multiple text sub-regions are obtained through an image segmentation model;

[0205] The character image block prediction model, image segmentation model and graph convolutional network model are trained together through the following steps:

[0206] Acquire a sample data set; wherein the sample data set includes a plurality of sample images to be detected, a plurality of sample images to be detected annotated with character image blocks, a plurality of sample images to be detected annotated with text sub-regions, and a plurality of sample images to be detected annotated with detection frames;

[0207] Using the sample data set to train the character image block prediction model, image segmentation model and graph convolutional network model until the total loss meets the preset conditions, and obtain the trained character image block prediction model, image segmentation model and graph convolutional network model;

[0208] Among them, the total loss is obtained by weighting the classification loss of the character image block, the coordinate regression loss of the character image block, the segmentation loss of the text sub-region, and the classification loss of the text sub-region. The classification loss of the character image block is used to represent the loss between the predicted character image block classification and the real character image block classification, the coordinate regression loss is used to represent the loss between the predicted character image block coordinates and the real character image block coordinates, the segmentation loss is used to represent the loss between the predicted text sub-region and the real text sub-region, and the classification loss of the text sub-region is used to represent the loss between the predicted classification results of each text sub-region and the real classification results of each text sub-region.

[0209] Based on the same inventive concept, the present application embodiment provides a text recognition system, please continue to refer to Figure 3 , the text recognition system 300 comprises:

[0210] The method comprises the text detection device 200 and the text recognition device 310 discussed above, wherein:

[0211] The text recognition device 310 recognizes the character strings in each detection frame according to the detection results marked with the detection frames output by the text detection device 200 .

[0212] The text detection device 200 in the embodiment of the present application can implement any of the text detection methods discussed above.

[0213] Based on the same inventive concept, the present application embodiment provides a text detection device, please refer to Fig.11 The text detection device 1100 is in the form of a general-purpose computing device. The components of the text detection device 1100 may include, but are not limited to: at least one processor 1110, at least one memory 1120, and a bus 1130 connecting different system components (including the processor 1110 and the memory 1120).

[0214] Bus 1130 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0215] The memory 1120 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 1121 and / or a cache memory 1122 , and may further include a read-only memory (ROM) 1123 .

[0216] The memory 1120 may also include a program / utility 1126 having a set (at least one) of program modules 1125, such program modules 1125 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The processor 1110 is used to execute program instructions stored in the memory 1120 to implement the text detection method discussed above.

[0217] The text detection device 1100 can also communicate with one or more external devices 1140 (e.g., keyboards, pointing devices, etc.), can also communicate with one or more devices that enable a terminal device to interact with the text detection device 1100, and / or communicate with any device that enables the text detection device 1100 to communicate with one or more other devices (e.g., routers, modems, etc.). This communication can be carried out through an input / output (I / O) interface 1150. In addition, the text detection device 1100 can also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs) and / or public networks, such as the Internet) through a network adapter 1160. As shown in the figure, the network adapter 1160 communicates with other modules for the text detection device 11000 through a bus 1130. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the text detection device 1100, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0218] As an example, Fig.11 The text detection device in can implement the functions of any of the text detection devices 200 discussed above, or implement the text detection method discussed above.

[0219] As an example, Fig.11 The text detection device in can be used to implement the functions of the server 420 discussed above.

[0220] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on a computer, the computer executes the text detection method discussed above.

[0221] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0222] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A text detection method, characterized in that: include: Obtain an image to be detected with text; Determine each text sub-region in the image to be detected, and determine a character image block corresponding to each character in the image to be detected; According to the image feature matrix and adjacency matrix of each text sub-region, a topological feature map of the text sub-region is obtained; wherein the image feature matrix is ​​used to describe the image features of each character image block in the text sub-region, the adjacency matrix is ​​used to describe the correlation between each character image block in the text sub-region, and the topological feature map is used to describe the graph features of the text sub-region; Determine the similarity between each two topological feature graphs, and classify two text sub-regions whose similarity meets preset conditions into the same category; All characters in the text sub-region belonging to the same category are marked as the same detection box; Among them, according to the image feature matrix and adjacency matrix of each text sub-region, the topological feature map of the text sub-region is obtained, including: For each text sub-region, obtaining an image feature vector of each character image block in the text sub-region; The corresponding image feature vectors in the text sub-region are combined to obtain the image feature matrix of the text sub-region, and the adjacency matrix of the text sub-region is determined; The adjacency matrix and image feature matrix are input into the trained graph convolutional network model to obtain the topological feature map of the text sub-region.

2. The method according to claim 1, characterized in that Determining each text sub-region in the image to be detected, and determining a character image block corresponding to each character in the image to be detected, comprising: Performing feature extraction on the image to be detected, and performing upsampling processing on the feature extraction result to obtain a preprocessing feature map; The pixels belonging to the text in the preprocessing feature map are divided into regions to obtain various text sub-regions of the image to be detected, and each character in the preprocessing feature map is detected to determine the character image block corresponding to each character in the image to be detected.

3. The method according to claim 1, characterized in that Before obtaining the topological feature map of each text sub-region according to the image feature matrix of each text sub-region and the adjacency matrix of each text sub-region, it includes: For each character image block, a text sub-region whose ratio of the first area to the second area is greater than a preset value is determined as the text sub-region to which the character image block belongs; wherein the first area is the area of ​​the overlapping area between the character image block and the text sub-region, and the second area is the area of ​​the character image block.

4. The method according to claim 1, characterized in that Determine the adjacency matrix of the text subregion, including: Determine a similarity vector between image feature vectors corresponding to every two character image blocks in the text sub-region, and combine the determined multiple similarity vectors to obtain a similarity matrix; The similarity matrix is ​​normalized to obtain an adjacency matrix.

5. The method according to claim 1, characterized in that The graph convolutional network model includes a plurality of transformation units connected in sequence, each transformation unit includes a normalization layer and a nonlinear layer connected in sequence, and the output of each transformation unit is expressed as follows: Z = (ReLU (LayerNorm (GXW)) + X Among them, X represents the output of the previous transformation unit, G represents the adjacency matrix constructed using X, W is the weight matrix of the transformation unit, LayerNorm represents the normalization layer, and ReLU represents the nonlinear layer.

6. The method according to any one of claims 1 to 5, characterized in that: All characters in the text sub-region belonging to the same category are marked as the same detection box, including: Determine the center point of each character image block in all character image blocks under each category, and expand the character image blocks with the average height of a preset number of character image blocks adjacent to each center point as the radius to obtain an initial detection frame corresponding to the character image block under each category; Perform edge smoothing on each initial detection frame to obtain a detection frame for each category.

7. The method according to any one of claims 1, 4 and 5, characterized in that The character image block is obtained through a character image block prediction model, and each text sub-region is obtained through an image segmentation model; The character image block prediction model, the image segmentation model and the graph convolutional network model are jointly trained through the following steps: Acquire a sample data set; wherein the sample data set includes a plurality of sample images to be detected, a plurality of sample images to be detected annotated with character image blocks, a plurality of sample images to be detected annotated with text sub-regions, and a plurality of sample images to be detected annotated with detection frames; Using the sample data set to train a character image block prediction model, an image segmentation model, and a graph convolutional network model until the total loss meets a preset condition, thereby obtaining a trained character image block prediction model, an image segmentation model, and a graph convolutional network model; Among them, the total loss is obtained by weighting the classification loss of the character image block, the coordinate regression loss of the character image block, the segmentation loss of the text sub-region and the classification loss of the text sub-region. The classification loss of the character image block is used to represent the loss between the predicted character image block classification and the real character image block classification, the coordinate regression loss is used to represent the loss between the predicted character image block coordinates and the real character image block coordinates, the segmentation loss is used to represent the loss between the predicted text sub-region and the real text sub-region, and the classification loss of the text sub-region is used to represent the loss between the predicted classification results of each text sub-region and the real classification results of each text sub-region.

8. A text detection device, characterized in that: include: A transceiver module, used for acquiring an image to be detected with text; A detection and segmentation module, used to determine each text sub-region in the image to be detected, and to determine a character image block corresponding to each character in the image to be detected; A classification module, which obtains a topological feature map of the text sub-region according to the image feature matrix and the adjacency matrix of each text sub-region, wherein the image feature matrix is ​​used to describe the image features of each character image block in the text sub-region, the adjacency matrix is ​​used to describe the correlation between each character image block in the text sub-region, and the topological feature map is used to describe the graph features of the text sub-region; and determines the similarity between each two topological feature maps, and classifies two text sub-regions whose similarity meets a preset condition into the same category; A labeling module is used to label all characters in a text sub-region belonging to the same category as the same detection box; Among them, according to the image feature matrix and adjacency matrix of each text sub-region, the topological feature map of the text sub-region is obtained, including: For each text sub-region, obtaining an image feature vector of each character image block in the text sub-region; The corresponding image feature vectors in the text sub-region are combined to obtain the image feature matrix of the text sub-region, and the adjacency matrix of the text sub-region is determined; The adjacency matrix and image feature matrix are input into the trained graph convolutional network model to obtain the topological feature map of the text sub-region.

9. The device according to claim 8, characterized in that The detection and segmentation module is specifically used for: Performing feature extraction on the image to be detected, and performing upsampling processing on the feature extraction result to obtain a preprocessing feature map; The pixels belonging to the text in the preprocessing feature map are divided into regions to obtain various text sub-regions of the image to be detected, and each character in the preprocessing feature map is detected to determine the character image block corresponding to each character in the image to be detected.

10. The device according to claim 8, characterized in that The detection and segmentation module is also used for: For each character image block, a text sub-region whose ratio of the first area to the second area is greater than a preset value is determined as the text sub-region to which the character image block belongs; wherein the first area is the area of ​​the overlapping area between the character image block and the text sub-region, and the second area is the area of ​​the character image block.

11. The device according to claim 8, characterized in that The classification module is specifically used for: Determine a similarity vector between image feature vectors corresponding to every two character image blocks in the text sub-region, and combine the determined multiple similarity vectors to obtain a similarity matrix; The similarity matrix is ​​normalized to obtain an adjacency matrix.

12. A text recognition system, characterized in that: The method comprises a text detection device and a text recognition device as described in any one of claims 8 to 11, wherein: The text recognition device recognizes the character strings in each detection frame according to the detection results marked with the detection frame output by the text detection device.

13. A text detection device, characterized in that: include: at least one processor, and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the at least one processor implements the method according to any one of claims 1 to 7 by executing the instructions stored in the memory.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on a computer, the computer is enabled to execute the method according to any one of claims 1 to 7.

Images