Image recognition, text sorting method and device

Abstract

The embodiment of the present application provides an image recognition and text sorting method and device. The method includes the following steps: identifying multiple text information to be sorted contained in the image to be recognized; determining the reading order of the multiple text information to be sorted according to the features corresponding to each of the multiple text information to be sorted; wherein the features carry semantic features; sorting the multiple text information to be sorted according to the reading order to obtain a sequence of text information to be sorted. The sorting method provided in the embodiment of the present application is applicable to images of any text typesetting format, and has a wide range of application and good applicability.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image recognition and text sorting method and device. Background Art

[0002] With the development of computer technology, image text recognition technology came into being. Through this technology, the device can automatically recognize the text in the image.

[0003] In the prior art, the multiple characters recognized from the image are usually sorted by reading from left to right and from top to bottom by default. However, this simple reading sorting method is only suitable for those pictures with simple layout, and will fail for those pictures with complex layout (for example, column layout, circular layout), because this simple reading order will destroy the original semantic coherence.

[0004] It can be seen that the reading sorting method in the prior art has poor applicability or versatility. Summary of the invention

[0005] In view of the above problems, the present application is proposed to provide an image recognition and text sorting method and device that solves the above problems or at least partially solves the above problems.

[0006] Therefore, in one embodiment of the present application, an image recognition method is provided. The method comprises:

[0007] Identify multiple text information to be sorted contained in the image to be identified;

[0008] Determine the reading order of the plurality of text information to be sorted according to the features corresponding to each of the plurality of text information to be sorted; wherein the features include semantic features;

[0009] The plurality of text information to be sorted are sorted according to the reading order to obtain a sequence of text information to be sorted.

[0010] In another embodiment of the present application, a method for sorting text is provided. The method comprises:

[0011] Get multiple text information to be sorted;

[0012] Determine the reading order of the plurality of text information to be sorted by comprehensively considering the features corresponding to each of the plurality of text information to be sorted and the adjacency relationship between the plurality of text information to be sorted;

[0013] The plurality of first text information are sorted according to the reading order to obtain a first text information sequence.

[0014] In one embodiment of the present application, an image recognition method is provided. The method comprises:

[0015] Identify multiple text information to be sorted contained in the image to be identified;

[0016] Determining the text types of the plurality of text information to be sorted;

[0017] Obtaining the arrangement rule corresponding to the text type;

[0018] According to the arrangement rule, the plurality of text information to be sorted are sorted to obtain a sequence of text information to be sorted.

[0019] In another embodiment of the present application, an electronic device is provided. The device includes: a memory and a processor, wherein:

[0020] The memory is used to store programs;

[0021] The processor is coupled to the memory and is configured to execute the program stored in the memory to:

[0022] Identify multiple text information to be sorted contained in the image to be identified;

[0023] Determine the reading order of the plurality of text information to be sorted according to the features corresponding to each of the plurality of text information to be sorted; wherein the features include semantic features;

[0024] The plurality of text information to be sorted are sorted according to the reading order to obtain a sequence of text information to be sorted.

[0025] In another embodiment of the present application, an electronic device is provided. The device includes: a memory and a processor, wherein:

[0026] The memory is used to store programs;

[0027] The processor is coupled to the memory and is configured to execute the program stored in the memory to:

[0028] Get multiple text information to be sorted;

[0029] Determine the reading order of the plurality of text information to be sorted by comprehensively considering the features corresponding to each of the plurality of text information to be sorted and the adjacency relationship between the plurality of text information to be sorted;

[0030] The plurality of first text information are sorted according to the reading order to obtain a first text information sequence.

[0031] In another embodiment of the present application, an electronic device is provided. The device includes: a memory and a processor, wherein:

[0032] The memory is used to store programs;

[0033] The processor is coupled to the memory and is configured to execute the program stored in the memory to:

[0034] Identify multiple text information to be sorted contained in the image to be identified;

[0035] Determining the text types of the plurality of text information to be sorted;

[0036] Obtaining the arrangement rule corresponding to the text type;

[0037] According to the arrangement rule, the plurality of text information to be sorted are sorted to obtain a sequence of text information to be sorted.

[0038] The technical solution provided in the embodiment of the present application, after identifying multiple text information to be sorted contained in the image to be identified, combines the semantics corresponding to each of the multiple text information to be sorted to sort the multiple text information to be sorted. The sorting method provided in the embodiment of the present application is applicable to images of any text typesetting format, has a wide range of application and good applicability.

[0039] In the technical solution provided in the embodiment of the present application, when reading and sorting multiple text information to be sorted, not only the characteristics of each text information are taken into consideration, but also the adjacency relationship between the multiple text information to be sorted is taken into consideration. In this way, the accuracy of sorting can be effectively improved, and the semantic relevance of the final text information sequence can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1a An example diagram of an image recognition method provided by an embodiment of the present application;

[0042] Figure 1b An example diagram of an image recognition method provided by another embodiment of the present application;

[0043] Figure 1c A flowchart of an image recognition method provided in one embodiment of the present application;

[0044] Figure 2 A flowchart of a text sorting method provided in one embodiment of the present application;

[0045] Figure 3 A structural block diagram of an image recognition device provided in one embodiment of the present application;

[0046] Figure 4 A structural block diagram of a text sorting device provided in another embodiment of the present application;

[0047] Figure 5 A structural block diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0048] Currently, existing image text recognition products all provide a simple reading order from left to right and from top to bottom by default. This simple sorting scheme will not work for images with complex layouts.

[0049] In the prior art, there are two types of methods: one is the typesetting analysis method, and the other is the structured template method.

[0050] Among them, there are two specific implementation methods for typesetting analysis: bottom-up and top-down. The bottom-up method is to use the visual information of text blocks, such as distance, size, color and other features, to synthesize paragraphs through rules; after synthesizing paragraphs, the text sorting within the paragraphs still adopts the reading order from left to right and from top to bottom. The top-down method is to directly use the image segmentation method to divide the image into paragraphs; after completing the segmentation of the paragraphs, the paragraphs are read sequentially in the order from left to right and from top to bottom.

[0051] The typesetting analysis method can handle most document images, that is, images composed of large pieces of neat text. However, it is not effective for complex mixed text and image layouts, such as online advertising images and e-commerce description images.

[0052] The structured template method outputs the text structure according to the configured template rules. It can handle more complex typesetting situations, but it can only be applied to relatively simple typesetting formats, such as invoices, certificates, bank cards, etc. For general situations, it cannot generate semantically related sequences.

[0053] In order to improve the applicability or universality of the reading sorting method, an embodiment of the present application proposes a method for reading sorting text based on semantics.

[0054] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0055] In addition, some of the processes described in the specification, claims and the above-mentioned figures of the present application include multiple operations that appear in a specific order, and these operations may not be executed in the order in which they appear in this article or may be executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0056] Figure 1c A flow chart of an image processing method provided by an embodiment of the present application is shown. The execution subject of the method may be a client or a server. The client may be a hardware with an embedded program integrated on a terminal, an application software installed in the terminal, or a tool software embedded in a terminal operating system, etc., which is not limited in the embodiment of the present application. The terminal may be any terminal device including a mobile phone, a tablet computer, a smart speaker, etc. The server may be a common server, a cloud or a virtual server, etc., which is not specifically limited in the embodiment of the present application.

[0057] like Figure 1c As shown, the method includes:

[0058] 101. Identify, from the image to be identified, a plurality of text information to be sorted contained therein.

[0059] 102. Determine a reading order of the plurality of text information to be sorted according to respective corresponding features of the plurality of text information to be sorted.

[0060] 103. Sort the plurality of text information to be sorted according to the reading order to obtain a sequence of text information to be sorted.

[0061] In the above 101, the above image to be identified refers to an image containing text information, such as: an online advertisement image, an e-commerce description image, an invoice image, a certificate image, a bank card image, etc.

[0062] An image character recognition algorithm, such as OCR (Optical Character Recognition), can be used to recognize the image to be recognized and obtain multiple text information to be sorted contained therein. The specific implementation of the image character recognition algorithm can be referred to the prior art and will not be described in detail here. Generally, the image character recognition algorithm is used to recognize the image to be recognized, and not only multiple text information to be sorted contained therein can be recognized, but also the position of each text information to be sorted in the image to be recognized can be recognized.

[0063] The text information to be sorted refers to a single word or a phrase. For example, "I" is a single word, and "we" is a phrase. In practical applications, the text information to be sorted usually refers to a single word.

[0064] In the above 102, the features include semantic features. A natural language processing algorithm can be used to extract the semantic features corresponding to each of the identified multiple text information to be sorted. The extraction process of the semantic features can be referred to the prior art and will not be described in detail here. In one example, the semantic features corresponding to each text information to be sorted can be used as the features corresponding to each text information to be sorted.

[0065] In another example, the features may also carry visual features. Combining the semantic features and visual features can effectively improve the accuracy of sorting. For example, the text information identified in the image to be identified includes "I", "We" and "You". If we only look at it from the semantic aspect, "I" and "You" can both be ranked before "We" and form "We" and "You" with "We"; if we combine it with the visual aspect, it is found that the fonts of "I" and "We" are inconsistent; the fonts of "You" and "We" are consistent, then it can be determined that "You" is ranked before "We".

[0066] It should be noted that the features involved in the embodiments of the present application may be in vector form.

[0067] In an achievable solution, the reading order of the plurality of text information to be sorted may be determined specifically according to the semantic features and preset syntax corresponding to each of the plurality of text information to be sorted.

[0068] In the above 103, the sequence of text information to be sorted obtained by sorting the plurality of text information to be sorted according to the reading order is semantically coherent.

[0069] The technical solution provided in the embodiment of the present application, after identifying multiple text information to be sorted contained in the image to be identified, combines the semantics corresponding to each of the multiple text information to be sorted to sort the multiple text information to be sorted. The sorting method provided in the embodiment of the present application is applicable to images of any text typesetting format, has a wide range of application and good applicability.

[0070] In the image to be identified, there is a certain association relationship hidden between the multiple text information to be sorted, for example, semantic, visual and / or positional association relationships. These association relationships are very important information. If they can be used, the accuracy of sorting can be effectively improved. Therefore, in one example, the above 102 "determine the reading order of the multiple text information to be sorted according to the corresponding features of the multiple text information to be sorted" can be implemented by the following steps:

[0071] 1021. Determine the adjacency relationship between the multiple text information to be sorted.

[0072] 1022. Determine a reading order of the plurality of text information to be sorted by comprehensively considering the features corresponding to each of the plurality of text information to be sorted and the adjacency relationship between the plurality of text information to be sorted.

[0073] In the above 1021, the adjacency relationship between the multiple text information to be sorted can indicate the association relationship between each of the text information to be sorted and other text information to be sorted.

[0074] The adjacency relationship between the plurality of text information to be sorted may be determined by one or more of the following methods:

[0075] Method 1: For each piece of text information to be sorted, search in the image to be identified for the text information to be sorted within a set range centered on the position of the text information to be sorted in the image to be identified; determine whether the text information to be sorted is adjacent to the text information to be sorted within the set range, and also determine whether the text information to be sorted is not adjacent to the text information to be sorted outside the set range.

[0076] In method one, the adjacency relationship is specifically the adjacency relationship of the relevant positions.

[0077] Method 2: Determine the adjacency relationship between the plurality of text information to be sorted according to the features corresponding to each of the plurality of text information to be sorted.

[0078] In method 2, the following steps can be used to implement it:

[0079] S11. Calculate the correlation between each two pieces of text information to be sorted according to features corresponding to each two pieces of text information to be sorted in the plurality of pieces of text information to be sorted.

[0080] S12: Determine whether each two pieces of text information to be sorted are adjacent to each other based on the correlation between each two pieces of text information to be sorted.

[0081] In the above S11, in an achievable solution, the similarity between the features corresponding to each two text information to be sorted can be directly calculated, and the similarity between the features corresponding to each two text information to be sorted can be used as the correlation between each two text information to be sorted. In practical application, the features can be in vector form; the inner product between the features corresponding to each two text information to be sorted can be used as the similarity.

[0082] In the above S12, a correlation threshold may be set in advance. If the correlation between any two text information to be sorted is greater than or equal to the correlation threshold, the two text information to be sorted are determined to be adjacent.

[0083] The above features carry semantic features, so the adjacency relationship is specifically an adjacency relationship related to semantics.

[0084] In another example, the above features may also carry visual features, so the adjacency relationship is specifically a semantic and visual adjacency relationship.

[0085] When the features also carry visual features, in order to improve the accuracy of correlation calculation, the above S11 "calculating the correlation between each two text information to be sorted according to the features corresponding to each two text information to be sorted in the plurality of text information to be sorted" can be implemented by the following steps:

[0086] A. Calculate the first similarity between the semantic features corresponding to each two pieces of text information to be sorted.

[0087] B. Calculate the second similarity between the visual features corresponding to each two pieces of text information to be sorted.

[0088] C. Determine the correlation between each two pieces of text information to be sorted by combining the first similarity and the second similarity.

[0089] In the above step A, the semantic features are specifically in vector form, and the inner product between the semantic features can be used as the first similarity.

[0090] In the above step B, the visual features are specifically in vector form, and the inner product between the visual features can be used as the second similarity.

[0091] In the above step C, in an achievable solution, the sum of the first similarity and the second similarity may be used as the correlation between each two pieces of text information to be sorted.

[0092] In another feasible solution, the first similarity and the second similarity may be weighted and summed to obtain the correlation between each two text information to be sorted. The weight corresponding to the first similarity and the weight of the second similarity may be set according to actual needs, for example, they may be determined in combination with prior experience, and the embodiment of the present application does not specifically limit this.

[0093] In the above 1021, the adjacency relationship between the multiple text information to be sorted indicates whether every two text information to be sorted are adjacent to each other in the multiple text information to be sorted.

[0094] In the above 1022, the features corresponding to each of the multiple text information to be sorted only contain their own information, and do not contain the relevant information around them (for example: the information of other adjacent text information to be sorted). In other words, the expression of the features is not good enough and not comprehensive enough. In order to improve the feature expression, the features corresponding to each of the multiple text information to be sorted and the adjacency relationship between the multiple text information to be sorted can be combined to update the features corresponding to each of the multiple text information to be sorted, and obtain the updated features corresponding to each. According to the updated features corresponding to each of the multiple text information to be sorted, the reading order of the multiple text information to be sorted is determined. The updated features corresponding to each of the text information to be sorted not only contain its own information, but also contain the information of other adjacent text information to be sorted. The feature expression is becoming more and more abstract and better, which is conducive to improving the accuracy of sorting.

[0095] In one example, the above step 1022 of “combining the features corresponding to each of the plurality of text information to be sorted and the adjacency relationship between the plurality of text information to be sorted, and determining the reading order of the plurality of text information to be sorted” can be implemented by the following steps:

[0096] S21. Construct a graph structure with nodes and edges according to the adjacency relationship between the plurality of text information to be sorted.

[0097] The nodes in the graph structure are used to represent the text information to be sorted; and the edges in the graph structure are used to represent whether the nodes are adjacent.

[0098] S22. Using the features corresponding to each of the multiple text information to be sorted and the graph structure as input of a trained graph convolutional neural network model, and executing the graph convolutional neural network model to obtain the reading order of the multiple text information to be sorted.

[0099] In the above S21, the nodes in the graph structure are used to represent the text information to be sorted; the edges in the graph structure are used to indicate whether the nodes are adjacent. If there is an edge between two nodes, it means that the two nodes are adjacent. The graph structure can be represented by an adjacency matrix. The above graph structure can also be called a topological structure.

[0100] In the above S22, the graph convolutional neural network model can extract features well and effectively improve the sorting accuracy.

[0101] The graph convolutional neural network model is specifically used for:

[0102] S31. According to the features corresponding to each of the plurality of text information to be sorted and the graph structure, a graph convolution operation is performed to obtain updated features corresponding to each of the plurality of text information to be sorted.

[0103] S32: Determine the reading order of the plurality of text information to be sorted according to the updated features corresponding to each of the plurality of text information to be sorted.

[0104] In the above S31, through the graph convolution operation, the graph structure information can be embedded into the features corresponding to the multiple text information to be sorted, and the updated features corresponding to the multiple text information to be sorted can be obtained. In order to obtain higher-dimensional features, multiple graph convolution operations can be performed to obtain features.

[0105] Specifically, the graph convolutional neural network model may include a feature extraction subnetwork; the feature extraction subnetwork may include multiple graph convolutional network layers; each graph convolutional network layer performs a graph convolution operation. The features corresponding to each of the multiple text information to be sorted and the adjacency matrix used to represent the graph structure may be used as the input of the first graph convolutional network layer among the multiple graph convolutional network layers; the adjacency matrix used to represent the graph structure and the features corresponding to each of the multiple text information to be sorted output by the previous graph convolutional network layer among the multiple graph convolutional network layers are used as the input of the next graph convolutional network layer; the features corresponding to each of the multiple text information to be sorted output by the last graph convolutional network layer among the multiple graph convolutional network layers are used as the updated features corresponding to each of the multiple text information to be sorted. It should be noted that the features corresponding to each of the multiple text information to be sorted output by each graph convolutional network layer are different from the features corresponding to each of the multiple text information to be sorted as its input, and the features are more abstract and higher in dimension.

[0106] It should be noted that each graph convolutional network layer contains a trained feature extraction parameter matrix; each graph convolutional network layer will perform the graph convolution operation in combination with the feature extraction parameter matrix. The specific implementation of the graph convolution operation can be designed according to actual needs, and this embodiment does not specifically limit this.

[0107] In an achievable solution, a plurality of text information to be sorted includes a first text information to be sorted, and the first text information to be sorted refers to any one of the plurality of text information to be sorted. Each graph convolutional network layer is specifically used to: for the first text information to be sorted, in combination with the adjacency matrix, determine at least one second text information to be sorted that is adjacent to the first text information to be sorted; respectively splice the features corresponding to the input first text information to be sorted with the features corresponding to the input at least one second text information to be sorted, to obtain at least one first spliced ​​feature; combine at least one first spliced ​​feature into a spliced ​​feature matrix; perform matrix multiplication of the spliced ​​feature matrix with the trained feature extraction parameter matrix to obtain a first matrix; perform pooling processing on the first matrix to obtain the features corresponding to the output first text information to be sorted. The pooling processing may specifically be average pooling or maximum pooling.

[0108] For example: the feature corresponding to the first text information to be sorted is an h-dimensional vector; the feature corresponding to the second text information to be sorted is a j-dimensional vector; then the first concatenated feature is a (h+j)-dimensional vector.

[0109] In an achievable solution, the above-mentioned step of “determining the reading order of the plurality of text information to be sorted according to the updated features corresponding to each of the plurality of text information to be sorted” in S32 can be specifically implemented by the following steps:

[0110] S321, comprehensively analyzing the updated features corresponding to the plurality of text information to be sorted, and calculating a global text information feature as a starting reference feature.

[0111] S322. Calculate the attention weight corresponding to each of at least one of the multiple text information to be sorted that has not been output, based on the reference features and the updated features corresponding to each of the at least one of the multiple text information to be sorted that has not been output.

[0112] S323. Output the text information to be sorted corresponding to the maximum attention weight, and use the updated features corresponding to the text information to be sorted corresponding to the maximum attention weight as new reference features, and continue to execute the above attention weight calculation steps until all the multiple text information to be sorted are output.

[0113] S324: Determine the output order of the plurality of text information to be sorted as the reading order of the plurality of text information to be sorted.

[0114] In practical applications, the graph convolutional neural network model may further include an attention sub-network, and the attention sub-network performs the above steps S321 and S322.

[0115] In the above S321, in one example, the updated features corresponding to the plurality of text information to be sorted may be pooled to obtain a global text information feature. The pooling process may be average pooling or maximum pooling. The global text information feature incorporates the features of the plurality of text information to be sorted.

[0116] Determining the global text information feature can facilitate subsequent searching for the first text information ranked first in the first text information sequence.

[0117] In another example, the graph convolution operation can be continued to further update the updated features corresponding to the multiple text information to be sorted, and obtain further updated features; and then the further updated features corresponding to the multiple text information to be sorted are pooled to obtain global text information features. That is, the attention subnetwork includes a graph convolution network layer and a pooling layer, and the graph convolution network layer can be a fully connected network layer.

[0118] In the above S322, at least one of the plurality of text information to be sorted that has not been output includes the third text information to be sorted. The third text information to be sorted refers to any one of the at least one text information to be sorted. Taking the feature in vector form as an example, the reference feature and the updated feature corresponding to the third text information to be sorted can be spliced ​​to obtain a second spliced ​​feature; the second spliced ​​feature is dot-multiplied with the attention parameter vector to obtain the attention weight corresponding to the third text information to be sorted.

[0119] For example: the reference feature is an n-dimensional vector; the updated feature corresponding to the third text information to be sorted is an m-dimensional vector; then the second concatenated feature is an (n+m)-dimensional vector.

[0120] In the above S323, the text information to be sorted corresponding to the maximum attention weight is output, and the updated features corresponding to the text information to be sorted corresponding to the maximum attention weight are used as new reference features.

[0121] If the plurality of text information to be sorted have not been completely output, the attention weight of the text information to be sorted that has not yet been output is continuously calculated based on the new reference features.

[0122] If all the text information to be sorted is output, the attention weight calculation step is stopped.

[0123] In the above S324, the output order of the plurality of text information to be sorted is also the reading order of the plurality of text information to be sorted.

[0124] In the above embodiment, the text information to be sorted corresponding to the above maximum attention weight is one by default. When there are multiple text information to be sorted, multiple reading orders will appear at this time. That is, the above graph convolutional neural network model can determine the multiple reading orders of the multiple text information to be sorted. The multiple text information to be sorted can be sorted according to multiple reading orders respectively to obtain multiple text information sequences to be sorted. The above method may also include: displaying the multiple text information sequences to be sorted on the user interface for user selection. In addition, the above model can also be optimized according to the target text information sequence to be sorted selected by the user. Specifically, the model can be trained once in combination with the image to be identified and the target text information sequence to be sorted, which means that the optimization of the above model is achieved.

[0125] The training process of the above graph convolutional neural network model is as follows:

[0126] 104. Obtain a sample image and an expected text information sequence corresponding to a plurality of sample text information included in the sample image.

[0127] 105. Optimize the graph convolutional neural network model according to the sample features corresponding to the multiple sample text information, the graph structures corresponding to the multiple sample text information, and the sample text information sequence.

[0128] In the above 105, the sample features corresponding to each of the plurality of sample text information and the graph structures corresponding to the plurality of sample text information can be input into the graph convolutional neural network model to determine the predicted reading order corresponding to each of the plurality of sample text information; the plurality of sample text information can be sorted according to the predicted reading order to obtain a predicted text information sequence; and the graph convolutional neural network model can be optimized according to the difference between the predicted text information sequence and the expected text information sequence. The specific parameter optimization process can be referred to the prior art and will not be described in detail here.

[0129] In practical application, the above method may further include:

[0130] 106. Extract semantic features corresponding to each of the plurality of identified text information to be sorted.

[0131] 107. Extract visual features corresponding to each of the plurality of text information to be sorted.

[0132] 108. The semantic features and visual features corresponding to the plurality of text information to be sorted are integrated to obtain the features corresponding to the plurality of text information to be sorted.

[0133] In the above 106, a natural language processing algorithm may be used to extract the semantic features corresponding to each of the plurality of identified text information to be sorted.

[0134] In one feasible solution, the step of “extracting visual features corresponding to each of the plurality of text information to be sorted” in 107 above can be implemented by the following steps:

[0135] 1071. Determine, from the image to be recognized, sub-image regions where the multiple pieces of text information to be sorted are located, according to respective positions of the multiple pieces of text information to be sorted in the image to be recognized.

[0136] 1072. Extract visual features corresponding to each of the plurality of text information to be sorted according to the sub-image regions where each of the plurality of text information to be sorted is located.

[0137] In the above 1071, the image text information recognition technology is used to identify the multiple text information to be sorted included in the image to be processed and the position of each text information to be sorted in the image to be processed.

[0138] In one example, the sub-image region where the text information to be sorted is located may specifically be a compact rectangular frame region surrounding the text information to be sorted.

[0139] In the above 1072, the visual features may specifically include information such as font, font color, background texture, etc.

[0140] In practical applications, some traditional feature extraction algorithms, such as SIFT (Scale-invariant feature transform), can be used to extract visual features.

[0141] The visual features extracted by the traditional feature extraction algorithm are low-dimensional information, not high-dimensional information, that is, the feature expression is poor. In order to improve the expressiveness of visual features, in one example, a trained convolutional neural network can be used to extract visual features. For example, the sub-image areas where multiple text information to be sorted are located can be respectively input into the trained convolutional neural network to obtain the visual features corresponding to the multiple text information to be sorted. The specific implementation and training process of the convolutional neural network can be referred to the prior art, which will not be repeated here.

[0142] In the above 108, the plurality of text information to be sorted includes the first text information to be sorted; the semantic features and visual features corresponding to the first text information to be sorted may be concatenated to obtain the features corresponding to the first text information to be sorted.

[0143] In practical applications, there are usually multiple text regions in the image to be recognized; the distance between each text region is relatively far. At this time, the text regions in the image to be recognized can be divided, and then the texts within each text region can be sorted. This can not only reduce the subsequent sorting difficulty but also improve the sorting accuracy. Therefore, in one example, the above method may further include:

[0144] 109. Identify multiple text information contained in the image to be recognized and the positions of each text information in the image to be recognized.

[0145] 110. Divide the multiple text information using a clustering algorithm according to the positions of each text information in the image to be recognized, and obtain at least one text information cluster.

[0146] 111. Select multiple text information in one of the at least one text information clusters as the multiple text information to be sorted.

[0147] In the above 109, the OCR algorithm can be specifically used for implementation. For the specific implementation, reference can be made to the corresponding content in the above embodiments, which will not be elaborated here.

[0148] In the above 110, the hierarchical clustering algorithm can be specifically used for the clustering algorithm. The distance between text information in the same text information cluster is less than the distance between text information in different text information clusters. For the specific implementation process of the clustering algorithm, reference can be made to the prior art, which will not be elaborated here.

[0149] In the above 111, select multiple text information in one of the at least one text information clusters as the multiple text information to be sorted.

[0150] In practical applications, the multiple text information in each text information cluster can be sorted respectively.

[0151] Next, Figure 1a An image recognition method provided in an embodiment of the present application will be introduced by way of example:

[0152] Step 1: In the image to be recognized, perform text recognition, and identify that the image to be recognized contains the three texts "year", "festival", and "goods".

[0153] Step 2: Use a natural language processing algorithm to extract the semantic features corresponding to the three texts "year", "festival", and "goods" respectively.

[0154] Step 3: Sort the multiple texts according to the semantic features corresponding to each text, and obtain the text sequence "New Year's Goods Festival".

[0155] Step 4: Output the text sequence to the interface for display.

[0156] The following will combine with Figure 1b to give an example introduction to the image recognition method provided by another embodiment of this application:

[0157] Step a: In the image to be recognized, perform text recognition to recognize that the image to be recognized contains the three characters "year", "festival", and "goods" and their respective positions in the image to be recognized. According to the positions of the characters in the image to be recognized, extract the sub-images where each character is located (i.e., the above-mentioned sub-image regions).

[0158] Step b: Use natural language processing algorithms to respectively extract the semantic features corresponding to the three characters "year", "festival", and "goods"; and through a convolutional neural network CNN, respectively perform visual feature extraction on the sub-images where each character is located to obtain the visual features corresponding to each character.

[0159] Step c: Calculate the correlation between any two characters according to the similarity of visual features and the similarity of semantic features; based on the correlation, construct a graph structure.

[0160] Step d: Concatenate the semantic features and visual features corresponding to each character to obtain the features corresponding to each character; input the graph structure and the features corresponding to each character into the trained graph convolutional neural network model, execute the graph convolutional neural network model, and obtain the reading order of these three characters.

[0161] Step e: Sort these three characters according to the reading order of these three characters to obtain the character sequence "New Year's Goods Festival", and output the character sequence to the interface for display.

[0162] The method relied on by this solution does not depend on templates and can generate a certain sequence in any layout situation, thus having a better scope of application. The evaluation indicators on ordinary horizontal text and columnar layout text in the test set are both higher than 80%.

[0163] Another embodiment of this application provides an image recognition method, and this method includes:

[0164] 501. From the image to be recognized, recognize the multiple text information to be sorted contained therein.

[0165] 502. Determine the text types of the multiple text information to be sorted.

[0166] 503. Obtain the sorting rules corresponding to the text types.

[0167] 504. Sort the multiple text information to be sorted according to the sorting rules to obtain a sequence of text information to be sorted.

[0168] The specific implementation of the above 501 can refer to the corresponding content in the above embodiments, which will not be repeated here.

[0169] In the above 502, different text types usually have different arrangement rules. For example, ancient texts in the image to be processed are usually arranged from top to bottom and from right to left; modern texts in the image to be processed are usually arranged from left to right and from top to bottom.

[0170] In one example, the text type may include an ancient text type and a modern text type.

[0171] In the above 503, the arrangement rule corresponding to the text type can be obtained according to the correspondence between the text type and the arrangement rule established in advance. For the ancient text type, the arrangement rules from top to bottom and from right to left can be configured in advance; for the modern text type, the arrangement rules from left to right and from top to bottom can be configured.

[0172] In the above 504, the plurality of text information to be sorted may be sorted according to the arrangement rule and the positions of the plurality of text information to be sorted in the image to be processed to obtain a sequence of text information to be sorted.

[0173] In this embodiment, the text information is sorted according to different arrangement rules for different text types, which can effectively improve the applicability and accuracy of the sorting scheme.

[0174] It should be noted here that: for the contents of each step not fully described in detail in the method provided in the embodiment of the present application, please refer to the corresponding contents in the above embodiment, which will not be repeated here. In addition, in addition to the above steps, the method provided in the embodiment of the present application may also include other parts or all of the steps in the above embodiments, which can be specifically referred to the corresponding contents in the above embodiments, which will not be repeated here.

[0175] Figure 2 A flow chart of a text sorting method provided by another embodiment of the present application is shown. The execution subject of the method can be a client or a server. Among them, the client can be a hardware with an embedded program integrated on the terminal, or it can be an application software installed in the terminal, or it can be a tool software embedded in the terminal operating system, etc., which is not limited by the embodiment of the present application. The terminal can be any terminal device including a mobile phone, a tablet computer, a smart speaker, etc. Among them, the server can be a common server, a cloud or a virtual server, etc., which is not specifically limited by the embodiment of the present application.

[0176] like Figure 2 As shown, the method includes:

[0177] 201. Obtain multiple text information to be sorted.

[0178] 202. Determine a reading order of the plurality of text information to be sorted by comprehensively considering the features corresponding to each of the plurality of text information to be sorted and the adjacency relationship between the plurality of text information to be sorted.

[0179] 203. Sort the plurality of first text information according to the reading order to obtain a first text information sequence.

[0180] In the above 201, the plurality of text information to be sorted may be identified from the image to be identified; or may be input by the user.

[0181] For example, a text sorting function can be embedded in a tutoring machine for primary school students. When primary school students encounter the problem of "connecting words into sentences" during their homework, they can input multiple words in the question into the tutoring machine. These multiple words are multiple text information to be sorted.

[0182] In the above 202, the adjacency relationship between the multiple text information to be sorted may be a semantic adjacency relationship, or may be an adjacency relationship in other aspects, which is not specifically limited in the embodiment of the present application.

[0183] The specific implementation process of the above 202 and 203 can refer to the corresponding content in the above embodiments, which will not be repeated here.

[0184] In the technical solution provided in the embodiment of the present application, when reading and sorting multiple text information to be sorted, not only the characteristics of each text information are taken into consideration, but also the adjacency relationship between the multiple text information to be sorted is taken into consideration. In this way, the accuracy of sorting can be effectively improved, and the semantic relevance of the final text information sequence can be improved.

[0185] Optionally, the above method may further include:

[0186] 204. Determine an adjacency relationship between the plurality of first text messages according to the features corresponding to each of the plurality of first text messages.

[0187] The specific implementation of the above 204 can refer to the corresponding content in the above embodiments, which will not be repeated here.

[0188] Optionally, the above-mentioned step 202 of “combining the features corresponding to each of the plurality of text information to be sorted and the adjacency relationship between the plurality of text information to be sorted, and determining the reading order of the plurality of text information to be sorted” can be implemented by the following steps:

[0189] 2021. Construct a graph structure with nodes and edges based on the adjacency relationship between the multiple text information to be sorted.

[0190] The nodes in the graph structure are used to represent the text information to be sorted; and the edges in the graph structure are used to represent whether the nodes are adjacent.

[0191] 2022. Use the features corresponding to each of the multiple text information to be sorted and the graph structure as input to a trained graph convolutional neural network model, and execute the graph convolutional neural network model to obtain the reading order of the multiple text information to be sorted.

[0192] The specific implementation process of the above 2021 and 2022 can be found in the corresponding content of the above embodiments, which will not be repeated here.

[0193] It should be noted here that: for the contents of each step not fully described in detail in the method provided in the embodiment of the present application, please refer to the corresponding contents in the above embodiment, which will not be repeated here. In addition, in addition to the above steps, the method provided in the embodiment of the present application may also include other parts or all of the steps in the above embodiments, which can be specifically referred to the corresponding contents in the above embodiments, which will not be repeated here.

[0194] Figure 3 FIG. 1 shows a structural block diagram of an image recognition device provided by an embodiment of the present application. Figure 3 As shown, the device comprises:

[0195] The first recognition module 301 is used to recognize a plurality of text information to be sorted contained in the image to be recognized.

[0196] The first determining module 302 is used to determine the reading order of the plurality of text information to be sorted according to the features corresponding to each of the plurality of text information to be sorted.

[0197] The features include semantic features.

[0198] The first sorting module 303 is used to sort the plurality of text information to be sorted according to the reading order to obtain a sequence of text information to be sorted.

[0199] Optionally, the above device may further include:

[0200] A first acquisition module, used to acquire a sample image and an expected text information sequence corresponding to a plurality of sample text information contained in the sample image;

[0201] The first optimization module is used to optimize the graph convolutional neural network model according to the sample features corresponding to the multiple sample text information, the graph structures corresponding to the multiple sample text information, and the sample text information sequence.

[0202] Optionally, the above device may further include:

[0203] A first extraction module is used to extract the semantic features corresponding to the plurality of identified text information to be sorted; and to extract the visual features corresponding to the plurality of identified text information to be sorted;

[0204] The first fusion module is used to fuse the semantic features and visual features corresponding to the plurality of text information to be sorted, so as to obtain the features corresponding to the plurality of text information to be sorted.

[0205] It should be noted here that: the image recognition device provided in the above embodiment can implement the technical solutions described in the above method embodiments. The specific implementation principles of the above modules or units can be found in the corresponding contents in the above method embodiments, which will not be repeated here.

[0206] Figure 4 FIG. 1 shows a structural block diagram of a text sorting device provided by an embodiment of the present application. Figure 4 As shown, the device comprises:

[0207] The second acquisition module 401 is used to acquire a plurality of text information to be sorted;

[0208] The second determination module 402 is used to determine the reading order of the plurality of text information to be sorted by comprehensively considering the features corresponding to each of the plurality of text information to be sorted and the adjacency relationship between the plurality of text information to be sorted;

[0209] The second sorting module 403 is used to sort the plurality of first text information according to the reading order to obtain a first text information sequence.

[0210] Optionally, the above device may further include:

[0211] The third determination module is used to determine the adjacency relationship between the multiple first text messages according to the features corresponding to each of the multiple first text messages.

[0212] It should be noted here that: the text sorting device provided in the above embodiment can implement the technical solutions described in the above method embodiments. The specific implementation principles of the above modules or units can be found in the corresponding contents in the above method embodiments, which will not be repeated here.

[0213] Another embodiment of the present application provides an image recognition device, the device comprising:

[0214] The second recognition module is used to recognize a plurality of text information to be sorted contained in the image to be recognized.

[0215] The fourth determination module is used to determine the text type of the multiple text information to be sorted.

[0216] The third acquisition module is used to acquire the arrangement rule corresponding to the text type.

[0217] The third sorting module is used to sort the multiple text information to be sorted according to the arrangement rule to obtain a sequence of text information to be sorted.

[0218] It should be noted here that: the image recognition device provided in the above embodiment can implement the technical solutions described in the above method embodiments. The specific implementation principles of the above modules or units can be found in the corresponding contents in the above method embodiments, which will not be repeated here.

[0219] Figure 5 The structural diagram of an electronic device provided by an embodiment of the present application is shown. As shown in the figure, the electronic device includes a memory 1101 and a processor 1102. The memory 1101 can be configured to store various other data to support operations on the electronic device. Examples of these data include instructions for any application or method for operating on the electronic device. The memory 1101 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk or an optical disk.

[0220] The memory is used to store programs;

[0221] The processor 1102 is coupled to the memory 1101 and is used to execute the program stored in the memory 1101 to implement the image recognition method or the text sorting method in the above embodiments.

[0222] Further, if Figure 5 As shown, the electronic device also includes: a communication component 1103, a display 1104, a power component 1105, an audio component 1106 and other components. Figure 5 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 5 Components shown.

[0223] Accordingly, an embodiment of the present application also provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a computer, it can implement the steps or functions of the image recognition method and text sorting method provided in the above embodiments.

[0224] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0225] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0226] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image recognition method, characterized in that: include: Identify a plurality of text information to be sorted contained in the image to be identified; The features and graph structures corresponding to the plurality of text information to be sorted are used as inputs of a trained graph convolutional neural network model, and the graph convolutional neural network model is executed to obtain a reading order of the plurality of text information to be sorted; wherein the features carry semantic features; the graph structure is constructed according to the adjacency relationship between the plurality of text information to be sorted, and the nodes in the graph structure are used to represent the text information to be sorted; and the edges in the graph structure are used to represent whether the nodes are adjacent; The plurality of text information to be sorted are sorted according to the reading order to obtain a sequence of text information to be sorted.

2. The method according to claim 1, characterized in that Also includes: Determine the adjacency relationship between the multiple text information to be sorted.

3. The method according to claim 2, characterized in that Also includes: According to the adjacency relationship between the plurality of text information to be sorted, a graph structure having nodes and edges is constructed.

4. The method according to claim 3, characterized in that The graph convolutional neural network model is used to: According to the features corresponding to the plurality of text information to be sorted and the graph structure, obtaining updated features corresponding to the plurality of text information to be sorted through a graph convolution operation; The reading order of the plurality of text information to be sorted is determined according to the updated features corresponding to each of the plurality of text information to be sorted.

5. The method according to claim 4, characterized in that Determining the reading order of the plurality of text information to be sorted according to the updated features corresponding to each of the plurality of text information to be sorted includes: The updated features corresponding to the plurality of text information to be sorted are integrated to calculate the global text information features as the starting reference features; Calculate the attention weight corresponding to each of the at least one text information to be sorted that has not been output among the multiple text information to be sorted according to the reference feature and the updated feature corresponding to each of the at least one text information to be sorted that has not been output among the multiple text information to be sorted; Output the text information to be sorted corresponding to the maximum attention weight, and use the updated features corresponding to the text information to be sorted corresponding to the maximum attention weight as new reference features, and continue to perform the above attention weight calculation step until all the multiple text information to be sorted are output; The output order of the plurality of text information to be sorted is determined as the reading order of the plurality of text information to be sorted.

6. The method according to any one of claims 3 to 5, characterized in that: Also includes: Acquire a sample image and an expected text information sequence corresponding to a plurality of sample text information contained in the sample image; The graph convolutional neural network model is optimized according to the sample features corresponding to each of the multiple sample text information, the graph structures corresponding to the multiple sample text information, and the expected text information sequence.

7. The method according to any one of claims 2 to 5, characterized in that Determining the adjacency relationship between the plurality of text information to be sorted includes: The adjacency relationship between the multiple text information to be sorted is determined according to the features corresponding to each of the multiple text information to be sorted.

8. The method according to claim 7, characterized in that Determining the adjacency relationship between the plurality of text information to be sorted according to the features corresponding to each of the plurality of text information to be sorted includes: Calculating the correlation between each two pieces of text information to be sorted according to the features corresponding to each two pieces of text information to be sorted in the plurality of pieces of text information to be sorted; Whether the two pieces of text information to be sorted are adjacent is determined according to the correlation between the two pieces of text information to be sorted.

9. The method according to claim 8, characterized in that The features also carry visual features; Calculating the correlation between each two pieces of text information to be sorted according to the features corresponding to each two pieces of text information to be sorted among the plurality of pieces of text information to be sorted includes: Calculating the first similarity between the semantic features corresponding to each two pieces of text information to be sorted; Calculating the second similarity between the visual features corresponding to each two pieces of text information to be sorted; The first similarity and the second similarity are combined to determine the correlation between each two pieces of text information to be sorted.

10. The method according to any one of claims 1 to 5, characterized in that Also includes: Extracting semantic features corresponding to each of the identified plurality of text information to be sorted; Extracting visual features corresponding to each of the plurality of text information to be sorted; The semantic features and visual features corresponding to the plurality of text information to be sorted are integrated to obtain the features corresponding to the plurality of text information to be sorted.

11. The method according to claim 10, characterized in that Extracting visual features corresponding to each of the plurality of text information to be sorted includes: According to the positions of the plurality of text information to be sorted in the image to be identified, determining the sub-image regions where the plurality of text information to be sorted are located from the image to be identified; Visual features corresponding to the plurality of text information to be sorted are extracted respectively according to the sub-image regions where the plurality of text information to be sorted are respectively located.

12. The method according to any one of claims 1 to 5, characterized in that From the image to be identified, multiple text information to be sorted contained therein are identified, including: Identify, from the image to be identified, a plurality of text information contained therein and the position of each text information in the image to be identified; According to the position of each piece of text information in the image to be recognized, the plurality of text information are divided by using a clustering algorithm to obtain at least one text information cluster; A plurality of text information in one of the at least one text information cluster is selected as the plurality of text information to be sorted.

13. A method for sorting characters, characterized in that: include: Get multiple text information to be sorted; Determining the reading order of the plurality of text information to be sorted by comprehensively considering the features corresponding to each of the plurality of text information to be sorted and the adjacency relationship between the plurality of text information to be sorted, including: constructing a graph structure with nodes and edges according to the adjacency relationship between the plurality of text information to be sorted; the nodes in the graph structure are used to represent the text information to be sorted; the edges in the graph structure are used to indicate whether the nodes are adjacent; using the features corresponding to each of the plurality of text information to be sorted and the graph structure as inputs of a trained graph convolutional neural network model, executing the graph convolutional neural network model, so as to obtain the reading order of the plurality of text information to be sorted; The plurality of text information to be sorted are sorted according to the reading order to obtain a first text information sequence.

14. The method according to claim 13, characterized in that Also includes: The adjacency relationship between the multiple text information to be sorted is determined according to the features corresponding to each of the multiple text information to be sorted.

15. An electronic device, characterized in that: include: A memory and a processor, wherein The memory is used to store programs; The processor is coupled to the memory and is configured to execute the program stored in the memory to: Identify a plurality of text information to be sorted contained in the image to be identified; The features and graph structures corresponding to the plurality of text information to be sorted are used as inputs of a trained graph convolutional neural network model, and the graph convolutional neural network model is executed to obtain a reading order of the plurality of text information to be sorted; wherein the features carry semantic features; the graph structure is constructed according to the adjacency relationship between the plurality of text information to be sorted, and the nodes in the graph structure are used to represent the text information to be sorted; and the edges in the graph structure are used to represent whether the nodes are adjacent; The plurality of text information to be sorted are sorted according to the reading order to obtain a sequence of text information to be sorted.

16. An electronic device, characterized in that: include: A memory and a processor, wherein The memory is used to store programs; The processor is coupled to the memory and is configured to execute the program stored in the memory to: Get multiple text information to be sorted; Determining the reading order of the plurality of text information to be sorted by comprehensively considering the features corresponding to each of the plurality of text information to be sorted and the adjacency relationship between the plurality of text information to be sorted, including: constructing a graph structure with nodes and edges according to the adjacency relationship between the plurality of text information to be sorted; the nodes in the graph structure are used to represent the text information to be sorted; the edges in the graph structure are used to indicate whether the nodes are adjacent; using the features corresponding to each of the plurality of text information to be sorted and the graph structure as inputs of a trained graph convolutional neural network model, executing the graph convolutional neural network model, so as to obtain the reading order of the plurality of text information to be sorted; The plurality of text information to be sorted are sorted according to the reading order to obtain a first text information sequence.

Images