Information processing method and device, and computer readable storage medium
By performing character recognition and feature vector processing on the target image, character association relationships in table cells are determined, which solves the problem of low accuracy in table cells recognition in the prior art, and achieves higher recognition accuracy.
Patent Information
- Application Number
- CN202210441425.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-25
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-04-25
AI Technical Summary
In the prior art, the method that relies on character semantics has low accuracy in table cell recognition, which is difficult to meet the requirements of industrial use.
By obtaining the target image, character recognition processing is performed to obtain text information and coordinate information, input it to the trained language representation model to obtain feature vectors, and relationship correlation processing is performed to determine the correlation relationship between characters, improving the accuracy of table cell recognition sex.
It improves the accuracy of table cell recognition, can effectively determine whether characters are in the same table cell, and meets the accuracy requirements for industrial use.
Smart Images

Figure CN114973262B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an information processing method and device thereof, and a computer-readable storage medium. Background Art
[0002] At present, digitizing paper form data can facilitate archiving and maintenance of form data. In related technologies, optical character recognition (OCR) technology is generally used to recognize the text information of characters in an image, and then the characters are analyzed based on their semantics to determine whether they belong to the same table unit. However, this method that relies on the semantics of characters has a low accuracy rate and is difficult to meet the requirements of industrial use. How to improve the accuracy of table unit recognition is a major technical problem that needs to be solved urgently. Summary of the invention
[0003] The purpose of the present invention is to solve at least one of the technical problems existing in the prior art and to provide an information processing method capable of improving the recognition accuracy of table cells.
[0004] In a first aspect, an embodiment of the present invention provides an information processing method, including:
[0005] Acquire a target image, wherein the target image includes a first target character and a second target character;
[0006] Performing character recognition processing on the target image to obtain first text information of the first target character, first coordinate information of the first target character in the target image, second text information of the second target character, and second coordinate information of the second target character in the target image, wherein the first coordinate information includes first upper left corner coordinate information of a first character box corresponding to the first target character and first center point coordinate information of the first character box, and the second coordinate information includes second upper left corner coordinate information of a second character box corresponding to the second target character and second center point coordinate information of the second character box;
[0007] Inputting the first text information, the second text information, the first upper left corner coordinate information, the first center point coordinate information, the second upper left corner coordinate information, and the second center point coordinate information into a trained language representation model to obtain a first feature vector of the first target character and a second feature vector of the second target character;
[0008] Performing relationship association processing on the first eigenvector and the second eigenvector to obtain a target relationship matrix;
[0009] It is determined whether the first target character and the second target character conform to a preset association relationship according to the target relationship matrix.
[0010] According to the information processing method of the embodiment of the present invention, there is at least the following beneficial effect: acquiring a target image, the target image including a first target character and a second target character; performing character recognition processing on the target image to obtain first text information of the first target character, first coordinate information of the first target character in the target image, second text information of the second target character and second coordinate information of the second target character in the target image, wherein the first coordinate information includes first upper left corner coordinate information of a first character box corresponding to the first target character and first center point coordinate information of the first character box, and the second coordinate information includes second upper left corner coordinate information of a second character box corresponding to the second target character and second center point coordinate information of the second character box; inputting the first text information, the second text information, the first upper left corner coordinate information, the first center point coordinate information, the second upper left corner coordinate information and the second center point coordinate information into a trained language representation model to obtain a first feature vector of the first target character and a second feature vector of the second target character; performing relationship association processing on the first feature vector and the second feature vector to obtain a target relationship matrix; determining whether the first target character and the second target character meet a preset association relationship according to the target relationship matrix. According to the technical solution of the present invention, whether the first target character and the second target character conform to a preset association relationship is determined based on the first text information, the second text information, the first upper left corner coordinate information, the first center point coordinate information, the second upper left corner coordinate information and the second center point coordinate information, thereby determining whether the first target character and the second target character are in the same table unit, thereby improving the accuracy of table unit recognition.
[0011] In a second aspect, an embodiment of the present invention provides a data processing device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the information processing method as described in the first aspect when executing the computer program.
[0012] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the information processing method as described in the first aspect.
[0013] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] The accompanying drawings are used to provide a further understanding of the technical solution of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the technical solution of the present invention and do not constitute a limitation on the technical solution of the present invention.
[0015] The present invention is further described below in conjunction with the accompanying drawings and embodiments;
[0016] Figure 1 is a flowchart of the steps of the information processing method provided by an embodiment of the present invention;
[0017] Figure 2 yes Figure 1 Flow chart of the specific method of step S120;
[0018] Figure 3 yes Figure 1 Flow chart of the specific method of step S130;
[0019] Figure 4 yes Figure 1 Flow chart of the specific method of step S140;
[0020] Figure 5 yes Figure 4 Flow chart of the specific method of step S142;
[0021] Figure 6 is a flow chart of an information processing method provided by another embodiment of the present invention;
[0022] Figure 7 yes Figure 1 Flow chart of the specific method of step S150;
[0023] Figure 8 is a flow chart of an information processing method provided by another embodiment of the present invention;
[0024] Fig. 9 is a schematic diagram of character information obtained by character recognition processing in an information processing method provided by another embodiment of the present invention;
[0025] Fig.10 It is a schematic diagram of a target relationship matrix in an information processing method provided by another embodiment of the present invention. DETAILED DESCRIPTION
[0026] This section will describe in detail the specific embodiments of the present invention. The preferred embodiments of the present invention are shown in the accompanying drawings. The purpose of the accompanying drawings is to supplement the description of the text part of the specification with graphics, so that people can intuitively and vividly understand each technical feature and the overall technical solution of the present invention, but it cannot be understood as a limitation on the scope of protection of the present invention.
[0027] In the description of the present invention, if there is a description of first and second, it is only for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.
[0028] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.
[0029] An embodiment of the present invention provides an information processing method and device thereof, and a computer-readable storage medium, comprising acquiring a target image, the target image comprising a first target character and a second target character; performing character recognition processing on the target image to obtain first text information of the first target character, first coordinate information of the first target character in the target image, second text information of the second target character, and second coordinate information of the second target character in the target image, wherein the first coordinate information comprises first upper left corner coordinate information of a first character box corresponding to the first target character and first center point coordinate information of the first character box, and the second coordinate information comprises second upper left corner coordinate information of a second character box corresponding to the second target character and second center point coordinate information of the second character box; inputting the first text information, the second text information, the first upper left corner coordinate information, the first center point coordinate information, the second upper left corner coordinate information, and the second center point coordinate information into a trained language representation model to obtain a first feature vector of the first target character and a second feature vector of the second target character; performing relationship association processing on the first feature vector and the second feature vector to obtain a target relationship matrix; and determining whether the first target character and the second target character meet a preset association relationship according to the target relationship matrix. According to the technical solution of the present invention, whether the first target character and the second target character conform to a preset association relationship is determined based on the first text information, the second text information, the first upper left corner coordinate information, the first center point coordinate information, the second upper left corner coordinate information and the second center point coordinate information, thereby improving the accuracy of table cell recognition.
[0030] The embodiments of the present invention are further described below in conjunction with the accompanying drawings.
[0031] Reference Figure 1A first aspect of the present invention provides an information processing method, which may include but is not limited to steps S110 to S150.
[0032] Step S110: Acquire a target image, where the target image includes a first target character and a second target character.
[0033] In this step, the target image refers to the image to be processed, and the target image may be input by the user. In an optional embodiment, the target image is an image of the insurance policy obtained by photographing or scanning. The first target character and the second target character refer to characters contained in the target image, and the first target character and the second target character are different characters in the target image.
[0034] It can be understood that the target image is acquired in order to process the target image in subsequent steps to obtain the first text information of the first target character, the first coordinate information of the first target character in the target image, the second text information of the second target character and the second coordinate information of the second target character in the target image.
[0035] Step S120: Perform character recognition processing on the target image to obtain first text information of the first target character, first coordinate information of the first target character in the target image, second text information of the second target character, and second coordinate information of the second target character in the target image, wherein the first coordinate information includes the first upper left corner coordinate information of the first character box corresponding to the first target character and the first center point coordinate information of the first character box, and the second coordinate information includes the second upper left corner coordinate information of the second character box corresponding to the second target character and the second center point coordinate information of the second character box.
[0036] In this step, the first character frame corresponding to the first target character refers to the character frame that frames the first character in the character recognition process, and the second character frame corresponding to the second target character refers to the character frame that frames the second character in the character recognition process. Character recognition can be a technology for character recognition of an image in any form in the relevant technology. Character recognition processing of the target image refers to recognizing the character information in the image, and retaining the text information of the character and the character frame information corresponding to the character. In an optional embodiment, the recognition result is as follows: Fig. 9 As shown, each row represents all the data corresponding to a target character, the first eight digits represent the coordinate information of the four endpoints of the character box, the ninth digit represents the character recognition confidence, and the tenth element is the text information of the recognized character.
[0037] It should be noted that, after character recognition processing, the target image can obtain the text information of the characters and the information of each endpoint of the character box corresponding to the character. Therefore, the first coordinate information and the second coordinate information can be obtained, wherein the first coordinate information includes the first upper left corner coordinate information of the first character box corresponding to the first target character and the first center point coordinate information of the first character box, and the second coordinate information includes the second upper left corner coordinate information of the second character box corresponding to the second target character and the second center point coordinate information of the second character box.
[0038] It should also be noted that the first target character and the second target character represent different target characters, and the first text information, the first coordinate information, the second text information and the second coordinate information are obtained to facilitate obtaining the first feature vector and the second feature vector in subsequent steps.
[0039] Step S130: Input the first text information, the second text information, the first upper left corner coordinate information, the first center point coordinate information, the second upper left corner coordinate information and the second center point coordinate information into the trained language representation model to obtain the first feature vector of the first target character and the second feature vector of the second target character.
[0040] In this step, the first text information refers to the text information corresponding to the first target character, and the second text information refers to the text information corresponding to the second target character; the language representation model can be any form of language representation model in the relevant technology; the first upper left corner coordinate information refers to the coordinate information of the upper left corner of the first character frame, the first center point coordinate information refers to the coordinate information of the center point of the first character frame, the second upper left corner coordinate information refers to the coordinate information of the upper left corner of the second character frame, and the second center point coordinate information refers to the coordinate information of the center point of the second character frame.
[0041] It should be noted that the language representation model can be a pre-trained BERT model or a LayoutLM model. In an optional implementation, feature extraction is performed through the LayoutLM model to obtain a first feature vector and a second feature vector. The BASE model in the LayoutLM model has the same structure as the BERT model, that is, a 12-layer Transformer with 768 neurons (hidden size) and 12 information extraction heads (attention heads), containing parameters of about 113M in size. It is worth noting that Transformer refers to a model in the related technology that uses the attention mechanism to improve the model training speed.
[0042] It should also be noted that the first feature vector corresponds to the first target character, and the second feature vector corresponds to the second target character. After the first text information, the second text information, the first upper left corner coordinate information, the first center point coordinate information, the second upper left corner coordinate information and the second center point coordinate information are passed through the language representation model, the first feature vector and the second feature vector are obtained. The language representation model can be used as a feature extractor.
[0043] It should also be noted that the number of characters corresponding to the first text information and the second text information is not specifically limited herein. In an optional implementation, the number of characters corresponding to the first text information and the second text information is one.
[0044] It should also be noted that the first feature vector and the second feature vector are obtained in order to determine whether there is a preset association relationship between the first target character and the second target in the subsequent steps.
[0045] Step S140: performing relationship association processing on the first eigenvector and the second eigenvector to obtain a target relationship matrix.
[0046] In this step, the target relationship matrix refers to a relationship matrix that can determine whether a preset relationship exists between the first target character and the second target character. Relationship association processing refers to performing a classification task based on the first eigenvector and the second eigenvector to obtain the target relationship matrix. In an optional embodiment, relationship association processing can be performing association processing on the first eigenvector and the second eigenvector and inputting them into the fully connected layer to obtain the target relationship matrix.
[0047] It should be noted that the target relationship matrix is obtained in order to determine whether the first target character and the second target character conform to the preset association relationship in the subsequent steps.
[0048] Step S150: determining whether the first target character and the second target character conform to a preset association relationship according to the target relationship matrix.
[0049] In this step, since the target relationship matrix is obtained by performing association processing on the first eigenvector and the second eigenvector, and the first eigenvector and the second eigenvector are obtained by the first text information, the first upper left corner coordinate information, the first center point coordinate information, the second text information, the second upper left corner coordinate information and the second center point coordinate information, whether the preset association relationship is met refers to whether there is a semantic association and a spatial association between the first target character and the second target character, wherein the semantic association refers to the association between the first text information and the second text information, and the spatial association refers to the association relationship between the first coordinate information and the second coordinate information.
[0050] It is understandable that when there is a preset association relationship between the first target character and the second target character, that is, the first target character and the second target character are in the same table unit. This technical solution of obtaining a feature vector based on the text information of the character, the coordinate information of the upper left corner of the character box corresponding to the character, and the coordinate information of the center point of the character box corresponding to the character, and then determining the association relationship between the characters based on the feature vector can improve the accuracy of table unit recognition.
[0051] In this embodiment, through steps S110 to S150, a target image is acquired, and the target image includes a first target character and a second target character; character recognition processing is performed on the target image to obtain first text information of the first target character, first coordinate information of the first target character in the target image, second text information of the second target character, and second coordinate information of the second target character in the target image, wherein the first coordinate information includes the first upper left corner coordinate information of the first character box corresponding to the first target character and the first center point coordinate information of the first character box, and the second coordinate information includes the second upper left corner coordinate information of the second character box corresponding to the second target character and the second center point coordinate information of the second character box; the first text information, the second text information, the first upper left corner coordinate information, the first center point coordinate information, the second upper left corner coordinate information, and the second center point coordinate information are input into the trained language representation model to obtain a first feature vector of the first target character and a second feature vector of the second target character; relationship association processing is performed on the first feature vector and the second feature vector to obtain a target relationship matrix; and it is determined whether the first target character and the second target character meet a preset association relationship according to the target relationship matrix. According to the technical solution of the present invention, whether the first target character and the second target character conform to a preset association relationship is determined based on text information, upper left corner coordinate information and center point coordinate information, thereby determining whether the first target character and the second target character are in the same table unit, thereby improving the accuracy of table unit recognition.
[0052] In one embodiment, if Figure 2 As shown, step S120 is further described, which includes but is not limited to steps S121 to S123.
[0053] Step S121: Perform character recognition processing on the target image to obtain first text information of the first target character, first endpoint coordinate information of the first character box corresponding to the first target character, second text information of the second target character, and second endpoint coordinate information of the second character box corresponding to the second target character, wherein the first endpoint coordinate information includes first upper-left corner coordinate information, and the second endpoint coordinate information includes second upper-left corner coordinate information.
[0054] In this step, the character recognition process may be a conventional OCR character recognition process technology in the related art, or a character recognition process technology of a machine learning algorithm in the related art, which is not limited here. The first character box and the second character box may both be rectangular boxes, the first endpoint coordinate information refers to the coordinate information of the four vertices of the first character box corresponding to the first target character, and the second endpoint coordinate information refers to the coordinate information of the four vertices of the second character box corresponding to the first target character.
[0055] It can be understood that the first upper left corner coordinate information refers to the coordinate information of the endpoint of the upper left corner of the first character frame, and the second upper left corner coordinate information refers to the coordinate information of the endpoint of the upper left corner of the second character frame. The first endpoint coordinate information and the second endpoint coordinate information are obtained in order to obtain the first center point coordinate information and the second center point coordinate information in subsequent steps.
[0056] Step S122: Obtaining first center point coordinate information according to the first endpoint coordinate information.
[0057] In this step, the first endpoint coordinate information includes the coordinate information of each endpoint of the first character box. In an optional embodiment, the first character box is a rectangular box, and the first center point coordinate information refers to the coordinate information corresponding to the position of the center point of the rectangular box. Therefore, the coordinate information corresponding to the horizontal coordinate of the rectangle can be determined according to the first upper left corner coordinate information and the first upper right corner coordinate information, and the coordinate information corresponding to the vertical coordinate of the rectangle can be determined according to the first lower left corner coordinate information and the first lower right corner coordinate information, thereby obtaining the first center point coordinate information.
[0058] Step S123: Obtain the second center point coordinate information according to the second endpoint coordinate information.
[0059] In this step, the second endpoint coordinate information includes the coordinate information of each endpoint of the second character box. In an optional embodiment, the second character box is a rectangular box, and the second center point coordinate information refers to the coordinate information corresponding to the position of the center point of the rectangular box. Therefore, the coordinate information corresponding to the horizontal coordinate of the rectangle can be determined according to the second upper left corner coordinate information and the second upper right corner coordinate information, and the coordinate information corresponding to the vertical coordinate of the rectangle can be determined according to the second lower left corner coordinate information and the second lower right corner coordinate information, so as to obtain the second center point coordinate information.
[0060] In this embodiment, through steps S121 to S123, character recognition processing is performed on the target image to obtain the first text information of the first target character, the first endpoint coordinate information of the first character box corresponding to the first target character, the second text information of the second target character, and the second endpoint coordinate information of the second character box corresponding to the second target character, wherein the first endpoint coordinate information includes the first upper left corner coordinate information, and the second endpoint coordinate information includes the second upper left corner coordinate information; the first center point coordinate information is obtained according to the first endpoint coordinate information; and the second center point coordinate information is obtained according to the second endpoint coordinate information. According to the technical solution of the present invention, the coordinate information of the character box corresponding to the character is retained through character recognition processing to obtain the first upper left corner coordinate information, the first center point coordinate information, the second upper left corner coordinate information, and the second center point coordinate information, and fully utilize the spatial information of the first target character and the second target character, thereby determining whether the first target character and the second target character have a preset association relationship, so as to achieve the purpose of improving the accuracy of table recognition.
[0061] In one embodiment, if Figure 3 As shown, the steps before step S130 are further explained, and the steps before this step may include but are not limited to steps S310 to S320.
[0062] Step S310: obtaining the maximum coordinate value of the first character and the second character in the target image and the minimum coordinate value of the first character and the second character in the target image according to the first endpoint coordinate information and the second endpoint coordinate information.
[0063] In this step, the first endpoint coordinate information includes the coordinate information of the endpoint of the first character box, and the second endpoint coordinate information includes the coordinate information of the endpoint of the second character box. According to the information of each endpoint, the corresponding maximum coordinate value and minimum coordinate value of the first character box and the second character box in the target image can be obtained.
[0064] It can be understood that the maximum coordinate value and the minimum coordinate value are only the relative maximum coordinate value and the minimum coordinate value corresponding to the first character box and the second character box, which do not depend on the size of the input target image and can reduce the limitation on the size of the target image, thereby achieving the purpose of improving the versatility of the feature.
[0065] Step S320: Map the first upper left corner coordinate information, the first center point coordinate information, the second upper left corner coordinate information and the second center point coordinate information within a preset numerical range according to the maximum coordinate value and the minimum coordinate value to obtain the standardized first upper left corner coordinate information, the standardized first center point coordinate information, the standardized second upper left corner coordinate information and the standardized second center point coordinate information.
[0066] In this step, the preset numerical range refers to the preset range value, and the first upper left corner coordinate information, the first center point coordinate information, the second upper left corner coordinate information and the second center point coordinate information are mapped to the preset numerical range, which can standardize the coordinate information and facilitate subsequent calculation and processing.
[0067] In an optional embodiment, the preset numerical range refers to 0 to 1000, so that the maximum coordinate value is (x_max, y_max), and the minimum coordinate value is (x_min, y_min). For any coordinate information (x, y), the standardized x_standardized = (x-x_min) / (x_max-x_min)*1000, and the standardized y_standardized = (y-y_min) / (y_max-y_min)*1000.
[0068] It should be noted that inputting the standardized first upper left corner coordinate information, the standardized first center point coordinate information, the standardized second upper left corner coordinate information and the standardized second center point coordinate information into the trained language representation model can reduce the impact of the size of the target image on the coordinate information and improve the rationality of the obtained feature vector.
[0069] In this embodiment, through steps S310 to S320, the maximum coordinate values of the first character and the second character in the target image and the minimum coordinate values of the first character and the second character in the target image are obtained according to the first endpoint coordinate information and the second endpoint coordinate information; the first upper left corner coordinate information and the first center point coordinate information are mapped to a preset numerical range according to the first maximum coordinate value and the first minimum coordinate value, and the second upper left corner coordinate information and the second center point coordinate information are mapped to a preset numerical range according to the second maximum coordinate value and the second minimum coordinate value, so as to obtain the standardized first upper left corner coordinate information, the standardized first center point coordinate information, the standardized second upper left corner coordinate information and the standardized second center point coordinate information. According to the technical solution of the present invention, the first coordinate information and the second coordinate information can be standardized, and the relative position is adopted, which can reduce the influence of the size of the target image on the coordinate information, thereby improving the rationality of the extracted first feature vector and the second feature vector.
[0070] In one embodiment, if Figure 4 As shown, step S140 is further described, which includes but is not limited to steps S141 to S143.
[0071] Step S141: Obtain a first eigenvector and a second eigenvector.
[0072] In this step, the first feature vector and the second feature vector refer to the feature vectors obtained by the trained language representation model for the first text information, the second text information, the first upper left corner coordinate information, the first center point coordinate information, the second upper left corner coordinate information, and the second center point coordinate information. It can be understood that obtaining the first feature vector and the second feature vector is to facilitate obtaining the target relationship matrix in the subsequent steps.
[0073] Step S142: Add the first feature vector and the second feature vector to obtain a representation relationship matrix.
[0074] In this step, by adding the first feature vector and the second feature vector, a representation relationship matrix is obtained. The representation relationship matrix combines the first feature vector and the second feature vector, and can obtain the correlation relationship between the first feature vector and the second feature vector, which is convenient for obtaining the target relationship matrix in the subsequent steps.
[0075] It can be understood that both the first target character and the second target character can contain only a single character, and there can be multiple first target characters and multiple second target characters. Adding the first feature vector and the second feature vector means adding the pairwise different first feature vectors and the pairwise different second feature vectors. That is to say, for different feature vectors, addition is required. Refer to Fig.10 , each character can respectively correspond to a feature vector, and the feature vector corresponding to one character is added to the feature vectors corresponding to other characters respectively. For example, for the feature vector corresponding to "Ping", it needs to be added to the feature vectors corresponding to "An", "Fu", "2", "Yuan", "e", "Sheng", "Bao", "3", and "Yuan" respectively.
[0076] Step S143: Input the representation relationship matrix into the fully connected layer to obtain the target relationship matrix.
[0077] In this step, the fully connected layer refers to any form of fully connected layer in the related art. The fully connected layer plays a role in classification. Inputting the representation relationship matrix into the fully connected layer to obtain the target relationship matrix. Among them, the target relationship matrix can be an identity matrix. In an optional implementation manner, the element in the m-th row and n-th column of the target relationship matrix represents the correlation relationship between the m-th character and the n-th character. Among them, the number of rows of the target relationship matrix is greater than or equal to m, and the number of columns of the target relationship matrix is greater than or equal to n. When there is a correlation relationship between the m-th character and the n-th character, the value of the target relationship matrix in the m-th row and n-th column is 1, where the number of rows of the target relationship matrix is greater than or equal to m, and the number of columns of the target relationship matrix is greater than or equal to n.
[0078] In this embodiment, through steps S141 to S143, the first eigenvector and the second eigenvector are obtained; the first eigenvector and the second eigenvector are added respectively to obtain a representation relationship matrix; the representation relationship matrix is input into the fully connected layer to obtain a target relationship matrix. According to the technical solution of the present invention, the first eigenvector and the second eigenvector are fused, and then classified through the fully connected layer to obtain the target relationship matrix, which can achieve the purpose of being able to determine whether there is a preset association relationship between the first target character and the second target character.
[0079] In one embodiment, if Figure 5 As shown, step S142 is further described, which includes but is not limited to steps S1421 to S1423.
[0080] Step S1421: merge the first eigenvector and the second eigenvector to form an intermediate matrix.
[0081] In this step, merging the first eigenvector and the second eigenvector to form an intermediate matrix refers to any method of merging vectors into a matrix in the related art to form a matrix, and the intermediate matrix includes the first eigenvector and the second eigenvector.
[0082] It can be understood that the intermediate matrix is formed to facilitate obtaining the representation relationship matrix in subsequent steps.
[0083] Step S1422: Obtain a preset matching matrix, where the matching matrix is a unit matrix, the dimension of the intermediate matrix is n×1, and the dimension of the matching matrix is The matching matrix is used to add the vectors in the intermediate matrix together.
[0084] In this step, the preset matching matrix refers to a matrix that can be used to add the vectors in the intermediate matrix in pairs. The matching matrix is a unit matrix. In an optional embodiment, each row in the matching matrix has two 1s, and the other positions in the row are filled with zeros. When the matching matrix is multiplied by the intermediate matrix, the position of 1 in the matching matrix multiplies the eigenvector information by 1, and the eigenvector information multiplied by 1 is retained.
[0085] It is understandable that the matching matrix needs to be multiplied by the intermediate matrix, so when the dimension of the intermediate matrix is n×1, the dimension of the matching matrix is Where n is any natural number.
[0086] Step S1423: multiply the matching matrix and the intermediate matrix to obtain a representation relationship matrix.
[0087] In this step, the matching matrix is multiplied by the intermediate matrix. Since the matching matrix is a unit matrix, when the eigenvector in the intermediate matrix is multiplied by 0 in the matching matrix, the eigenvector is not calculated, and when the eigenvector in the intermediate matrix is multiplied by 1 in the matching matrix, the eigenvector is retained, thereby achieving the purpose of adding the eigenvectors in the intermediate matrix in pairs. In addition, this technical solution can reduce the loss of memory resources caused by the traversal required for the pairwise multiplication and improve the computing efficiency.
[0088] In this embodiment, through steps S1421 to S1423, the first eigenvector and the second eigenvector are combined to form an intermediate matrix; a preset matching matrix is obtained, wherein the matching matrix is a unit matrix, and the dimension of the intermediate matrix is set to n×1, then the dimension of the matching matrix is The matching matrix is used to add the vectors in the intermediate matrix in pairs; the matching matrix is multiplied by the intermediate matrix to obtain the representation relationship matrix. According to the technical solution of the present invention, the purpose of adding the eigenvectors in the intermediate matrix in pairs is achieved through a preset matching matrix, thereby obtaining the representation relationship matrix, which improves the calculation efficiency.
[0089] It is worth noting that the addition of two vectors in the intermediate matrix can also be done in a traversal manner, by traversing the eigenvectors in the intermediate matrix and adding two eigenvectors in the intermediate matrix to obtain a matrix representing the relationship.
[0090] In one embodiment, if Figure 6 As shown, the steps before step S130 are further explained, and before step S130 there are including but not limited to steps S610 to S620.
[0091] Step S610: Obtaining the character distance between the first target character and the second target character according to the first center point coordinate information and the second center point coordinate information.
[0092] In this step, the character distance refers to the distance between the first target character and the second target character. In an optional embodiment, the first target character and the second target character may include multiple characters, and the character distance refers to the distance between the center point of the first character frame and the center point of the second character frame.
[0093] Step S620: When the character distance is greater than a preset distance threshold, it is determined that the first target character and the second target character do not conform to a preset association relationship.
[0094] In this step, the preset distance threshold can be preset manually, and the preset distance threshold can be set according to a preset association relationship. In an optional implementation, the insurance policy is used as the target image, and the preset distance threshold is 9, indicating that the distance between each character in the same table unit will not be greater than 9. When the character distance value between the first target character and the second target character is greater than 9, it can be directly determined that the first target character and the second target character do not meet the preset association relationship, that is, the first target character and the second target character are not in the same table unit.
[0095] In this embodiment, through steps S610 to S620, the first center point coordinate information and the second center point coordinate information are obtained; the character distance between the first target character and the second target character is obtained according to the first center point coordinate information and the second center point coordinate information; when the character distance is greater than a preset distance threshold, it is determined that the first target character and the second target character do not conform to the preset association relationship. According to the technical solution of the present invention, when the character distance is greater than the preset distance threshold, it is directly determined that the first target character and the second target character do not conform to the preset association relationship, which can reduce the size of the obtained target relationship matrix, thereby reducing the amount of calculation, and achieving the purpose of improving calculation efficiency.
[0096] It is worth noting that when the character distance between the first target character and the second target character is equal to 0, that is, the first target character is adjacent to the second target character, it is also necessary to consider whether the first text information and the second text information conform to the preset association relationship. It is not possible to directly determine whether there is an association relationship between the first target character and the second target character.
[0097] In one embodiment, if Figure 7 As shown, step S150 is further described, and the step includes but is not limited to step S151.
[0098] Step S151: According to the first row number corresponding to the first target character in the target relationship matrix and the first column number corresponding to the second target character in the target relationship matrix, obtain the first relationship value corresponding to the first row number and the first column number of the target relationship matrix, and determine whether the first target character and the second target character meet the preset association relationship according to the first relationship value; or, according to the second column number corresponding to the first target character in the target relationship matrix and the second row number corresponding to the second target character in the target relationship matrix, obtain the second relationship value corresponding to the second row number and the second column number of the target relationship matrix, and determine whether the first target character and the second target character meet the preset association relationship according to the second relationship value.
[0099] In this step, the value in the m-th row and n-th column of the target relationship matrix represents whether there is a preset association relationship between the m-th character and the n-th character. The target relationship matrix can be an identity matrix. When the value in the matrix is 0, it means not relevant, and when the value is 1, it means relevant, where m and n are arbitrary natural numbers.
[0100] It can be understood that in an alternative embodiment, m and n are arbitrary natural numbers, there are m first target characters and n second target characters, and the dimension of the target relationship matrix can be (m + n) × (m + n). That is to say, for all the first target characters and all the second target characters, they can be represented by the rows and columns of the matrix respectively. This setting method can facilitate determining whether there is a preset association relationship between the first target character and the second target character.
[0101] It should be noted that with reference to Fig.10 , the table represents the target relationship matrix. The first target character can be "Ping" and the second target character can be "An". According to Fig.10 the target relationship matrix in, for the two characters "Ping" and "An", the value obtained from the corresponding target relationship matrix is 1, and the two characters "Ping" and "An" meet the preset association relationship.
[0102] In this embodiment, through step S151, according to the first row number corresponding to the first target character in the target relationship matrix and the first column number corresponding to the second target character in the target relationship matrix, the first relationship value corresponding to the first row number and the first column number in the target relationship matrix is obtained, and according to the first relationship value, it is determined whether the first target character and the second target character meet the preset association relationship; or, according to the second column number corresponding to the first target character in the target relationship matrix and the second row number corresponding to the second target character in the target relationship matrix, the second relationship value corresponding to the second row number and the second column number in the target relationship matrix is obtained, and according to the second relationship value, it is determined whether the first target character and the second target character meet the preset association relationship. According to the technical solution of the present invention, the rows and columns of the target relationship matrix respectively represent a target character, so that it can be quickly determined whether there is a preset association relationship between the first target character and the second target character, so as to achieve the purpose of improving the efficiency of the algorithm.
[0103] In one embodiment, as Figure 8 shown, step S140 is further described. Before step S130, there are also steps S810 to S840, including but not limited to.
[0104] Step S810: Obtain training image samples, and the training image samples include training characters.
[0105] In this step, the training image samples may be training image samples in the training image dataset in the related art, or may be a customized dataset used to improve the accuracy of the trained language representation model, such as insurance policy data, etc. The training characters refer to the characters included in the training image samples.
[0106] It is understandable that the purpose of obtaining training image samples is to perform augmented training on the language pattern model in subsequent steps.
[0107] Step S820: Perform character recognition processing on the training image to obtain word vector information of the training characters, text vector information of the training characters, and position vector information of the training characters.
[0108] In this step, the character recognition process can adopt any form of character recognition process technology in the relevant technology, which is not limited here. After the training pattern sample is subjected to character recognition process, the word vector information of the training character, the text vector information of the training character and the position vector information of the training character are obtained, which are used as the input of the language representation model.
[0109] It can be understood that the language representation model can be a pre-trained BASE model in the LayoutLM model, that is, a pre-trained BERT model. The word vector refers to a vector obtained by converting a character into a computer scale. The word vector information, text vector information and position vector information are obtained to facilitate the training of the language representation model in subsequent steps to obtain a trained language representation model.
[0110] Step S830: according to a preset mask ratio value, the text vector information is replaced with blank text vector information or the text vector information is replaced with mask text vector information or the text vector information is kept unchanged to obtain the text vector information to be trained.
[0111] In this step, the preset mask ratio value can be a random value generated by a computer or a manually set ratio value. In an optional implementation, 10% of the text vector information is converted into mask text vector information, 80% of the text vector information is converted into blank information, and 10% of the text vector information remains unchanged.
[0112] It is understandable that the text vector information obtained after processing is to facilitate the training of the language representation model in the subsequent steps.
[0113] Step S840: Train the language representation model according to the character vector information, the text vector information to be trained and the position vector information, and update the language representation model.
[0114] In this step, the language representation model is trained according to the word vector information, the text vector information to be trained and the position vector information. It can be a training method in related technologies, for example: traversing the input word vector information, the text vector information to be trained and the position vector information, comparing the obtained results with the actual results, evaluating the accuracy of the language representation model, and optimizing the language representation model until the traversal is completed or the accuracy reaches the preset target accuracy, and outputting the retrained language representation model as the trained language representation model.
[0115] In this embodiment, through steps S810 to S840, a training image sample is obtained, and the training image sample includes training characters; character recognition processing is performed on the training image to obtain word vector information of the training characters, text vector information of the training characters, and position vector information of the training characters; the text vector information is replaced with blank text vector information or the text vector information is replaced with mask text vector information or the text vector information is kept unchanged according to a preset mask ratio value to obtain text vector information to be trained; the language representation model is trained according to the word vector information, the text vector information to be trained, and the position vector information to update the language representation model. According to the technical solution of the present invention, the language representation model is retrained according to the training image sample to improve the accuracy of the language representation model, so that it can more accurately determine whether the first target character and the second target character meet the preset association relationship, thereby achieving the purpose of improving the recognition accuracy of the table unit.
[0116] In addition, an embodiment of the present invention further provides a data processing device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor and the memory may be connected via a bus or other means.
[0117] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0118] The non-transient software program and instructions required to implement the information processing method of the above embodiment are stored in the memory. When executed by the processor, the information processing method of the above embodiment is executed, for example, the above described Figure 1 Steps S110 to S150 of the method, Figure 2Steps S121 to S123 of the method, Figure 3 Steps S310 to S320 of the method, Figure 4 Steps S141 to S143 of the method, Figure 5 Steps S1421 to S1423 of the method, Figure 6 Steps S610 to S620 of the method, Figure 7 Step S151 of the method, Figure 8 Method steps S810 to S840 in.
[0119] In addition, an embodiment of the present invention further provides a computer-readable storage medium, which stores computer-executable instructions. The computer-executable instructions are executed by a processor or a controller, for example, by a processor in the above-mentioned device embodiment, so that the above-mentioned processor can execute the information processing method in the above-mentioned embodiment, for example, execute the above-mentioned Figure 1 Steps S110 to S150 of the method, Figure 2 Steps S121 to S123 of the method, Figure 3 Steps S310 to S320 of the method, Figure 4 Steps S141 to S143 of the method, Figure 5 Steps S1421 to S1423 of the method, Figure 6 Steps S610 to S620 of the method, Figure 7 Step S151 of the method, Figure 8 Method steps S810 to S840 in.
[0120] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.
[0121] The embodiments of the present invention are described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made within the knowledge scope of ordinary technicians in the technical field without departing from the purpose of the present invention.
Claims
1. An information processing method, characterized in that: include: Acquire a target image, wherein the target image includes a first target character and a second target character; Performing character recognition processing on the target image to obtain first text information of the first target character, first coordinate information of the first target character in the target image, second text information of the second target character, and second coordinate information of the second target character in the target image, wherein the first coordinate information includes first upper left corner coordinate information of a first character box corresponding to the first target character and first center point coordinate information of the first character box, and the second coordinate information includes second upper left corner coordinate information of a second character box corresponding to the second target character and second center point coordinate information of the second character box; Inputting the first text information, the second text information, the first upper left corner coordinate information, the first center point coordinate information, the second upper left corner coordinate information, and the second center point coordinate information into a trained language representation model to obtain a first feature vector of the first target character and a second feature vector of the second target character; Performing relationship association processing on the first eigenvector and the second eigenvector to obtain a target relationship matrix; Determining whether the first target character and the second target character conform to a preset association relationship according to the target relationship matrix; The performing relationship association processing on the first eigenvector and the second eigenvector to obtain a target relationship matrix includes: Adding the first eigenvector and the second eigenvector to obtain a representation relationship matrix; Input the representation relationship matrix into the fully connected layer to obtain the target relationship matrix; The step of adding the first eigenvector and the second eigenvector to obtain a representation relationship matrix comprises: Combining the first eigenvector and the second eigenvector to form an intermediate matrix; Obtain a preset matching matrix, wherein the matching matrix is a unit matrix, the dimension of the intermediate matrix is n×1, and the dimension of the matching matrix is n is an arbitrary natural number, and the matching matrix is used to add two vectors in the intermediate matrix; Multiplying the matching matrix by the intermediate matrix to obtain a representation relationship matrix; The determining, according to the target relationship matrix, whether the first target character and the second target character conform to a preset association relationship includes: According to a first row number corresponding to the first target character in a target relationship matrix and a first column number corresponding to the second target character in the target relationship matrix, a first relationship value corresponding to the first row number and the first column number of the target relationship matrix is obtained, and according to the first relationship value, whether the first target character and the second target character meet a preset association relationship; or, According to the second column number corresponding to the first target character in the target relationship matrix and the second row number corresponding to the second target character in the target relationship matrix, a second relationship value corresponding to the second row number and the second column number in the target relationship matrix is obtained, and according to the second relationship value, it is determined whether the first target character and the second target character meet a preset association relationship.
2. The information processing method according to claim 1, characterized in that: The performing character recognition processing on the target image to obtain first text information of the first target character, first coordinate information of the first target character in the target image, second text information of the second target character, and second coordinate information of the second target character in the target image includes: Performing character recognition processing on the target image to obtain first text information of the first target character, first endpoint coordinate information of a first character frame corresponding to the first target character, second text information of the second target character, and second endpoint coordinate information of a second character frame corresponding to the second target character, wherein the first endpoint coordinate information includes the first upper-left corner coordinate information, and the second endpoint coordinate information includes the second upper-left corner coordinate information; Obtaining the first center point coordinate information according to the first endpoint coordinate information; The second center point coordinate information is obtained according to the second endpoint coordinate information.
3. The information processing method according to claim 2, characterized in that: Before inputting the first text information, the second text information, the first upper left corner coordinate information, the first center point coordinate information, the second upper left corner coordinate information, and the second center point coordinate information into the trained language representation model to obtain the first feature vector of the first target character and the second feature vector of the second target character, the method includes: Obtaining maximum coordinate values of the first character and the second character in the target image and minimum coordinate values of the first character and the second character in the target image according to the first endpoint coordinate information and the second endpoint coordinate information; According to the maximum coordinate value and the minimum coordinate value, the first upper left corner coordinate information, the first center point coordinate information, the second upper left corner coordinate information and the second center point coordinate information are mapped within a preset numerical range to obtain standardized first upper left corner coordinate information, standardized first center point coordinate information, standardized second upper left corner coordinate information and standardized second center point coordinate information.
4. The information processing method according to claim 1, characterized in that: Before inputting the first text information, the second text information, the first upper left corner coordinate information, the first center point coordinate information, the second upper left corner coordinate information, and the second center point coordinate information into the trained language representation model to obtain the first feature vector of the first target character and the second feature vector of the second target character, the method includes: Obtaining a character distance between the first target character and the second target character according to the first center point coordinate information and the second center point coordinate information; When the character distance is greater than a preset distance threshold, it is determined that the first target character and the second target character do not conform to the preset association relationship.
5. The information processing method according to claim 1, characterized in that: The language representation model is trained by the following steps: Acquire a training image sample, wherein the training image sample includes training characters; Performing character recognition processing on the training image samples to obtain word vector information of the training characters, text vector information of the training characters, and position vector information of the training characters; According to a preset mask ratio value, the text vector information is replaced with blank text vector information or the text vector information is replaced with mask text vector information or the text vector information is kept unchanged to obtain the text vector information to be trained; The language representation model is trained according to the character vector information, the text vector information to be trained and the position vector information, and the language representation model is updated.
6. An information processing device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the information processing method according to any one of claims 1 to 5 when executing the computer program.
7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the information processing method according to any one of claims 1 to 5.
