A case information extraction sequence prediction method, device, equipment and medium

Abstract

The present application relates to the technical field of artificial intelligence, and in particular to a case information extraction sequence prediction method and device, equipment and medium. The above method is applied to the medical field, extracts N text boundary boxes and text content in the to-be-extracted case, first sorts the N text boundary boxes, performs window sliding on the first sorted row to obtain the corresponding text boundary box sequence, selects a basic text boundary box, constructs a pair of paired text boundary boxes with the basic text boundary box, determines a target text boundary box adjacent to the basic text boundary box according to the prediction result of each corresponding paired text boundary box, second sorts the text boundary boxes of each row according to the basic text boundary box and the target text boundary box, and determines the extraction sequence of the to-be-extracted case according to the text content corresponding to the sorted text boundary boxes. Two sorts of extraction are performed, the extraction accuracy of the text boundary box is improved, and thus the extraction accuracy of the text content is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and medium for predicting the extraction order of case information. Background Technology

[0002] Digital healthcare is a new and modern medical approach that applies modern computer and information technologies to the entire medical process. It represents the development direction and management goal of public healthcare. In digital healthcare research, various documents such as medical records, pathology reports, CT scans, MRIs, and surgical records are typically used as data sources. Information is extracted from these sources to obtain corresponding information. With the rapid development of deep learning and artificial intelligence technologies, automated document information extraction based on optical character recognition has become a crucial step, effectively improving the efficiency of manual data entry. However, in the information extraction process, current technologies are relatively accurate for extracting information from standard case images. When the case images are irregular, the information extraction order becomes chaotic, resulting in lower accuracy in the information extraction order within the case. Therefore, improving the accuracy of the case information extraction order is an urgent problem to be solved. Summary of the Invention

[0003] In view of this, embodiments of this application provide a method, apparatus, device and medium for predicting the extraction order of case information, in order to solve the problem of low accuracy of information extraction order during the information extraction process.

[0004] In a first aspect, embodiments of this application provide a method for predicting the extraction order of case information, the extraction order prediction method comprising:

[0005] Obtain the cases to be extracted, perform text recognition on the cases to be extracted, and extract N text bounding boxes and the text content in each text bounding box from the cases to be extracted, where N is an integer greater than 1;

[0006] Based on the coordinate information of the N text bounding boxes, the N text bounding boxes are sorted in the first sorting to obtain M text bounding boxes corresponding to the first sorting rows and the text content in the corresponding text bounding boxes, where M is an integer less than or equal to N;

[0007] For any first sorting row, the window of the first sorting row is slid according to the preset window size and preset step size. Each time the window is slid, the text bounding box sequence corresponding to each window slid is obtained.

[0008] In each window sliding sequence, a text bounding box is selected as the base text bounding box. Paired text bounding boxes are constructed with the base text bounding box. The relationship between each pair of text bounding boxes is predicted by the trained prediction model to obtain the prediction result of each corresponding pair of text bounding boxes.

[0009] determine a target text bounding box adjacent to the base text bounding box according to the prediction result of each corresponding pair text bounding box, perform second sorting on the text bounding box of each row according to the base text bounding box and the target text bounding box, obtain the sorted text bounding box, and determine the extraction order of the case to be extracted according to the text content corresponding to the sorted text bounding box.

[0010] In a second aspect, an embodiment of the present application provides a case information extraction order prediction device, which comprises:

[0011] The acquisition module is configured to acquire a case to be extracted, perform text recognition on the case to be extracted, extract N text bounding boxes in the case to be extracted and text content in each text bounding box, where N is an integer greater than 1.

[0012] The first sorting module is configured to perform first sorting on the N text bounding boxes according to coordinate information in the N text bounding boxes, and obtain text bounding boxes corresponding to M first sorting rows and text content in the corresponding text bounding boxes, where M is an integer less than or equal to N.

[0013] The sliding module is configured to, for any first sorting row, perform window sliding on the first sorting row according to a preset window size and a preset step length, and obtain a text bounding box sequence corresponding to each window sliding at each time of window sliding.

[0014] The prediction module is configured to select a text bounding box as a base text bounding box in the text bounding box sequence corresponding to each window sliding at each time, construct pair text bounding boxes paired with the base text bounding box, perform relationship prediction on each pair text bounding box through a trained prediction model, and obtain a prediction result of each corresponding pair text bounding box.

[0015] The second sorting module is configured to determine a target text bounding box adjacent to the base text bounding box according to the prediction result of each corresponding pair text bounding box, perform second sorting on the text bounding box of each row according to the base text bounding box and the target text bounding box, obtain the sorted text bounding box, and determine the extraction order of the case to be extracted according to the text content corresponding to the sorted text bounding box.

[0016] In a third aspect, an embodiment of the present application provides a terminal device, which comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the case information extraction order prediction method of the first aspect when executing the computer program.

[0017] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium storing a computer program, and the computer program is executed by a processor to implement the case information extraction sequence prediction method according to the first aspect

[0018] Compared with the prior art, the present application has the following beneficial effects:

[0019] The case to be extracted is obtained, text recognition is performed on the case to be extracted, N text boundary boxes in the case to be extracted and text content in each text boundary box are extracted, N is an integer greater than 1, the N text boundary boxes are first sorted according to coordinate information in the N text boundary boxes, M first sorting rows are obtained, and text boundary boxes corresponding to the M first sorting rows and text content in the corresponding text boundary boxes are obtained, M is an integer less than or equal to N, for any first sorting row, a window sliding is performed on the first sorting row according to a preset window size and a preset step, at each window sliding, a text boundary box sequence corresponding to each window sliding is obtained, one text boundary box in each window sliding text boundary box sequence is selected as a basic text boundary box, a paired text boundary box paired with the basic text boundary box is constructed, a relationship prediction is performed on each paired text boundary box by using a trained prediction model, a prediction result of each corresponding paired text boundary box is obtained, a target text boundary box adjacent to the basic text boundary box is determined according to the prediction result of each corresponding paired text boundary box, a second sorting is performed on the text boundary boxes of each row according to the basic text boundary box and the target text boundary box, a sorted text boundary box is obtained, and an extraction sequence of the case to be extracted is determined according to text content corresponding to the sorted text boundary box. In the present application, the text boundary boxes of each row are first sorted, the next target text boundary box is predicted by using the trained prediction model, the next target text boundary box corresponding to the text boundary boxes of each row is obtained, the extraction accuracy of the text boundary box sequence is improved, and thus the extraction accuracy of the text content sequence is improved. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without any creative labor.

[0021] Figure 1 is an application environment schematic diagram of a case information extraction sequence prediction method provided by the embodiments of the present application;

[0022] Figure 2 is a flow schematic diagram of a case information extraction sequence prediction method provided by the embodiments of the present application;

[0023] Figure 3 FIG. 1 is a structural schematic diagram of a case information extraction sequence prediction device provided by an embodiment of the present application;

[0024] Figure 4 FIG. 2 is a structural schematic diagram of a terminal device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0026] It should be understood that, when used in the specification and the appended claims of the present application, the term “comprising” indicates the presence of the described features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0027] It should also be understood that, when used in the specification and the appended claims of the present application, the term “and / or” refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0028] As used in the specification and the appended claims of the present application, the term “if” can be interpreted as “when” or “upon” or “in response to a determination” or “in response to detecting” depending on the context. Similarly, the phrase “if determined” or “if detected [the described condition or event]” can be interpreted as meaning “upon determining” or “in response to determining” or “upon detecting [the described condition or event]” or “in response to detecting [the described condition or event]” depending on the context.

[0029] In addition, in the description of the present application and the appended claims, the terms “first”, “second”, “third”, etc. are only used for differentiation in description, and cannot be understood as indicating or implying relative importance.

[0030] Reference within the specification of this document to "one embodiment" or "some embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrase "in one embodiment" or "in some embodiments" in various places within specified

[0031] The embodiments of the present application can acquire and process related data based on artificial intelligence technology. Wherein, artificial intelligence (AI) is to use digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results.

[0032] The basic technology of artificial intelligence generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The software technology of artificial intelligence mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0033] It should be understood that the size of the serial number of each step in the following embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0034] In order to illustrate the technical solutions of the present application, the following will be explained by specific embodiments.

[0035] An embodiment of the present application provides a case information extraction sequence prediction method, which can be applied in an application environment such as Figure 1 The local end includes but is not limited to a palmtop computer, a desktop computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), and the like. The server can be implemented by an independent server or a server cluster generated by multiple servers.

[0036] Referring to Figure 2is a flowchart of a case information extraction sequence prediction method provided by an embodiment of the present application. The case information extraction sequence prediction method can be applied to a server in Figure 1 , which is connected to a corresponding local terminal, as shown in Figure 2 . The case information extraction sequence prediction method can include the following steps.

[0037] S201: Obtain a case to be extracted, perform text recognition on the case to be extracted, and extract N text boundary boxes in the case to be extracted and text content in each text boundary box.

[0038] In step S201, the case to be extracted is a case image obtained by photographing or scanning the case. The text recognition is performed on the case to be extracted, and N text boundary boxes in the case to be extracted and text content in each text boundary box are extracted, where N is an integer greater than 1.

[0039] In this embodiment, the case to be extracted is obtained by scanning or photographing a paper medical record. Before photographing, a photographing parameter can be set above the photographing area. The photographing color can be set as color, black and white, or grayscale. The photographing edge cutting method can be set as manual cutting, automatic cutting, or no cutting. Whether to remove shadows during photographing can be set. The text area in the case to be extracted contains printed and / or handwritten text.

[0040] When extracting N text boundary boxes in the case to be extracted and text content in each text boundary box, an optical character recognition technology can be used to perform text recognition on the obtained case image to obtain multiple text segments and multiple boundary boxes. The optical character recognition technology can refer to detecting and recognizing text in an image. For example, the optical character recognition technology can include two parts of text detection and text recognition. Text detection locates the area where text exists in the image, that is, finds the text boundary box of a word or a sentence. Text recognition recognizes the located text to obtain the corresponding text content.

[0041] In another example, the case image to be extracted can also be input into a recognition model to obtain multiple text segments and multiple boundary boxes. The recognition model can be implemented as any target detection algorithm.

[0042] It should be noted that the text segments extracted from the case image to be extracted can be words or sentences. In addition, the extracted text boundary box has a corresponding position, which can be composed of two pairs of horizontal and vertical coordinates. In one example, the two pairs of horizontal and vertical coordinates include the horizontal and vertical coordinates of the top left vertex of the text boundary box and the horizontal and vertical coordinates of the bottom right vertex.

[0043] Optionally, the N text bounding boxes in the case to be extracted and the text content in each text bounding box are extracted, including:

[0044] Text detection is performed on the case to be extracted to obtain text bounding boxes representing text in the case to be extracted.

[0045] Text recognition is performed on the text bounding boxes to obtain the text content of the text in the text bounding boxes.

[0046] In this embodiment, the text detection network is used to perform text detection on the case image to be extracted to obtain text bounding boxes representing text in the case image to be extracted. The text detection network can include a convolutional layer, a batch normalization layer, a fusion layer, and an activation function.

[0047] It should be noted that before the text detection network is used to perform text detection on the case image to be extracted, the initial detection network and the training sample set are obtained, the initial detection network is supervised trained using the training sample set to obtain the trained detection network, and the trained detection network is determined as the preset detection network, wherein the training sample set is training data containing the case image to be extracted and the corresponding text bounding box label. The text detection network is used to perform text detection on the text image to obtain a text detection box, including: using the convolutional layer to perform image feature extraction on the case image to be extracted to obtain a feature image, using the batch normalization layer (Batch Normalization, BN) to perform standardization operation on the feature image to obtain a standard feature image, using the fusion layer to fuse the bottom layer feature of the case image to be extracted and the standard feature image to obtain a target feature image, using the activation function to output the detection result of the target feature image, and generating the text bounding box representing the text according to the detection result.

[0048] Text recognition is performed on the text bounding boxes to obtain the text content of the text in the text bounding boxes. When performing text recognition on the text bounding boxes, a preset recognition network is used to extract and recognize the text. The recognition network can include a convolutional neural network, a long short-term memory network, and a time series classification network. The convolutional neural network is used to recognize the feature image of the text bounding box, the long short-term memory network is used to extract the text sequence of the feature image, and the time series classification network is used to solve the problem that characters in the text feature sequence cannot be aligned. Further, the convolutional neural network includes a convolutional layer, a pooling layer, and a fully connected layer, and the long short-term memory network includes an input gate, a forget gate, and an output gate.

[0049] It should be noted that before the text boundary box is subjected to text recognition by using the preset recognition network, the embodiment of the present application further comprises: obtaining an initial recognition network and a training sample set, using the training sample set to supervise training of the initial recognition network to obtain a trained recognition network, and determining the trained recognition network as the preset recognition network, wherein the training sample set is label training data containing the text boundary box and corresponding text content.

[0050] It should be noted that when the text is subjected to extraction and recognition by using the preset recognition network, the text boundary box is subjected to feature extraction by using the convolutional neural network to obtain a feature image, specifically, the text boundary box is subjected to convolution feature extraction by using a convolution layer in the convolutional neural network to obtain an initial feature image, the initial feature image is subjected to dimension reduction by using a pooling layer in the convolutional neural network to obtain a reduced dimension feature image, and the reduced dimension feature image is output by using a full connection layer in the convolutional neural network to obtain the feature image. The feature image is subjected to text position sequence recognition by using the long short-term memory network to generate initial text content, specifically, a state value of the feature image is calculated by using an input gate of the long short-term memory network, an activation value of the feature image is calculated by using a forgetting gate of the long short-term memory network, a state update value of the feature image is calculated according to the state value and the activation value, and a text position sequence of the state update value is calculated by using an output gate of the long short-term memory network to generate the initial text content. The initial text content is subjected to character alignment by using the time sequence classification network to generate corresponding text content.

[0051] S202: According to the coordinate information in the N text boundary boxes, the N text boundary boxes are subjected to first sorting to obtain the text boundary boxes corresponding to the M first sorting rows and the text content in the corresponding text boundary boxes, M being an integer less than or equal to N.

[0052] In step S202, according to the coordinate information in the N text boundary boxes, the text boundary boxes in each row are sorted from left to right to obtain the text boundary boxes corresponding to the M first sorting rows and the text content in the corresponding text boundary boxes, M being an integer less than or equal to N.

[0053] In this embodiment, the first row is sorted according to the ordinate in the coordinate information in the text boundary box, and when the first sorting is performed, the irregular text boundary box is subjected to initial correction. For example, when the to-be-extracted case image is a tilted image, the extracted text boundary box is also tilted. The text boundary box is subjected to initial correction and first sorting. When the first sorting is performed, the sorting can be corrected according to the least square method to obtain the text boundary box corresponding to the M first sorting rows and the text content in the corresponding text boundary box.

[0054] Optionally, according to the coordinate information in the N text boundary boxes, the N text boundary boxes are subjected to first sorting to obtain the text boundary boxes corresponding to the M first sorting rows, comprising:

[0055] According to the coordinate information in each text boundary box, a sorting result of the multiple original lines is obtained.

[0056] For any original line, the number of text boundary boxes in the sorting result of the corresponding original line is determined, and if the number is greater than a preset threshold, the original line is determined as a high-confidence line. All original lines are traversed to obtain K high-confidence lines, K being an integer greater than 1.

[0057] For any high-confidence line, the slope of the high-confidence line is calculated according to the coordinate information of each text boundary box in the high-confidence line. K slopes are obtained by traversing the K high-confidence lines.

[0058] The mean of the K slopes is calculated to obtain a mean slope, and the vertical intercept of each text boundary box is calculated according to the mean slope and the coordinate information in the text boundary box.

[0059] According to the vertical intercept of each text boundary box, the text boundary box corresponding to each first sorting line is determined to obtain M text boundary boxes corresponding to the first sorting lines.

[0060] In this embodiment, when the corresponding text boundary box is extracted using the optical character recognition technology, it is scanned from left to right to obtain the corresponding text boundary box. When the first line is scanned, the obtained text boundary box is arranged in the first line, and when the second line is scanned, the obtained text boundary box is arranged in the second line. The difference between the vertical coordinates in the coordinate information in the text boundary box can also be calculated. When the difference between the vertical coordinates is less than a difference threshold, it is considered that the corresponding text boundary box is in the same line, and then the corresponding text boundary box is sorted from left to right according to the horizontal coordinates to obtain the sorting result of the multiple original lines.

[0061] For any original line, the number of text boundary boxes in the sorting result of the corresponding original line is determined, and if the number is greater than a preset threshold, the original line is determined as a high-confidence line. All original lines are traversed to obtain K high-confidence lines, K being an integer greater than 1. For any high-confidence line, the slope of the high-confidence line is calculated according to the coordinate information of each text boundary box in the high-confidence line. K slopes are obtained by traversing the K high-confidence lines. When calculating the slope, the text boundary boxes in the high-confidence line are curve-fitted using the least square method to calculate the slope of the curve, thereby obtaining the slope of the corresponding high-confidence line. The mean of the K slopes is calculated to obtain a mean slope, and the vertical intercept of each text boundary box is calculated according to the mean slope and the coordinate information in the text boundary box.

[0062] It should be noted that when the text bounding box in the high confidence row is curve fitted using the least square method, the center point coordinates in each text bounding box are first calculated, the center point coordinates in each text bounding box are used for curve fitting, the slope in the curve is calculated, and the slope corresponding to the high confidence row is obtained.

[0063] The vertical intercepts of each text bounding box are clustered, and the text bounding boxes corresponding to the vertical intercepts in the same cluster are text bounding boxes in the same row, so that the text bounding boxes corresponding to each first sorted row are determined, and M text bounding boxes corresponding to the first sorted rows are obtained.

[0064] Optionally, the vertical intercept of each text bounding box is calculated according to the mean slope and the coordinate information in the text bounding box, including:

[0065] For any text bounding box, the center coordinates in the text bounding box are calculated according to the coordinate information in the text bounding box.

[0066] The vertical intercept of the center coordinates is calculated according to the center coordinates and the mean slope, and N

[0067] text bounding boxes are traversed to obtain the vertical intercept of each text bounding box.

[0068] In this embodiment, when the vertical intercept is calculated, the corresponding coordinates used are the center point coordinates in the text bounding box, the vertical intercept of the center coordinates is calculated according to the center coordinates and the mean slope, and the corresponding vertical intercept formula is as follows:

[0069]

[0070] where b ′ is the vertical intercept of the corresponding text bounding box, y0 is the vertical coordinate of the center point of the corresponding text bounding box, x0 is the horizontal coordinate of the center point of the corresponding text bounding box, is the mean slope. N text bounding boxes are traversed to obtain the vertical intercept of each text bounding box.

[0071] Optionally, the text bounding box corresponding to each first sorted row is determined according to the vertical intercept of each text bounding box, including:

[0072] The vertical intercepts are clustered to obtain M cluster results.

[0073] For any cluster result, the text bounding box corresponding to the vertical intercept in the cluster result is taken as the same first sorted row, and the text bounding box corresponding to each first sorted row is determined.

[0074] In this embodiment, the longitudinal intercepts are clustered, the longitudinal intercepts are clustered into M cluster results, the maximum difference value between the longitudinal intercepts in the same cluster result is less than a preset threshold, the text bounding boxes corresponding to the longitudinal intercepts in the same cluster result are sequentially sorted as the same row, each text bounding box is rotated according to the mean slope, and the text bounding box corresponding to each first sorted row is determined.

[0075] It should be noted that, in order to improve the accuracy of the first sorting, an equation in the vertical direction of the row direction can also be calculated, and the calculation formula is as follows:

[0076]

[0077] wherein b ′ is the longitudinal intercept of the corresponding text bounding box, is the mean slope, the intersection with the X axis is calculated using the equation, that is, the horizontal intercept, and the difference between any two text bounding boxes is calculated, when the difference is less than a preset difference threshold, and the two text bounding boxes are in the same sorted row, the corresponding text bounding boxes are merged to improve the processing efficiency.

[0078] S203: For any first sorted row, a window is slid according to a preset window size and a preset step size, and a text bounding box sequence corresponding to each window sliding is obtained.

[0079] In step S203, any first sorted row is slid to obtain a text bounding box sequence corresponding to each sliding, and the number of text bounding box sequences can be multiple.

[0080] In this embodiment, for example, when the text bounding boxes in any first sorted row are serial numbers, project names, English abbreviations and dates, the text bounding boxes are box1, box2, box3 and box4. When the preset window size is 3 and the preset step size is 1, the window is slid from left to right to obtain the text bounding box sequence box1, box2, box3 of the first window sliding, and the text bounding box sequence box2, box3, box4 of the second window sliding. When the preset window size is 3 and the preset step size is 1, the window is slid from left to right to obtain the text bounding box sequence box1, box2, box3, box4 of the first window sliding.

[0081] S204: In the text bounding box sequence corresponding to each window sliding, a text bounding box is selected as a basic text bounding box, a paired text bounding box is constructed with the basic text bounding box, a relationship prediction is performed on each paired text bounding box through a trained prediction model, and a prediction result of each corresponding paired text bounding box is obtained.

[0082] In step S204, according to the text bounding box sequence obtained by each window sliding, the probability value of the next text bounding box predicted according to the basic text bounding box in the text bounding box sequence is calculated, and according to the probability value, the next text bounding box in the corresponding basic text bounding box is obtained. Each text bounding box in each first sorted row is traversed to obtain the prediction result corresponding to each text bounding box.

[0083] In this embodiment, a text bounding box is selected as a basic text bounding box in the text bounding box sequence corresponding to each window sliding, for example, when the text bounding box sequence corresponding to window sliding is box1, box2, box3, a text bounding box is selected as a basic text bounding box from the corresponding text bounding box sequence. The text bounding box box1 can be selected as the basic text bounding box. Since it is the prediction of the next text bounding box, the earlier text bounding box is generally selected as the basic text bounding box. The paired text bounding boxes paired with the basic text bounding box are constructed, for example, [box1, box2], [box1, box3]. The trained prediction model is used to predict the relationship of each paired text bounding box to obtain the prediction result of each corresponding paired text bounding box, that is, the prediction probability of the next text bounding box of the text bounding box box1. The prediction probability of the next text bounding box of the text bounding box box1 being box2 is p1, and the prediction probability of the next text bounding box of the text bounding box box1 being box3 is p2.

[0084] When the text bounding box sequence corresponding to window sliding is box2, box3, box4, a text bounding box is selected as a basic text bounding box from the corresponding text bounding box sequence. The text bounding box box2 can be selected as the basic text bounding box. Paired text bounding boxes paired with the basic text bounding box are constructed, for example, [box2, box3], [box2, box4]. The prediction probability of the next text bounding box of the text bounding box box2 is calculated. The prediction probability of the next text bounding box of the text bounding box box1 being box2 is p3, and the prediction probability of the next text bounding box of the text bounding box box1 being box3 is p4. The trained prediction model is used to predict the relationship of each paired text bounding box to obtain the prediction result of each corresponding paired text bounding box.

[0085] It should be noted that before using the trained prediction model to predict the relationship of each pair of text bounding boxes, the obtained initial prediction model needs to be trained. During training, the positive and negative sample pairs are used for training. First, the positive and negative sample pairs are constructed. The target text bounding boxes in the correct reading order document can be used as the positive sample pair, for example, [box1, box2, true]. The non-target text bounding boxes in the correct reading order document can be used as the negative sample pair, for example, [box1, box2, false]. According to the constructed positive and negative sample pairs, the initial prediction model is trained. During training, the loss of the positive sample pair is reduced, and the loss of the negative sample pair is increased to obtain the trained prediction model.

[0086] S205: According to the prediction result of each corresponding pair of text bounding boxes, determine the target text bounding box adjacent to the basic text bounding box, and according to the basic text bounding box and the target text bounding box, the text bounding boxes of each row are secondly sorted to obtain the sorted text bounding boxes. According to the text content corresponding to the sorted text bounding boxes, the extraction order of the to-be-extracted case is determined.

[0087] In step S205, according to the prediction result between the basic text bounding box, the text bounding box corresponding to the maximum prediction result is taken as the target text bounding box of the basic text bounding box. According to the target text bounding box, the sorted text bounding boxes are obtained. According to the text content corresponding to the sorted text bounding boxes, the extraction order of the to-be-extracted case is determined.

[0088] In this embodiment, according to the relationship prediction result in the pair of text bounding boxes, the text bounding boxes are secondly sorted to obtain the sorted text bounding boxes. According to the text content corresponding to the sorted text bounding boxes, the extraction order of the to-be-extracted case is determined. For example, the text content in the text bounding box box1 is the serial number, the text content in the text bounding box box2 is the project name, the text content in the text bounding box box3 is the English abbreviation, and the text content in the text bounding box box4 is the date. If the sorted text bounding boxes are box1, box2, box4, and box3, the extraction order of the text content of the to-be-extracted case is serial number, project name, date, and English abbreviation.

[0089] Optionally, according to the prediction result of each corresponding pair of text bounding boxes, the target text bounding box adjacent to the basic text bounding box is determined, comprising:

[0090] According to the prediction result of each corresponding pair of text bounding boxes, the pair of text bounding boxes when the prediction result is the maximum value is selected as the corresponding target pair of text bounding boxes;

[0091] According to the corresponding paired text bounding box of each target paired text bounding box, the target text bounding box adjacent to the basic text bounding box is determined.

[0092] In this embodiment, according to the prediction result of each corresponding paired text bounding box, the target paired text bounding box corresponding to the maximum prediction result is selected, and the target text bounding box adjacent to the basic text bounding box is determined from the target paired text bounding box. For example, the text bounding box box1 is selected as the basic text bounding box, the paired text bounding boxes paired with the basic text bounding box are constructed, and the paired text bounding boxes [box1, box2] and [box1, box3] are obtained. The trained prediction model is used to perform relationship prediction on each paired text bounding box, and the prediction result of each corresponding paired text bounding box is [0.95, 0.05], that is, the relationship prediction result of the paired text bounding box [box1, box2] is 0.95, and the relationship prediction result of the paired text bounding box [box1, box3] is 0.05. Then, the text bounding box box2 is taken as the target text bounding box of the text bounding box box1.

[0093] A case to be extracted is obtained, text recognition is performed on the case to be extracted, and N text bounding boxes and text content in each text bounding box in the case to be extracted are extracted, N is an integer greater than 1, the N text bounding boxes are first sorted according to coordinate information in the N text bounding boxes, M first sorting rows corresponding text bounding boxes and text content in the corresponding text bounding boxes are obtained, M is an integer less than or equal to N, for any first sorting row, a window sliding is performed on the first sorting row according to a preset window size and a preset step size, a text bounding box sequence corresponding to each window sliding is obtained at each window sliding, a text bounding box is selected as a basic text bounding box in each window sliding corresponding text bounding box sequence, paired text bounding boxes paired with the basic text bounding box are constructed, a trained prediction model is used to perform relationship prediction on each paired text bounding box, a prediction result of each corresponding paired text bounding box is obtained, a target text bounding box adjacent to the basic text bounding box is determined according to the prediction result of each corresponding paired text bounding box, and each row of text bounding boxes is second sorted according to the basic text bounding box and the target text bounding box. The sorted text bounding boxes are obtained, and the extraction order of the case to be extracted is determined according to the text content corresponding to the sorted text bounding boxes. In this application, the first sorting of each row of text bounding boxes is performed, the next target text bounding box is predicted by the trained prediction model, the next target text bounding box corresponding to each row of text bounding boxes is obtained, the extraction accuracy of the text bounding box is improved, and thus the extraction accuracy of the text content is improved.

[0094] Referring to Figure 3 , Figure 3A structural block diagram of a case information extraction sequence prediction device provided by an embodiment of the present application is shown. The extraction sequence prediction device is applied to the server. For ease of illustration, only the parts related to the embodiments of the present application are shown. The extraction sequence prediction device 30 includes an acquisition module 31, a first sorting module 32, a sliding module 33, a prediction module 34, and a second sorting module 35.

[0095] The acquisition module 31 is configured to acquire a case to be extracted, perform text recognition on the case to be extracted, and extract N text bounding boxes in the case to be extracted and text content in each text bounding box, where N is an integer greater than 1.

[0096] The first sorting module 32 is configured to perform first sorting on the N text bounding boxes according to coordinate information in the N text bounding boxes, to obtain text bounding boxes corresponding to M first sorting rows and text content in the corresponding text bounding boxes, where M is an integer less than or equal to N.

[0097] The sliding module 33 is configured to, for any first sorting row, perform window sliding on the first sorting row according to a preset window size and a preset step length, and obtain a text bounding box sequence corresponding to each window sliding at each time of window sliding.

[0098] The prediction module 34 is configured to select one text bounding box from the text bounding box sequence corresponding to each window sliding as a basic text bounding box, construct a paired text bounding box paired with the basic text bounding box, perform relationship prediction on each paired text bounding box by using a trained prediction model, and obtain a prediction result of each corresponding paired text bounding box.

[0099] The second sorting module 35 is configured to determine a target text bounding box adjacent to the basic text bounding box according to the prediction result of each corresponding paired text bounding box, perform second sorting on the text bounding boxes of each row according to the basic text bounding box and the target text bounding box, obtain sorted text bounding boxes, and determine an extraction sequence of the case to be extracted according to text content corresponding to the sorted text bounding boxes.

[0100] Optionally, the acquisition module 31 includes:

[0101] A detection unit is configured to perform text detection on the case to be extracted to obtain text bounding boxes representing text in the case to be extracted.

[0102] An identification unit is configured to perform text recognition on the text bounding boxes to obtain text content of the text in the text bounding boxes.

[0103] Optionally, the first sorting module 32 includes:

[0104] The original sorting unit is configured to obtain a sorting result of a plurality of original lines according to coordinate information in each text bounding box.

[0105] The judging unit is configured to determine, for any original line, a number of text bounding boxes in the original line corresponding to the sorting result, and determine the original line as a high-confidence line if the number is greater than a preset threshold.

[0106] The first calculating unit is configured to calculate, for any high-confidence line, a slope of the high-confidence line according to coordinate information of each text bounding box in the high-confidence line, and obtain K slopes by traversing the K high-confidence lines.

[0107] The second calculating unit is configured to calculate a mean value of the K slopes to obtain a mean slope, and calculate a vertical intercept of each text bounding box according to the mean slope and the coordinate information in the text bounding box.

[0108] The determining unit is configured to determine, according to the vertical intercept of each text bounding box, a text bounding box corresponding to each first sorting line to obtain M text bounding boxes corresponding to the M first sorting lines.

[0109] Optionally, the second calculating unit includes:

[0110] The first calculating sub-unit is configured to calculate, for any text bounding box, a center coordinate in the text bounding box according to coordinate information in the text bounding box.

[0111] The second calculating sub-unit is configured to calculate, according to the center coordinate and the mean slope, a vertical intercept of the center coordinate, and obtain a vertical intercept of each text bounding box by traversing N text bounding boxes.

[0112] Optionally, the determining unit includes:

[0113] The clustering sub-unit is configured to perform clustering processing on the vertical intercepts to obtain M clustering results.

[0114] The sorting sub-unit is configured to determine, for any clustering result, a text bounding box corresponding to a vertical intercept in the clustering result as a same first sorting line, and determine a text bounding box corresponding to each first sorting line.

[0115] Optionally, the second sorting module 35 includes:

[0116] The selecting unit is configured to select, according to a prediction result of each corresponding pair of text bounding boxes, a pair of text bounding boxes when the prediction result is a maximum value as a corresponding target pair of text bounding boxes.

[0117] The target determination unit is used to determine the corresponding text bounding box as the target text bounding box adjacent to the base text bounding box based on the text bounding boxes that are paired with the base text bounding box in the target paired text bounding box.

[0118] It should be noted that the information interaction and execution process between the above-mentioned units are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0119] Figure 4 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Figure 4 As shown, the terminal device of this embodiment includes: at least one processor ( Figure 4 Only one is shown in the diagram), a memory, and a computer program stored in the memory and capable of running on at least one processor, which, when executed by the processor, implements the steps in the embodiments of the method for predicting the extraction order of any of the case information described above.

[0120] The terminal device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 4 This is merely an example of a terminal device and does not constitute a limitation on the terminal device. A terminal device may include more or fewer components than shown in the figure, or a combination of certain components, or different components, such as network interfaces, displays, and input devices.

[0121] The processor referred to can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0122] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory can be the internal memory of the terminal device, and the internal memory provides an environment for the running of the operating system and the computer readable instructions in the readable storage medium. The readable storage medium can be the hard disk of the terminal device, and in other embodiments, can also be the external storage device of the terminal device, for example, the plug-in hard disk, the smart media card (SMC), the secure digital (SD) card, the flash card, etc. equipped on the terminal device. Further, the memory can include both the internal storage unit of the terminal device and the external storage device. The memory is used to store the operating system, the application program, the boot loader, the data, and other programs, such as the program code of the computer program, etc. The memory can also be used to temporarily store the data that has been output or will be output.

[0123] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of software functional unit. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the present application. The specific working process of the units and modules in the above device can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here. If the integrated unit is realized in the form of software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, all or part of the processes in the above embodiment methods can be completed by a computer program instructing related hardware. The computer program can be stored in a computer readable storage medium. When the processor executes the computer program, the steps of the above method embodiments can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable file or some intermediate form. The computer readable medium at least includes any entity or device that can carry computer program code, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium. For example, U disk, mobile hard disk, magnetic disk or optical disk, etc. In some jurisdictions, according to legislation and patent practice, computer readable medium cannot be electrical carrier signal and telecommunication signal.

[0124] The above embodiment methods can also be implemented by a computer program product, which, when running on a terminal device, enables the terminal device to execute the steps of the above method embodiments.

[0125] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0126] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0127] In the embodiments provided by the present application, it should be understood that the disclosed apparatus / terminal device and method can be implemented in other ways. For example, the division of the apparatus / terminal device embodiments described above is merely an example. For example, the division of the modules or units is merely a logical function division. There can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between the units can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or in other forms.

[0128] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0129] The above embodiments are only used to illustrate the technical solutions of the present application, but not limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalent; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method of predicting an extraction order of case information, characterized by, The extraction sequence prediction method comprises: Obtaining a case to be extracted, performing text recognition on the case to be extracted, and extracting N text boundary boxes in the case to be extracted and text content in each text boundary box, N being an integer greater than 1; According to the coordinate information in the N text boundary boxes, the N text boundary boxes are first sorted to obtain text boundary boxes corresponding to M first sorting rows and text content in the corresponding text boundary boxes, M being an integer less than or equal to N; The first sorting of the N text boundary boxes according to the coordinate information in the N text boundary boxes comprises: According to the coordinate information in each text boundary box, the sorting results of a plurality of original rows are obtained; For any original row, the number of text boundary boxes of the sorting result in the corresponding original row is determined, if the number is greater than a preset threshold, the original row is determined as a high-confidence row, all original rows are traversed to obtain K high-confidence rows, K being an integer greater than 1; For any high-confidence row, the slope of the high-confidence row is calculated according to the coordinate information of each text boundary box in the high-confidence row, and K slopes are obtained by traversing the K high-confidence rows; The mean value of the K slopes is calculated to obtain a mean value slope, and the vertical intercept of each text boundary box is calculated according to the mean value slope and the coordinate information in the text boundary box; According to the vertical intercept of each text boundary box, the text boundary box corresponding to each first sorting row is determined to obtain the text boundary box corresponding to M first sorting rows; For any first sorting row, a window sliding is performed on the first sorting row according to a preset window size and a preset step, and a text boundary box sequence corresponding to each window sliding is obtained at each window sliding; In the text boundary box sequence corresponding to each window sliding, a text boundary box is selected as a basic text boundary box, paired text boundary boxes paired with the basic text boundary box are constructed, a relationship prediction is performed on each paired text boundary box by using a trained prediction model, and a prediction result of each corresponding paired text boundary box is obtained; According to the prediction result of each corresponding paired text boundary box, a target text boundary box adjacent to the basic text boundary box is determined, the text boundary boxes of each row are second sorted according to the basic text boundary box and the target text boundary box, and a sorted text boundary box is obtained, and the extraction sequence of the case to be extracted is determined according to the text content corresponding to the sorted text boundary box. The determination of the target text boundary box adjacent to the basic text boundary box according to the prediction result of each corresponding paired text boundary box comprises: According to the prediction result of each corresponding paired text boundary box, the paired text boundary box when the prediction result is the maximum value is selected as the corresponding target paired text boundary box; According to the text boundary box paired with the basic text boundary box in the target paired text boundary box, the corresponding text boundary box is determined as the target text boundary box adjacent to the basic text boundary box.

2. The decoupled sequential prediction method of claim 1, wherein, The extracting the N text bounding boxes in the case to be extracted and the text content in each text bounding box comprises: text detection is performed on the case to be extracted to obtain text bounding boxes representing text in the case to be extracted; text recognition is performed on the text bounding boxes to obtain the text content of the text in the text bounding boxes.

3. The decoupled sequential prediction method of claim 1, wherein, The calculating the vertical intercept of each text bounding box according to the mean slope and the coordinate information in the text bounding box comprises: For any text bounding box, the center coordinate in the text bounding box is calculated according to the coordinate information in the text bounding box; The vertical intercept of the center coordinate is calculated according to the center coordinate and the mean slope, and the vertical intercept of each text bounding box is obtained by traversing the N text bounding boxes.

4. The decoupled sequential prediction method of claim 1, wherein, The determining the text bounding box corresponding to each first sorting row according to the vertical intercept of each text bounding box comprises: The vertical intercepts are clustered to obtain M clustering results; For any clustering result, the text bounding box corresponding to the vertical intercept in the clustering result is taken as the same first sorting row, and the text bounding box corresponding to each first sorting row is determined.

5. A case information extraction order prediction device characterized by comprising: The extraction sequence prediction device comprises: An acquisition module is configured to acquire a case to be extracted, perform text recognition on the case to be extracted, extract N text bounding boxes in the case to be extracted, and extract text content in each text bounding box, where N is an integer greater than 1. A first sorting module is configured to perform first sorting on the N text bounding boxes according to coordinate information in the N text bounding boxes to obtain text bounding boxes corresponding to M first sorting rows and text content in the corresponding text bounding boxes, where M is an integer less than or equal to N. The performing first sorting on the N text bounding boxes according to the coordinate information in the N text bounding boxes to obtain text bounding boxes corresponding to M first sorting rows comprises: The coordinate information in each text bounding box is used to obtain sorting results of a plurality of original rows. For any original row, the number of text bounding boxes in the sorting result of the corresponding original row is determined, and if the number is greater than a preset threshold, the original row is determined as a high-confidence row. All original rows are traversed to obtain K high-confidence rows, where K is an integer greater than 1. For any high-confidence row, the slope of the high-confidence row is calculated according to the coordinate information of each text bounding box in the high-confidence row, and K slopes are obtained by traversing the K high-confidence rows. A mean slope is obtained by calculating the mean of the K slopes, and the vertical intercept of each text bounding box is calculated according to the mean slope and the coordinate information in the text bounding box. The text bounding box corresponding to each first sorting row is determined according to the vertical intercept of each text bounding box to obtain text bounding boxes corresponding to M first sorting rows. A sliding module is configured to perform window sliding on any first sorting row according to a preset window size and a preset step size, and obtain a text bounding box sequence corresponding to each window sliding at each window sliding. The prediction module is configured to select a text bounding box in each sequence of text bounding boxes corresponding to a window slide as a basic text bounding box, construct a paired text bounding box paired with the basic text bounding box, perform relationship prediction on each paired text bounding box by using the trained prediction model, and obtain a prediction result of each corresponding paired text bounding box; The second sorting module is configured to determine a target text bounding box adjacent to the basic text bounding box according to the prediction result of each corresponding paired text bounding box, perform second sorting on the text bounding boxes of each line according to the basic text bounding box and the target text bounding box, obtain sorted text bounding boxes, and determine an extraction order of the case to be extracted according to text content corresponding to the sorted text bounding boxes. The determination of the target text bounding box adjacent to the basic text bounding box according to the prediction result of each corresponding paired text bounding box includes: According to the prediction result of each corresponding paired text bounding box, the paired text bounding box corresponding to a maximum value of the prediction result is selected as a corresponding target paired text bounding box. According to the target paired text bounding box paired with the basic text bounding box, a corresponding text bounding box is determined as the target text bounding box adjacent to the basic text bounding box.

6. The fetch order prediction apparatus of claim 5, wherein, The obtaining module includes: A detection unit configured to perform text detection on the case to be extracted to obtain text bounding boxes representing text in the case to be extracted; An identification unit configured to perform text identification on the text bounding boxes to obtain text content of the text bounding boxes.

7. A terminal device, characterized by comprising: The terminal device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the extraction order prediction method according to any one of claims 1 to 4 when executing the computer program.

8. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: The computer program is executed by the processor to implement the extraction order prediction method according to any one of claims 1 to 4.

Images