Laboratory sheet identification method, identification system, equipment and medium

Abstract

The invention discloses a laboratory sheet identification method and system, equipment and a medium. The method comprises the following steps: extracting coordinate information of a text region from a to-be-detected laboratory sheet image; performing content identification on the text area in the to-be-detected laboratory sheet image according to the coordinate information of the text area to obtain text content of the text area; constructing an identification result of the to-be-detected laboratory sheet image according to the text content of the text area; responding to a query instruction, and determining selected text content corresponding to the instruction according to the instruction; determining text coordinate information of the selected text content; according to the text coordinate information of the selected text content, labeling the corresponding text area in the recognition result to obtain a labeling result, so that the laboratory sheet content can be accurately recognized and converted into editable text content, the image area corresponding to the text content can be visually displayed according to the user demand, the laboratory sheet recognition efficiency and accuracy are improved, and the user experience is improved. Meanwhile, the data checking process is simplified, and convenience is provided for user operation.

Description

Technical Field

[0001] This application relates to the field of medical information technology, and particularly to a method, system, device and medium for recognizing test reports. Background Art

[0002] In the process of collecting data through various Electronic Data Capture Systems (EDCs) in traditional clinical trials, the input of reports on various laboratory tests of the subjects is generally manually input by users or directly recognized using Optical Character Recognition (OCR). The recognized results are presented in the input form of the EDC in the same tabular format. Users need to conduct multi-dimensional comparisons between the input form in the EDC and the recognized result form to determine whether the automatic recognition is correct.

[0003] The existing technology only uses OCR technology to achieve text recognition and cannot perform data verification. The data recognition accuracy is not high. Moreover, in the current clinical trial process, in some institutions, the results of multiple test items are printed on one A4 paper for the paper reports of the subjects. This leads to the appearance of other interfering information in the photographed photos of the paper test reports, resulting in recognition errors. In addition, due to the large number of domestic Laboratory Information Management System (LIS) manufacturers, the naming methods for the same type of test items by major manufacturers are diverse, further leading to a decrease in the correct rate of test report recognition. Therefore, there are technical problems of simple recognition results and low recognition accuracy. Summary of the Invention

[0004] The following is an overview of the subject matter described in detail in this article. This overview is not intended to limit the scope of protection of the claims.

[0005] The main objective of the embodiments of the present disclosure is to propose a method, system, device and storage medium for recognizing test reports, which can accurately recognize the content of test reports and improve the recognition efficiency and accuracy of test reports.

[0006] The first aspect of the embodiments of this application provides a method for recognizing test reports for a central controller. The method includes:

[0007] Extracting the coordinate information of the text area from the test report image to be detected;

[0008] Performing content recognition on the text area in the test report image to be detected according to the coordinate information of the text area to obtain the text content of the text area;

[0009] Construct the recognition result of the test report form image to be detected according to the text content of the text area;

[0010] In response to a query instruction, determine the selected text content corresponding to the instruction according to the instruction;

[0011] Determine the text coordinate information of the selected text content;

[0012] According to the text coordinate information of the selected text content, label the corresponding text area in the recognition result to obtain a labeled result.

[0013] The embodiment of the present application provides a method for recognizing a test report form. By extracting the coordinate information of the text area from the test report form image to be detected; performing content recognition on the text area in the test report form image according to the coordinate information of the text area to obtain the text content of the text area; constructing the recognition result of the test report form image to be detected according to the text content of the text area; in response to a query instruction, determining the selected text content corresponding to the instruction according to the instruction; determining the text coordinate information of the selected text content; according to the text coordinate information of the selected text content, labeling the corresponding text area in the recognition result to obtain a labeled result, it can not only accurately recognize the content of the test report form and convert it into editable text content, but also intuitively display the image area corresponding to the text content according to the user's needs, improve the efficiency and accuracy of test report form recognition, simplify the data verification process at the same time, and provide convenience for user operation.

[0014] In some embodiments of the present application, the performing content recognition on the text area in the test report form image according to the coordinate information of the text area to obtain the text content of the text area includes:

[0015] Crop the text area from the test report form image to be detected according to the coordinate information;

[0016] Extract the image features of the text area from the text area;

[0017] Perform mapping processing on the image features to obtain the text content of the text area.

[0018] In some embodiments of the present application, the constructing the recognition result of the test report form image to be detected according to the text content of the text area includes:

[0019] Perform semantic analysis on the text content to obtain a text analysis result;

[0020] Construct the recognition result of the test report form image to be detected according to the text analysis result, and the recognition result of the test report form image to be detected at least includes a basic information form and an inspection item table form.

[0021] In some embodiments of the present application, constructing the recognition result of the test report form image to be detected according to the text analysis result includes:

[0022] Obtain a test report form sample;

[0023] Establish a dynamic library of test report form mappings based on the test report form sample;

[0024] Based on the dynamic library of test report form mappings, map the text analysis result to obtain a mapped recognition result;

[0025] Construct the recognition result of the test report form image to be detected according to the mapped recognition result.

[0026] In some embodiments of the present application, extracting the coordinate information of the text region from the test report form image to be detected includes:

[0027] Extract multi-scale feature maps from the test report form image to be detected;

[0028] Generate a text probability map of the test report form image to be detected according to the multi-scale feature maps;

[0029] Based on the text probability map, extract the coordinate information of the text region from the test report form image to be detected.

[0030] In some embodiments of the present application, based on the text probability map, extracting the coordinate information of the text region from the test report form image to be detected includes:

[0031] Convert the text probability map through a preset activation function to obtain a binary result of the text probability map;

[0032] Extract the coordinate information from the test report form image to be detected according to the binary result.

[0033] In some embodiments of the present application, the formula for calculating the binary result of the text probability map includes:

[0034]

[0035] Wherein, is the binary result of the text probability map, P is the text probability map, T is a preset threshold, and k is the magnification factor of the text probability map.

[0036] To achieve the above object, a second aspect of the embodiments of the present invention provides a test report form recognition system, and the system includes:

[0037] An extraction module, configured to extract the coordinate information of the text region from the test report form image to be detected;

[0038] An identification module, configured to identify the content of the text area in the to-be-detected medical examination form image according to the coordinate information of the text area, so as to obtain the text content of the text area;

[0039] A construction module, configured to construct an identification result of the to-be-detected medical examination form image according to the text content of the text area;

[0040] A response module, configured to respond to a query instruction, and determine the selected text content corresponding to the instruction according to the instruction;

[0041] A selection module, configured to determine the text coordinate information of the selected text content;

[0042] A marking module, configured to mark the corresponding text area in the identification result according to the text coordinate information of the selected text content, so as to obtain a marking result.

[0043] To achieve the above object, a third aspect of the embodiments of the present invention provides an electronic device, including: at least one control processor and a memory communicatively connected to the at least one control processor; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor, so that the at least one control processor can execute the above-mentioned medical examination form identification method.

[0044] To achieve the above object, a fourth aspect of the embodiments of the present invention provides a computer-readable storage medium, where the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to cause a computer to execute the above-mentioned medical examination form identification method.

[0045] It can be understood that the beneficial effects of the above-mentioned second aspect to the fourth aspect compared with the related art are the same as those of the above-mentioned first aspect compared with the related art. For the relevant descriptions, reference can be made to the relevant descriptions in the above-mentioned first aspect, and details are not described herein again. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The above and / or additional aspects and advantages of the present application will become apparent and be easily understood from the following description of the embodiments in conjunction with the accompanying drawings, where:

[0047] Figure 1 is a schematic flowchart of a medical examination form identification method provided by an embodiment of the present application;

[0048] Figure 2 is a first medical examination form example diagram provided by an embodiment of the present application;

[0049] Figure 3 is a second medical examination form example diagram provided by an embodiment of the present application;

[0050] Figure 4 It is the third example diagram of the test report form provided by the embodiments of the present application;

[0051] Figure 5 It is a display example diagram provided by the embodiments of the present application;

[0052] Figure 6 It is another display example diagram provided by the embodiments of the present application;

[0053] Figure 7 It is an example diagram of a collected image provided by the embodiments of the present application;

[0054] Figure 8 It is a schematic structural diagram of a test report form recognition training system provided by the embodiments of the present application;

[0055] Figure 9 It is a schematic hardware structure diagram of an electronic device provided by the embodiments of the present application. Detailed implementation manners

[0056] The embodiments of the present application will be described in detail below. The examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements with the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are only used to explain the present application and should not be construed as a limitation to the present application.

[0057] In the description of the present application, if the first, second, etc. are described, they are only for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features or implicitly indicating the sequence relationship of the indicated technical features.

[0058] In the description of the present application, it should be understood that the orientation descriptions such as up, down, etc. indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus should not be construed as a limitation to the present application.

[0059] In the description of the present application, it should be noted that unless otherwise clearly defined, words such as setting, installing, connecting, etc. should be understood in a broad sense, and those skilled in the art can reasonably determine the specific meanings of the above words in the present application in combination with the specific content of the technical solution.

[0060] In the current medical information field, mainly through medical and health service institutions such as hospitals and clinics, test reports are generated after testing samples such as the blood and urine of subjects, reflecting the functional states and pathological changes of various tissues and organs of the subjects, and providing a scientific basis for medical staff to evaluate the health status of the subjects.

[0061] In the process of data collection in traditional clinical trials using various electronic data capture systems (EDCs), the entry of reports on various laboratory tests of subjects is generally done manually by users or directly using optical character recognition (OCR). The recognized results are presented in the form of a table in the EDC entry form, and users need to conduct multi-dimensional comparisons between the entry form in the EDC and the recognized result table to determine whether the automatic recognition is correct.

[0062] Among them, for manual entry by users, they usually need to first find the corresponding test item in the test report according to the corresponding test item in the EDC page form, then analyze whether the name of the test item in the paper report corresponds to the test item name in the EDC system, and then enter the memorized value using the keyboard in the EDC system by memorizing the test item results in the paper report. Finally, manually check whether the entered data is consistent with the printed value in the paper report. The overall entry process is not only time-consuming but also error-prone.

[0063] Currently, the common practice in the medical field is to directly use OCR recognition, and the recognized results are presented in the form of a table in the EDC entry form. Ultimately, users need to conduct multi-dimensional comparisons between the entry form in the EDC and the recognized result table to determine whether the automatic recognition is correct.

[0064] Therefore, the existing technology essentially only uses OCR technology to achieve text recognition, and does not provide users with a fast and convenient verification method. Users still need to check by comparing multi-dimensional data up and down, which is not user-friendly and prone to errors.

[0065] In addition, during the current clinical trial process, some institutions of the subjects' paper reports print the results of multiple test items on one A4 paper, which leads to the appearance of other interfering information in the photographed paper report photos, causing recognition errors. Moreover, since there are many domestic laboratory information management system (LIS) manufacturers, and the naming methods for the same type of test items by major manufacturers are diverse, this further reduces the recognition accuracy of test reports.

[0066] Based on this, the embodiments of the present application provide a method for recognizing test reports, a recognition system, an electronic device, and a medium, aiming to accurately recognize the content of test reports and improve the recognition efficiency and accuracy of test reports.

[0067] The method for recognizing test reports, the recognition system, the electronic device, and the medium provided by the embodiments of the present application are specifically described through the following embodiments. First, the method for recognizing test reports in the embodiments of the present application is described.

[0068] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results.

[0069] Artificial intelligence basic technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. Artificial intelligence software technologies mainly include several major directions such as computer vision technology, robotics, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0070] The test report recognition method provided by the embodiments of the present application relates to the field of medical information technology. The test report recognition method provided by the embodiments of the present application can be applied to a terminal, can also be applied to a server side, or can also be software running on a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server side can be configured as an independent physical server, can also be configured as a server cluster or a distributed system composed of multiple physical servers, or can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the test report recognition method, etc., but is not limited to the above forms.

[0071] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet-type devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and so on. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment, where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0072] It should be noted that in each specific embodiment of the present application, when it comes to relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first. Moreover, the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiments of the present application need to obtain the user's sensitive personal information, the user's separate permission or separate consent will be obtained through methods such as pop-up windows or redirecting to a confirmation page. After clearly obtaining the user's separate permission or separate consent, the necessary user-related data for the normal operation of the embodiments of the present application will be obtained.

[0073] For this reason, referring to Figure 1 , the embodiments of the present application provide a test report form recognition method. This method is applied to a central controller. The controller can be a server, an electronic device, a mobile terminal, etc., and no specific limitation is made here. The method includes the following steps S110 to S160.

[0074] Step S110: Extract the coordinate information of the text area from the test report form image to be detected.

[0075] In this step, to extract the coordinate information of the text area from the test report form image to be detected, the test report form image to be detected can be preprocessed according to the subsequent data processing needs. Preferably, the size of the test report form image to be detected is adjusted as needed to ensure subsequent data processing.

[0076] Furthermore, perform grayscale or binarization operations on the test report form image to be detected, convert the color image into a grayscale image or further binarize it to reduce the amount of calculation and highlight the text features; use a filter (such as Gaussian blur) to remove the noise in the test report form image to be detected and improve the accuracy of text detection; enhance the contrast of the test report form image to be detected through histogram equalization or other techniques to make the text clearer.

[0077] Furthermore, effectively identify the text area in a complex background through a text detection model, such as DBNet (Detection and Recognition Neural Network), PSENet (Progressive Scale Expansion Network), EAST (Efficient and Accurate Scene Text Detector), etc.

[0078] In some embodiments, the preprocessed image is input into a trained text detection model, which outputs a series of bounding boxes including one or more sets of coordinate points. Each bounding box corresponds to a detected text region, and its coordinate points represent the position of the bounding box of each text region. For a rectangular bounding box, it generally includes the coordinates of the upper left corner and the lower right corner; for a polygonal bounding box, it may contain the coordinates of four vertices.

[0079] In addition, to eliminate bounding boxes with excessive overlap, the Non-Maximum Suppression (NMS) algorithm is applied to retain the bounding boxes that are most likely to represent independent text instances.

[0080] In some embodiments, first, a multi-scale feature map is extracted from the image of the test report to be detected by using a Convolutional Neural Network (CNN). Then, a sub-network for semantic segmentation (such as a U-Net structure) is applied to generate a text probability map of the image of the test report to be detected based on the multi-scale feature map. Based on the text probability map, the coordinate information of the text region is extracted from the image of the test report to be detected, providing accurate data for subsequent test report recognition and other processing steps, improving the efficiency and accuracy of text recognition, accelerating the data entry process, and reducing the error rate.

[0081] Step S120: Recognize the content of the text region in the image of the test report to be detected according to the coordinate information of the text region, and obtain the text content of the text region.

[0082] In this step, it is preferably to use a text recognition model to recognize each text region. In this embodiment, the Convolutional Recurrent Neural Network (CRNN) algorithm, which combines the characteristics of a Convolutional Neural Network (CNN) for feature extraction and a Recurrent Neural Network (RNN) for processing sequential data, is used to read continuous text information from the image, so as to recognize the text content of the text region.

[0083] Specifically, in the use of the CRNN algorithm in OCR, CRNN combines a Convolutional Neural Network (CNN) and a Recurrent Neural Network (RNN), and is widely used in Optical Character Recognition (OCR) tasks, aiming to process sequential data, especially text sequences in images. CRNN extracts image features through convolutional layers, and then uses recurrent layers (usually LSTM or GRU) to process these feature sequences, thereby realizing text recognition.

[0084] In some embodiments, the content of the text region in the test report image to be detected is recognized by a CRNN text recognition model. First, the text region image is preprocessed, adjusted to a fixed size according to subsequent data processing requirements, and normalized for easy network processing. Then, the text region image is processed by a convolutional layer to generate a feature map containing spatial information of the text.

[0085] Further, the output of the feature map is converted into a sequence format for input into the RNN. Then, the feature sequence is input into the RNN, and the long short-term memory network (LSTM) or gated recurrent unit (GRU) is used to process the sequence data to capture the context information of the text.

[0086] Further, a probability distribution of characters is generated through a fully connected layer, representing the prediction probability of each character at the current time step. Finally, the CTC (Connectionist Temporal Classification) loss function is used to calculate the difference between the prediction result and the true label, and the network parameters are updated through the backpropagation algorithm.

[0087] Step S130: Construct the recognition result of the test report image to be detected according to the text content of the text region.

[0088] In this step, constructing the recognition result of the test report image to be detected according to the text content of the text region specifically includes first performing semantic analysis on the text content to obtain the text analysis result. Then, the test report sample is obtained, and a test report mapping dynamic library is established based on the test report sample. Furthermore, based on the test report mapping dynamic library, the text analysis result is mapped to obtain the mapping recognition result. Finally, the recognition result of the test report image to be detected is constructed according to the mapping recognition result.

[0089] Step S140: Respond to the query instruction and determine the selected text content corresponding to the instruction.

[0090] In this step, when responding to the instruction that the user needs to query, the user's query instruction is parsed, and the selected text content corresponding to the instruction is determined according to the parsing result, such as specific keywords and field names, etc.

[0091] Step S150: Determine the text coordinate information of the selected text content.

[0092] In this step, for each selected text content, the previously saved text region coordinate information (i.e., the bounding box coordinates recorded when extracting text from the image) is used to determine the position in the test report image to be detected corresponding to the text.

[0093] Step S160: According to the text coordinate information of the selected text content, label the corresponding text area in the recognition result to obtain the labeling result.

[0094] In this step, according to the text coordinate information of the selected text content, draw a rectangular box or other shapes on the image of the test report to be detected or its copy to label the selected text area. In one embodiment, a short text description can be added next to the label box to indicate that the label corresponds to the specific text content selected according to the user's instruction, and the final labeling result is generated.

[0095] In some embodiments, in step S120, content recognition is performed on the text area in the image of the test report to be detected according to the coordinate information of the text area, and the text content of the text area is obtained, including the following steps S210 to S230.

[0096] Step S210: Crop the text area from the image of the test report to be detected according to the coordinate information;

[0097] Step S220: Extract the image features of the text area from the text area;

[0098] Step S230: Perform mapping processing on the image features to obtain the text content of the text area.

[0099] In this embodiment, for the subsequent recognition requirements of the text area, it is necessary to crop the text area in the image of the test report to be detected, which can be cropped by using an image processing library (such as OpenCV or PIL / Pillow) to facilitate defining the position and size of the cropping window. In addition, for each cropped text area, various image preprocessing techniques can be applied to enhance the OCR effect, such as grayscale conversion, binarization, denoising, skew correction, etc.

[0100] Further, extract the image features of the text area from the text area, and use an OCR tool (such as Tesseract) to map the image features to the corresponding text content to obtain the text content of the text area.

[0101] In some embodiments, in step S130, the recognition result of the image of the test report to be detected is constructed according to the text content of the text area, including the following steps S310 to S320.

[0102] Step S310: Perform semantic analysis on the text content to obtain the text analysis result;

[0103] Step S320: Construct the recognition result of the image of the test report to be detected according to the text analysis result. The recognition result of the image of the test report to be detected includes at least the basic information form and the inspection item table form.

[0104] In this embodiment, it is preferably to perform semantic analysis on the text content through natural language processing (NLP) technology. Specifically, tools such as the natural language processing library spaCy and the natural language toolkit NLTK can be used to tokenize and perform part-of-speech tagging on the extracted text content, so as to understand the role of each word and its position in the sentence, and obtain the text analysis result.

[0105] Furthermore, obtain the test report sample, establish a dynamic test report mapping library according to the test report sample, and then, based on the dynamic test report mapping library, map the text analysis result to obtain the mapping recognition result. Then, by obtaining the test report sample to construct a dynamic test report mapping library, map the identified key information (such as patient name, gender, age, sample number, etc.) to the predefined basic information form fields, or for information with a fixed format (such as phone number, ID number, etc.), regular expressions can be written to perform efficient and accurate matching. Finally, obtain the recognition result of the test report image to be detected in at least the form of a basic information form and an inspection item table.

[0106] In some embodiments, in step S320, constructing the recognition result of the test report image to be detected according to the text analysis result includes the following steps S410 to S440.

[0107] Step S410: Obtain the test report sample;

[0108] Step S420: Establish a dynamic test report mapping library according to the test report sample;

[0109] Step S430: Based on the dynamic test report mapping library, map the text analysis result to obtain the mapping recognition result;

[0110] Step S440: Construct the recognition result of the test report image to be detected according to the mapping recognition result.

[0111] In this embodiment, obtain different types of test report samples from hospitals or laboratories, including but not limited to various common and special format samples such as blood tests, urine analyses, and biochemical tests. Then establish a dynamic test report mapping library according to the test report sample, which specifically includes designing mapping rules and determining mapping methods.

[0112] Specifically, according to the collected samples, define a standardized field structure covering all possible information types that may appear on the test report (such as patient information, sample information, inspection items and their results, etc.), and then create a mapping table for each test report format to correspond the actual text content with the predefined field structure one by one.

[0113] Specifically, for the mapping method of constructing the mapping table, an automated tool can be developed using Python or other programming languages to read the newly collected test report samples and automatically identify and fill in the corresponding fields based on the existing mapping table. When there are multiple complex and non-fixed formats, machine learning models (such as classifiers, sequence models, etc.) can also be introduced to determine the best-matching mapping rules.

[0114] Furthermore, based on the test report mapping dynamic library, perform text analysis result mapping, fill the text content into the corresponding fields according to the mapping rules, and construct the recognition result of the test report image to be detected. Integrate the mapped basic information (such as patient name, age, gender, etc.) into a structured form, and for the inspection item part, create a table containing information such as item name, measurement value, unit, and reference range, and finally form the recognition result of the test report image to be detected.

[0115] In some embodiments, such as Figure 2 、 Figure 3 and Figure 4 shown, domestic LIS manufacturers have different output names for the naming methods of laboratory inspection items according to their respective designs. For example, for white blood cell count, there are three names: [WBC] White Blood Cells, *White Blood Cell Count, and White Blood Cell Count ().

[0116] Therefore, in this embodiment, by establishing a test report mapping dynamic library and providing multiple different naming mapping configurations to solve the problem of decreased recognition rate for different names, taking white blood cell count as the key, the mapping configuration is shown in Table 1 below:

[0117] Table 1

[0118]

[0119] In some embodiments, the test report image to be detected is not limited to the photos directly taken by the image processing device equipped with the test report recognition system (such as a visual intelligent monitor, intelligent terminal device), but also includes the pictures taken by other external devices (such as network cameras, portable cameras, etc.) and then imported or synchronized to the image processing device. In addition, the test report image to be detected may also come from a cloud storage platform and directly reach the image processing device through network transmission for image processing. Similarly, these images may also be taken by independent imaging devices, such as high-end digital single-lens reflex cameras or professional video cameras, in different environments and then uploaded to the image processing device through wireless transmission (Wi-Fi, Bluetooth, etc.), data cable transmission, etc.

[0120] In some embodiments, the user can use a peripheral device: a high-speed document scanner to take pictures of the test report to be detected to obtain a high-definition test report image to be detected.

[0121] Further, use a text detection model (such as DBNet, etc.) to process the input image, detect the coordinates of all text regions. In this embodiment, it is preferably to perform text detection through a text detection algorithm (DBNet network), identify all regions containing text in the image, and determine the specific coordinates of these regions. DBNet is trained through deep learning technology, which can efficiently locate the text position in the picture and accurately find the text even in complex backgrounds or irregular arrangements. Furthermore, it can achieve precise segmentation of the text region through differentiable binarization operations.

[0122] Further, use a text recognition model (such as PaddleOcr, ConvNextViT, CRNN, etc.) to recognize each detected text region. Specifically, it includes first simply reconstructing the text image according to the coordinates, then analyzing and marking the text images of all text blocks through semantic analysis. According to the analysis results, reconstruct the checklist into a basic information form and an inspection item table, extract the table information according to the required items of the EDC system, and display it side by side with the inspection items in the EDC system. Compare and adjust multiple two-dimensional tables to the comparison method of the original image result column and the recognition result column.

[0123] In this embodiment, it is preferably that CRNN combines a convolutional neural network (CNN) for feature extraction and a recurrent neural network (RNN) to process sequence data, which is used to read continuous text information from the image. It can not only recognize isolated characters but also correctly understand text content in rows or paragraphs.

[0124] In some embodiments, extracting the coordinate information of the text region from the image of the test checklist to be detected in step S110 includes the following steps S510 to S530.

[0125] Step S510: Extract multi-scale feature maps from the image of the test checklist to be detected;

[0126] Step S520: Generate a text probability map of the image of the test checklist to be detected according to the multi-scale feature maps;

[0127] Step S530: Based on the text probability map, extract the coordinate information of the text region from the image of the test checklist to be detected.

[0128] In this embodiment, text detection is performed through a text detection algorithm (DBNet network), aiming to achieve precise segmentation of the text region through differentiable binarization operations.

[0129] Specifically, first, preprocess the input image, adjust the input image to a fixed size, and perform normalization processing for easy network processing. Then, use a convolutional neural network (such as ResNet, VGG, etc.) to extract features from the test report image to be detected, extract multi-level features of the test report image to be detected, and obtain a multi-scale feature map. Among them, the feature map contains the spatial information and context information of the image, which helps with subsequent text region detection.

[0130] Furthermore, based on the multi-scale feature map, use a dynamic threshold through the differentiable binarization module in the DBNet network to convert the multi-scale feature map into a binary image. Among them, the dynamic threshold is adaptively calculated according to the local information of the feature map, which can better handle complex backgrounds and text with different brightness levels.

[0131] Further, use a convolutional layer to post-process the binarization result. Specifically, connect and aggregate adjacent binary regions to form a complete text box, and finally generate the final text region prediction. In addition, during the training process, use an appropriate loss function (such as cross-entropy loss) to evaluate the difference between the model's prediction result and the true annotation. Then, update the network parameters through the backpropagation algorithm to improve the detection accuracy. Finally, post-process the output of the network, such as non-maximum suppression (NMS), to remove redundant prediction boxes and retain the optimal text regions.

[0132] In one implementation manner of this embodiment, extract features through a convolutional neural network (CNN). By constructing a feature pyramid network, feature maps at different levels can be obtained, capturing information at different scales. Moreover, different-sized convolutional kernels can be used at different stages to capture features at different scales. For example, use smaller convolutional kernels in the shallow layer to capture fine-grained details, and use larger convolutional kernels in the deep layer to obtain context information in a larger range.

[0133] Furthermore, adopt a spatial pyramid pooling layer to further enhance the model's adaptability to inputs of different scales, enabling the model to process input images of any size without losing important feature information. After obtaining the multi-scale feature map, through a semantic segmentation sub-network (such as the U-Net structure), convert the feature map into the probability value that each pixel point belongs to the text.

[0134] Then, according to the generated text probability map, set an appropriate threshold to convert the continuous probability values into discrete binary values (0 or 1), thereby obtaining a clear text / non-text region division. Preferably, by adopting the differentiable binarization method in DBNet, directly optimize the result after binarization during the training process to ensure that the boundary is sharper and more accurate.

[0135] Furthermore, by using the contour detection function provided by libraries such as OpenCV, all connected text region contours are extracted from the binarized text probability map, and the orientation and shape of the text box can be adjusted through an additional geometric transformation module to better fit the actual text layout.

[0136] In addition, to remove overly overlapping candidate boxes, the NMS algorithm is applied to retain the bounding boxes most likely to represent independent text instances and remove those with too small an area or abnormal shapes. Finally, the coordinates of the bounding boxes of the determined text regions in the image of the test report to be detected are output for subsequent test report recognition or other processing flows. In this embodiment, the detected text region bounding boxes can be drawn on the image of the test report to be detected according to the coordinates to visually check the detection results and perform necessary manual corrections or verifications.

[0137] In some embodiments, in step S530, based on the text probability map, the coordinate information of the text region is extracted from the image of the test report to be detected, including the following steps S630 to S631.

[0138] Step S630: Convert the text probability map through a preset activation function to obtain the binarized result of the text probability map;

[0139] Step S631: Extract the coordinate information from the image of the test report to be detected according to the binarized result.

[0140] In this embodiment, the continuous probability values (real numbers between 0 and 1) are converted into binary values (0 or 1) through a preset activation function, preferably the Sigmoid function and the hyperbolic tangent function (Tanh).

[0141] In an implementation manner of this embodiment, a threshold is adaptively calculated and preset according to the local information of the feature map, and the multi-scale feature map is converted into a binary image through the preset threshold.

[0142] Among them, the formula for calculating the binarized result of the text probability map includes:

[0143]

[0144] Among them, is the binarized result of the text probability map, P is the text probability map, T is the preset threshold, and k is the magnification factor of the text probability map.

[0145] Further, by using the findContours function of a computer vision library such as OpenCV, all connected text regions are found from the binary image, and the contours of the text regions are returned. Each contour is a polygon containing multiple point coordinates, defining the boundary of the text region. For the contour of each detected text region, the minimum bounding rectangle of the text region is calculated to obtain the exact bounding box coordinates of the text region.

[0146] In an implementation manner of this embodiment, the detected bounding box can be drawn on the image of the test report form to be detected, so as to facilitate the user to visually verify the detection effect.

[0147] In some embodiments, such as Figure 5 and Figure 6 shown, after the user uploads the image of the test report form to be detected taken by themselves, through the test report form recognition method, a large original image can be provided at the position shown in Figure 5 shown, and five columns of items, original image results, recognition results, reference ranges, and units are displayed in the recognition result table shown in Figure 6 shown.

[0148] Among them, the "Items" column displays the data content: the ordinary text style is the name of the test item in the EDC system, and the text displayed by the blue label is the name of the test item recognized by the test report form recognition method; the "Original Image Results" column is the small image of the text region result after cropping the text region according to the text coordinate information recognized from the text region; the content in the input box of the "Recognition Results" column is the value after text recognition based on the small image of the text region result, and it supports the user to perform manual verification according to the image of the test report form to be detected so as to quickly correct it.

[0149] Further, manual verification can be further performed according to the recognition result of the image of the test report form to be detected. During the verification process, a specified item can be selected for verification. When the user selects a certain item in the Figure 6 recognition result, Figure 5 the large image will respond to the user's operation and mark the corresponding item image, which is convenient for visually verifying the data. After the verification is confirmed, the verified recognition result is filled into the corresponding data area in the EDG system to obtain the final recognition result of the test report form to be detected.

[0150] Thus, the user only needs to quickly scan and check the two columns of "Original Image Results" and "Recognition Results" to determine whether the OCR recognition result is correct, solving the problem of inconvenient user verification method and improving the recognition efficiency.

[0151] Further, when responding to the user's need to determine the position of the recognized text area, according to the selection instruction of the user clicking on any row in the recognition result table, parse the instruction selected by the user, determine the selected text content corresponding to the instruction according to the parsing result, and mark the text area in the image of the test report to be detected uploaded on the left side of the user page according to the text coordinate information of the text content, so as to meet the user's need to quickly locate the text area.

[0152] In some embodiments, as Figure 7 shown, the user can use a peripheral device: a high-speed camera. After uploading, the user can select correction or rotation operations according to the picture situation to make the uploaded image better. In addition, when there are multiple test reports with the same name inspection items in the taken and uploaded pictures, the user is supported to manually crop and frame the necessary information area to remove redundant interference information. For example, when there are two "color" inspection items in the image of the test report to be detected, the user can select by cropping: manual cropping, combined with using the mouse to frame the upper part of the picture, and the picture cropping can be quickly completed. The user can also select automatic recognition and cropping to obtain the cropped result.

[0153] In this embodiment, by extracting the coordinate information of the text area from the image of the test report to be detected; performing content recognition on the text area in the image of the test report to be detected according to the coordinate information of the text area to obtain the text content of the text area; constructing the recognition result of the image of the test report to be detected according to the text content of the text area; responding to the query instruction, determining the selected text content corresponding to the instruction according to the instruction; determining the text coordinate information of the selected text content; and marking the corresponding text area in the recognition result according to the text coordinate information of the selected text content to obtain the marked result, not only can the content of the test report be accurately recognized and converted into editable text content, but also the image area corresponding to the text content can be intuitively displayed according to the user's needs, improving the recognition efficiency and accuracy of the test report, while simplifying the data verification process and providing convenience for the user's operation.

[0154] As Figure 8 shown, some embodiments of the present application provide a test report recognition system. The system includes an extraction module 810, a recognition module 820, a construction module 830, a response module 840, a selection module 850, and a marking module 860. Specifically:

[0155] The extraction module 810 is used to extract the coordinate information of the text area from the image of the test report to be detected;

[0156] The recognition module 820 is used to perform content recognition on the text area in the image of the test report to be detected according to the coordinate information of the text area to obtain the text content of the text area;

[0157] The construction module 830 is used to construct the recognition result of the test report form image to be detected according to the text content of the text area;

[0158] The response module 840 is used to respond to the query instruction and determine the selected text content corresponding to the instruction according to the instruction;

[0159] The selection module 850 is used to determine the text coordinate information of the selected text content;

[0160] The annotation module 860 is used to annotate the corresponding text area in the recognition result according to the text coordinate information of the selected text content to obtain the annotation result.

[0161] In some embodiments, the recognition module 820 may include: cropping out the text area from the test report form image to be detected according to the coordinate information.

[0162] In some embodiments, the recognition module 820 may include: extracting the image features of the text area from the text area.

[0163] In some embodiments, the recognition module 820 may include: performing mapping processing on the image features to obtain the text content of the text area.

[0164] In some embodiments, the construction module 830 may include: performing semantic analysis on the text content to obtain the text analysis result.

[0165] In some embodiments, the construction module 830 may include: constructing the recognition result of the test report form image to be detected according to the text analysis result, and the recognition result of the test report form image to be detected at least includes the basic information form and the inspection item table form.

[0166] In some embodiments, the construction module 830 may include: obtaining the test report form sample.

[0167] In some embodiments, the construction module 830 may include: establishing a test report form mapping dynamic library according to the test report form sample.

[0168] In some embodiments, the construction module 830 may include: mapping the text analysis result based on the test report form mapping dynamic library to obtain the mapping recognition result.

[0169] In some embodiments, the construction module 830 may include: constructing the recognition result of the test report form image to be detected according to the mapping recognition result.

[0170] In some embodiments, the extraction module 810 may include: extracting multi-scale feature maps from the test report form image to be detected.

[0171] In some embodiments, the extraction module 810 may include: generating a text probability map of the test report image to be detected according to the multi-scale feature map.

[0172] In some embodiments, the extraction module 810 may include: based on the text probability map, extracting the coordinate information of the text region from the test report image to be detected.

[0173] In some embodiments, the extraction module 810 may include: converting the text probability map through a preset activation function to obtain a binary result of the text probability map.

[0174] In some embodiments, the extraction module 810 may include: extracting the coordinate information from the test report image to be detected according to the binary result.

[0175] In some embodiments, the extraction module 810 may include:

[0176]

[0177] Wherein, is the binary result of the text probability map, P is the text probability map, T is the preset threshold, and k is the magnification factor of the text probability map.

[0178] It should be noted that the test report recognition system provided in this embodiment and the above-mentioned test report recognition method are based on the same inventive concept. Therefore, the relevant content of the above-mentioned test report recognition method also applies to the content of the test report recognition system. Therefore, it will not be elaborated here.

[0179] To solve the technical problems of simple recognition results and low recognition accuracy in the prior art, the system extracts the coordinate information of the text region from the test report image to be detected; recognizes the content of the text region in the test report image to be detected according to the coordinate information of the text region to obtain the text content of the text region; constructs the recognition result of the test report image to be detected according to the text content of the text region; responds to the query instruction, determines the selected text content corresponding to the instruction according to the instruction; determines the text coordinate information of the selected text content; and marks the corresponding text region in the recognition result according to the text coordinate information of the selected text content to obtain the marked result. In this way, it can not only accurately recognize the content of the test report and convert it into editable text content, but also intuitively display the image region corresponding to the text content according to the user's needs, improve the recognition efficiency and accuracy of the test report, and simplify the data verification process, providing convenience for user operation.

[0180] The embodiment of the present application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the above-mentioned test report recognition method is implemented.

[0181] Such asFigure 9 , Figure 9 is a schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present application. The electronic device includes:

[0182] at least one battery;

[0183] at least one memory;

[0184] at least one processor;

[0185] at least one program;

[0186] The program is stored in the memory, and the processor executes at least one program to implement the method for identifying a test report form as described above in the present disclosure.

[0187] The electronic device can be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), an in-vehicle computer, etc.

[0188] The following provides a detailed introduction to the electronic device according to the embodiment of the present application.

[0189] The processor 1600 can be implemented by using a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is used to execute relevant programs to implement the technical solutions provided by the embodiments of the present disclosure;

[0190] The memory 1700 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory 1700 can store an operating system and other application programs. When implementing the technical solutions provided by the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 1700 and are called by the processor 1600 to execute a method for identifying a test report form in the embodiments of the present disclosure.

[0191] The input / output interface 1800 is used to implement information input and output;

[0192] The communication interface 1900 is used to implement communication interaction between this device and other devices, and can implement communication through a wired method (such as USB, network cable, etc.) or through a wireless method (such as mobile network, WIFI, Bluetooth, etc.);

[0193] The bus 2000 transmits information between various components of the device (such as the processor 1600, the memory 1700, the input / output interface 1800, and the communication interface 1900).

[0194] Among them, the processor 1600, the memory 1700, the input / output interface 1800, and the communication interface 1900 are communicatively connected to each other inside the device through the bus 2000.

[0195] The embodiments of the present disclosure also provide a storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores computer-executable instructions for causing a computer to execute the above-mentioned method for identifying a test report form.

[0196] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0197] The embodiments described in the embodiments of the present disclosure are for more clearly illustrating the technical solutions of the embodiments of the present disclosure, and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Those skilled in the art will know that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present disclosure are equally applicable to similar technical problems.

[0198] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present disclosure, and may include more or fewer steps than those shown in the figures, or combine certain steps, or different steps.

[0199] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0200] Those of ordinary skill in the art can understand that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices can be implemented as software, firmware, hardware, and appropriate combinations thereof.

[0201] In the description of the present application and the above-mentioned accompanying drawings, terms such as "first", "second", "third", "fourth", etc. (if any) are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of the present application described herein can be implemented in an order different from those illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that comprises a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0202] It should be understood that in the present application, "at least one (item)" means one or more, and "a plurality" means two or more. "And / or" is used to describe the association relationship of associated objects and indicates that there can be three relationships. For example, "A and / or B" can mean: only A exists, only B exists, and both A and B exist at the same time. Among them, A and B can be singular or plural. The character " / " generally means that the associated objects before and after are in an "or" relationship. "At least one (one) of the following" or its similar expression means any combination of these items, including any combination of single item (one) or plural items (ones). For example, at least one (one) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0203] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of devices or units can be in electrical, mechanical, or other forms.

[0204] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place, or can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0205] In addition, in each embodiment of the present application, the functional units can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0206] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes multiple instructions for causing an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in each embodiment of the present application. The foregoing storage medium includes: various media that can store programs, such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

[0207] The above is a specific description of the preferred implementation of the embodiments of the present application. However, the embodiments of the present application are not limited to the above implementation manners. Those skilled in the art can make various equivalent deformations or substitutions without departing from the spirit of the embodiments of the present application, and these equivalent deformations or substitutions are all included within the scope defined by the claims of the embodiments of the present application.

[0208] The above has described the embodiments of the present application in detail with reference to the accompanying drawings. However, the present application is not limited to the above embodiments, and various changes can be made without departing from the purpose of the present application within the knowledge scope of those of ordinary skill in the art.

Claims

1. A method for identifying a test report, characterized in that: The method comprises: Extracting coordinate information of the text area from the test sheet image to be detected; Performing content recognition on the text area in the test order image to be detected according to the coordinate information of the text area to obtain the text content of the text area; Constructing a recognition result of the test order image to be detected according to the text content of the text area; In response to the query instruction, determine the selected text content corresponding to the instruction according to the instruction; Determining text coordinate information of the selected text content; According to the text coordinate information of the selected text content, the corresponding text area in the recognition result is marked to obtain a marking result.

2. The method for identifying a test report according to claim 1, characterized in that: The step of performing content recognition on the text area in the test order image to be detected according to the coordinate information of the text area to obtain the text content of the text area includes: According to the coordinate information, a text area is cut out from the test order image to be detected; Extracting image features of the text region from the text region; The image features are mapped to obtain text content in the text area.

3. The test report identification method according to claim 1, characterized in that: The step of constructing the recognition result of the test order image to be detected according to the text content of the text area includes: Performing semantic analysis on the text content to obtain a text analysis result; The recognition result of the test order image to be detected is constructed according to the text analysis result, and the recognition result of the test order image to be detected at least includes a basic information form and an inspection item table form.

4. The method for identifying a test report according to claim 3, characterized in that: The step of constructing the recognition result of the test order image to be detected according to the text analysis result includes: Obtain laboratory test sheet samples; Establishing a dynamic library of test order mapping according to the test order sample; Based on the test report mapping dynamic library, the text analysis result is mapped to obtain a mapping recognition result; The recognition result of the test order image to be detected is constructed according to the mapping recognition result.

5. The test report identification method according to claim 1, characterized in that: The step of extracting coordinate information of the text area from the test sheet image to be detected includes: Extracting multi-scale feature maps from the test order image to be tested; Generate a text probability map of the test order image to be detected according to the multi-scale feature map; Based on the text probability map, the coordinate information of the text area is extracted from the test order image to be detected.

6. The method for identifying a test report according to claim 5, characterized in that: The step of extracting the coordinate information of the text area from the test order image to be detected based on the text probability map includes: The text probability map is converted by a preset activation function to obtain a binarization result of the text probability map; The coordinate information is extracted from the test order image to be detected according to the binarization result.

7. The method for identifying a test report according to claim 6, characterized in that: The formula for calculating the binarization result of the text probability map includes: in, is the binarization result of the text probability map, P is the text probability map, T is the preset threshold, and k is the magnification factor of the text probability map.

8. A test report recognition system, characterized in that: The system comprises: An extraction module, used to extract the coordinate information of the text area from the test sheet image to be detected; A recognition module, used for performing content recognition on the text area in the test order image to be detected according to the coordinate information of the text area, and obtaining the text content of the text area; A construction module, used to construct a recognition result of the test order image to be detected according to the text content of the text area; A response module, used for responding to the query instruction and determining the selected text content corresponding to the instruction according to the instruction; A selection module, used to determine text coordinate information of the selected text content; The marking module is used to mark the corresponding text area in the recognition result according to the text coordinate information of the selected text content to obtain the marking result.

9. An electronic device, characterized in that: It includes at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute a test report identification method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute a test report identification method as described in any one of claims 1 to 7.

Images