Medical information input method and device, electronic equipment and readable storage medium

Abstract

The invention provides a medical information input method and device, electronic equipment and a readable storage medium, and the method comprises the steps: obtaining a medical document photo obtained by photographing characters in a medical document through a camera located in a ward; calling sound recording data and video recording data when the medical document is generated according to the shooting time information and the shooting position information of the medical document picture; extracting first document information from the medical document photo by using an OCR (Optical Character Recognition) technology; correcting the first document information based on the sound recording data and the video recording data; and inputting the corrected first document information into a database. Through the method, the accuracy of information input is improved.

Description

Technical Field

[0001] The present application relates to the field of medical technology, and in particular to a medical information entry method, device, electronic device and readable storage medium. Background Art

[0002] In the process of clinical medical treatment, data entry is an important step that is both time-consuming and prone to errors. The more traditional way is to manually enter case data into the CRF (Case Report Form). This method is not only labor-intensive, but also prone to errors during the data entry process, affecting data accuracy.

[0003] In order to improve the efficiency and accuracy of data entry, some technical teams have developed and started using OCR (Optical Character Recognition) technology for data entry. They obtain images of medical documents by scanning or photographing them, and then use OCR technology to perform text recognition on the images to obtain recognition results. Summary of the invention

[0004] In view of this, the purpose of the present application is to provide a medical information entry method, device, electronic device and readable storage medium to improve the accuracy of information entry.

[0005] In a first aspect, an embodiment of the present application provides a method for entering medical information, comprising: Obtaining a photo of the medical document obtained by photographing the text in the medical document through a camera located in the ward; According to the shooting time information and shooting location information of the medical document photo, calling the audio data and video data when the medical document was generated; Extracting first document information from the medical document photo using OCR text recognition technology; Correcting the first document information based on the audio data and the video data; The corrected first document information is entered into the database.

[0006] In combination with the first aspect, the embodiment of the present application provides a first possible implementation of the first aspect, wherein the camera is a portable camera provided on the doctor; and the step of obtaining a medical document photo obtained by photographing the text in the medical document with a camera located in the ward includes: Taking photos at predetermined intervals to obtain candidate photos; Performing foreground extraction on the candidate photo to extract a current foreground image in the candidate photo; the current foreground image includes a handheld medical document; Calculating the foreground image similarity between the current foreground image and the previously acquired historical foreground image; If the foreground image similarity is less than the preset similarity, the current foreground image is used as the medical document photo.

[0007] In combination with the first aspect, the embodiment of the present application provides a second possible implementation of the first aspect, wherein the calling of the audio data and video data when the medical document is generated according to the shooting time information and shooting location information of the medical document photo includes: Determine, based on the shooting location information of the medical document photo, a candidate room where the medical document photo was shot; Using visual positioning, the score of the reference image of each candidate room is calculated based on the medical document photos; Determine the candidate rooms whose scores exceed a preset value as target rooms; According to the shooting time information of the medical document photo, a target audio segment is selected from the video data of the target room as the audio data when calling the medical document generation, and a target video segment is selected as the video data when the medical document is generated.

[0008] In combination with the first aspect, the embodiment of the present application provides a third possible implementation of the first aspect, wherein the correcting the first document information based on the audio data and the video data includes: Performing speech recognition on the recorded data to obtain speech recognition text; Performing body movement analysis on the video data to obtain body movement feature data; Extracting reference second document information corresponding to the speech recognition text from a pre-established historical speech recognition text-reference second document information comparison table based on the speech recognition text; Extracting reference third document information corresponding to the body movement feature data from a pre-established historical body movement feature data-reference third document information comparison table based on the body movement feature data; The first document information is corrected based on the reference second document information and the reference third document information.

[0009] In combination with the second possible implementation of the first aspect, the embodiment of the present application provides a fourth possible implementation of the first aspect, wherein entering the corrected first document information into the database includes: If there are multiple corrected first document information, retrieve the medical information of the patient corresponding to the medical document photo; Displaying the medical information and the plurality of corrected first document information in a doctor terminal, so that one of the plurality of corrected first document information is selected as the final first document information according to a selection instruction received by the doctor terminal; The final first document information is stored in a database, and the historical speech recognition text-reference second document information comparison table and the historical body movement feature data-reference third document information comparison table are updated based on the corrected final first document information.

[0010] In a second aspect, the present application also provides a medical information input device, including: An acquisition module, used to acquire a photo of a medical document obtained by photographing the text in the medical document through a camera located in the ward; A calling module, used to call the audio data and video data when the medical document is generated according to the shooting time information and shooting location information of the medical document photo; An extraction module, used to extract first document information from the medical document photo using OCR text recognition technology; A correction module, used for correcting the first document information based on the audio data and the video data; The input module is used to input the corrected first document information into the database.

[0011] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps in any possible implementation of the first aspect above are performed.

[0012] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in any possible implementation of the first aspect described above are executed.

[0013] The medical information entry method provided in the embodiment of the present application corrects the first document information in text form extracted from the medical document photo by using the audio data and video data generated when taking the medical document photo, so that the accuracy of the text information originally directly entered into the database is further improved, thereby ensuring the accuracy of the information.

[0014] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 A flowchart of a medical information input method provided in an embodiment of the present application is shown; Figure 2 A flowchart showing another medical information input method provided in an embodiment of the present application is shown; Figure 3 A schematic diagram of the structure of a medical information input device provided in an embodiment of the present application is shown; Figure 4 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0017] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0018] In the related technology, when doctors are consulting with hospitalized patients, they usually record the information of their consultation, such as the patient's physical condition, recovery status, medication adjustment method, etc. After obtaining this information, the OCR technology can be used to enter the information written by the doctor into the system for subsequent viewing. However, when using OCR for text entry, misrecognition often occurs. There are two main reasons. First, when doctors are consulting patients in the ward, they do not record at the desk. They usually use the pad in their hands as a support and then write on the pad. In this way, the standard of writing will be much worse; second, doctors write faster and have more connected strokes, so the written text will be more casual, and it is difficult to accurately complete the recognition using general OCR recognition technology.

[0019] In view of the above situation, this application provides a method for entering medical information, such as Figure 1 As shown, the following steps are included: S101: Acquire a photo of a medical document by photographing text in the medical document using a camera located in a ward; S102: According to the shooting time information and shooting location information of the medical document photo, call the audio data and video data when the medical document is generated; S103: extracting first document information from the medical document photo using OCR text recognition technology; S104: Correcting the first document information based on the audio data and the video data; S105: Enter the corrected first document information into the database.

[0020] The application scenario of the solution provided in this application is a scenario where a doctor conducts inspections in various wards. In this scenario, the doctor needs to write on medical documents anytime and anywhere, so the standardization of the writing is low.

[0021] In step S101, the camera can be carried by the doctor or installed at a fixed position in the ward. If a portable camera carried by the doctor is used for shooting, the image quality and the remaining power should be considered. Therefore, when choosing a portable camera, it is necessary to give priority to using a camera that can capture images sufficient for subsequent image analysis, and consider setting a remaining power alarm function. Correspondingly, a fixed camera installed at a fixed position in the ward does not need to consider the issue of power endurance, as it is equipped with a power transmission line, and only the issue of clarity needs to be considered.

[0022] Generally speaking, a portable camera can be carried in the doctor's coat pocket, because when the doctor records medical documents, he or she places the medical documents in front of him or her for recording. In this way, the portable camera can easily complete the image acquisition. Correspondingly, if the camera is a fixed camera installed at a fixed position in the ward, the shooting angle should be considered (the doctor or other objects may block the camera from taking photos of medical documents). Therefore, under normal circumstances, if a fixed camera is used, it should be considered to set up multiple cameras in the room (usually one camera can be set up at each of the four corners of the top of the room). At the same time, the camera should also consider the resolution issue. This is mainly because the camera at the corner of the roof is far away from the medical document. If the shooting clarity is not high enough, it will affect the subsequent OCR recognition.

[0023] During the shooting process, the system can use computer vision technology to perform real-time image quality analysis, combined with a deep learning image quality assessment model, to provide real-time feedback of image quality information to the user, guiding the user to shoot high-quality images. The captured images can first be pre-processed, including denoising, binarization, dilation, and corrosion operations to improve the image's text recognizability. After OCR recognition is completed, the recognition results can also be post-processed, mainly including semantic correction using a deep learning-based language model to improve the accuracy of the recognition results.

[0024] Generally speaking, the shooting of medical document photos can be scheduled or triggered. Specifically, the doctor can manually trigger the camera controller to control the camera to shoot (the controller can be set on the camera or can be shot through a remote controller), or the camera can shoot once every certain period of time (usually a few seconds).

[0025] Specifically, if Figure 2 As shown, step S101 can be implemented as follows: S1011: taking photos at predetermined intervals to obtain candidate photos; S1012: performing foreground extraction on the candidate photo to extract a current foreground image in the candidate photo; the current foreground image includes the handheld medical document; S1013: Calculating the foreground image similarity between the current foreground image and the previously acquired historical foreground image; S1014: If the foreground image similarity is less than a preset similarity, the current foreground image is used as a medical document photo.

[0026] In step S1011, the camera is used to take pictures at a certain interval to obtain candidate photos. In this step, the camera is a portable camera installed on the doctor. The candidate photos should include photos of medical documents held by the doctor.

[0027] Furthermore, in step S1012, it is necessary to perform foreground extraction on the candidate photo. Since it is certain that the object of foreground extraction is a medical document, in order to improve the efficiency and accuracy of foreground extraction, the following two methods can be selected: Method 1: Add an identification mark to the medical document. The identification mark can be a graphic, image or other mark that is not common in hospital scenarios. For example, the mark can be a black star located in the upper right corner of the medical document. Then, image recognition can be used to determine whether the mark exists in the candidate photo. If it exists, the predetermined area to the lower left of the mark is used as the foreground image. In specific implementation, the mark can be designed as a directional mark. In this way, it can be ensured that when the mark faces a specific direction (such as upward), the area to its lower left or a relative position is the foreground image.

[0028] That is, step S1012 can be implemented as follows: Detect whether there is an identification mark in the current candidate photo; If it exists, the location information and shape information of the identification mark are used to extract a sample image from the current candidate photo.

[0029] Specifically, the position information and shape information can reflect the size and direction of the identification mark, and the size of the foreground image can be reflected according to the size of the identification mark (or the size ratio of the foreground image and the identification mark is fixed, so the size of the identification mark can reflect the size of the foreground image), which can assist in locating the foreground image. The direction reflects the relative position of the identification mark and the foreground image.

[0030] Method 2: Deep learning algorithms can be used for foreground extraction. The deep learning algorithms are mainly composed of convolutional neural network (CNN) segmentation models, automatic feature extraction mechanisms, and network models combined with encoder-decoder structures. The reason why this method can be used for foreground extraction is mainly because there is a relatively standardized format architecture in medical documents. Specifically, the content in medical documents is not simply blank paper, but is composed of multiple standard format diagrams and texts designed according to the rules of doctor's inquiry. Specifically, prompt texts and diagrams / tables are set at different positions in medical documents. These prompt texts and diagrams / tables are used to standardize the doctor's consultation records. Therefore, under certain standardization conditions, the above-mentioned deep learning method can be used for foreground extraction (when performing foreground extraction, these standardized contents can play a positioning role). At the same time, since it is difficult to encounter other content similar to the standardized content in hospital scenarios, it will not cause misidentification of deep learning algorithms.

[0031] In specific implementation, the above two methods can be reused, that is, method 1 is used to perform positioning and sampling frame extraction first, and then method 2 is used for specific extraction. This avoids the problem of too many sampling frames in method 2, which consumes too much computing power.

[0032] In specific implementation, step S1012 can be implemented as follows: Step 10121, detecting whether there is an identification mark in the current candidate photo; Step 10122, if it exists, determine the sample image of the current candidate photo using the location information and shape information of the identification mark; Step 10123: input the sampled image into a pre-trained foreground extraction model to obtain a current foreground image.

[0033] Since the sampling image is located based on the identification mark, the main function of the sampling image is to narrow the search range when the foreground extraction model is processed. Since there are many images generated by shooting, it is preferred to use steps 10121-10123 to perform preliminary screening of the foreground image. In some cases, the sampling frame of the foreground extraction model in step 10123 can be similar in size to the sampling image extracted in step 10122, which can further reduce the sampling range. Specifically, the size and step size of the sampling frame can be determined based on the current remaining computing power. The remaining computing power is negatively correlated with the sampling frame size, and the remaining computing power is negatively correlated with the step size.

[0034] In step S1013, the similarity between the current foreground image and the historical foreground image is mainly calculated. Generally speaking, there are two types of historical foreground images. One is a completely blank medical document (not handwritten by the doctor), which only contains standardized prompt text and box diagrams / tables. The other is a medical document taken some time ago, which is generally a foreground image extracted from a medical document photo taken 5-10 seconds ago.

[0035] When the historical foreground image is a completely blank medical document, by calculating the similarity, it is possible to know whether the doctor has handwritten text in the medical document (if handwritten text appears in the medical document, the similarity will decrease, and the more text, the lower the similarity). However, this method has certain defects, mainly as the doctor's handwritten content increases, the similarity will become lower and lower, that is, during the doctor's handwriting process, the situation of continuously triggering step S1014 may occur, resulting in a decrease in accuracy. In order to avoid this problem, a method of calculating the similarity of multiple consecutive photos can be used, that is, a candidate photo is obtained at intervals of a period of time (N seconds, such as 3 seconds), and then the candidate photo is used to calculate the similarity. Since the doctor has been handwriting content in the medical document, the similarity of the candidate photos obtained thereafter will continue to decrease. When the similarity of the candidate photos obtained within a period of time (M seconds, such as 10 seconds, or 3-5 times of N seconds) is no longer decreasing, or the foreground image cannot be extracted, then in step S1014, the current foreground image with the smallest similarity is used as the medical document photo.

[0036] When the historical foreground image is a medical document taken some time ago, the similarity is calculated mainly by looking at the similarity of the foreground images of the candidate photos taken two adjacent times, which reflects whether the doctor has performed handwriting during this period of time. If the doctor has a lot of handwritten content during this period of time, the execution of step S1014 will be triggered. In order to avoid misidentification, if the foreground image cannot be identified in the current candidate photo, the foreground image in the previously acquired candidate photo should be used as the medical document photo.

[0037] Although there may be people and other things in the background of medical document photos, and there may be standardized tables and text in the foreground image, when extracting the foreground image, these non-doctor abbreviation text parts can still be extracted to facilitate subsequent OCR recognition.

[0038] Specifically, step S102 can be implemented as follows: S1021: Determine a candidate room where the medical document photo was taken according to the shooting location information of the medical document photo; S1022: Calculate the score of the reference image of each candidate room according to the room identification recorded in the background image of the medical document photo by using a visual positioning method; S1023: Determine the candidate room whose score exceeds the preset value as the target room; S1024: According to the shooting time information of the medical document photo, select a target audio segment from the video data of the target room as the audio data when calling the medical document generation, and select a target video segment as the video data when the medical document is generated.

[0039] In step S1021, GPS positioning or Beidou positioning can be used to determine the candidate room where the medical document photo is taken. These two positioning methods cannot determine the floor where the doctor is located. Therefore, rooms with the same longitude and latitude coordinates (such as the leftmost room on each floor) may be identified as candidate rooms. Of course, in addition to the above two positioning methods, other positioning technologies can also be used to achieve this. Specifically, such as wifi positioning, but it should be noted that wifi positioning also has deviations. Even if wifi positioning and GPS / Beidou positioning are combined, there may be errors. Although there are other positioning technologies in the current technical environment, such as strapdown inertial navigation, infrared and other technologies, the practical value of these technologies in medical scenarios is not high. They can only be realized in theory, but the actual cost is high and difficult to use in practice. Therefore, in this solution, only GPS or Beidou positioning technology is usually used. Then all rooms corresponding to the coordinates can be considered as candidate rooms.

[0040] In step S1022, visual positioning can be used, because in this solution, medical document photos must be obtained. When obtaining medical document photos, it is generally impossible for the entire image to be a medical document. There will also be some background images, and then auxiliary positioning can be performed by setting different room logos on different floors, and then when the room logo is photographed, the floor of the medical document photo can be determined (the room logo will be photographed in the background image). Of course, a more complex method can also be used for processing, that is, using the visual positioning technology used on the automatic robot, performing direct image coordinate conversion and indirect image coordinate conversion through the photographed photos, and mapping the converted coordinates to a predetermined virtual three-dimensional space (the three-dimensional space is established according to the characteristics of the hospital building), and accurate positioning can be performed, but this positioning method is not practical, mainly because the computing power consumption is too large. For portable devices, too much computing power will cause the photographed photos to be unable to be analyzed in time. Of course, it does not deny that this method can achieve technical purposes.

[0041] After the room identification is identified by visual positioning, the reference score of the reference image of each candidate room can be determined (the reference score mainly reflects the probability that the user is in each candidate room when taking the medical document photo). Here, the reference image of the candidate room (including the image with the room identification) is obtained in advance. It can be that each room has a reference image (the room identification of each room is different), or the rooms on each floor share a reference image (the room identifications of the rooms on the same floor are the same, and the room identifications of the rooms on different floors are different). After determining the score, in step S1023, the candidate room with the highest score can be used as the target room.

[0042] By setting up a camera in each room in advance (generally at the corner of the room ceiling), in step S1024, the audio data segment whose shooting time matches the shooting time information of the medical document photo in the content obtained by the camera can be used as the recording data when calling the medical document generation, and the video segment whose shooting time matches the shooting time information of the medical document photo can be used as the video data when the medical document is generated. The audio data and video data when the medical document is generated here respectively reflect the language and body movements of the doctor when recording the medical document (which may not be the language and body movements of the doctor writing in the medical document).

[0043] Specifically, step S104 can be implemented as follows: S1041: Perform voice recognition on the recorded data to obtain voice recognition text; S1042: Analyze the body movements of the person on the video data to obtain body movement feature data; S1043: extracting reference second document information corresponding to the speech recognition text from a pre-established historical speech recognition text-reference second document information comparison table based on the speech recognition text; S1044: extracting reference third document information corresponding to the body movement feature data from a pre-established historical body movement feature data-reference third document information comparison table based on the body movement feature data; S1045: Correct the first document information based on the reference second document information and the reference third document information.

[0044] In step S1041, the conventional speech-to-text recognition is mainly performed (recognition of Mandarin is sufficient). In some cases, if the region where the method is implemented has been determined, it is possible to consider adding dialect speech recognition technology. Specifically, the text conversion process mainly relies on the acoustic model and the language model. Among them, the acoustic model is responsible for converting the speech signal into a representation at the phoneme or word level, while the language model converts these representations into the final text output. In the training stage, the system needs to establish an acoustic model and a language model by analyzing a large amount of speech data; in the recognition stage, the system automatically recognizes the real-time speech and generates a text result.

[0045] In step S1042, the main thing to do is to identify limb movements, which can be achieved specifically by using human key point detection, deep learning models, etc. Considering that the medical scene is indoors and well-lit, there is rarely any interference, so general technology can be used. Since the results of limb movement recognition (limb movement feature data) are directly used to search in the table in the subsequent steps, it is not necessary to give a clear meaning to each result of limb movement recognition, as long as it can be associated. For the purpose of clarity, only two examples are listed here for illustration. The meaning of the result of limb movement recognition can be nodding with hands behind the back, which may express that the patient is in a stage that meets the treatment expectations; for example, the meaning of the result of limb movement recognition can be a specific limb stretching action (such as horizontally stretching the arms in front of the body, lifting the big back, etc.), which means that the doctor is guiding the patient to perform some kind of action to verify rehabilitation.

[0046] The contents of steps S1043 and S1044 are similar, both of which are queries in the established tables. For example, the contents of historical voice recognition text-reference second document information comparison table and historical limb movement feature data-reference third document information comparison table are based on information that has been historically examined by doctors and is considered correct, and therefore has good reference value.

[0047] It can be considered that in the historical voice recognition text-reference second document information comparison table, the corresponding historical voice recognition text and reference second document information are respectively the text recognition result obtained by voice recognition in step S1041 and the text after the text recognition result has been confirmed / modified. Correspondingly, in the historical body movement feature data-reference third document information comparison table, the corresponding historical body movement feature data and reference third document information are respectively the body movement feature data recognized in the past in step S1042 and the text content obtained after the doctor recognized the body movement feature data / watched the video data.

[0048] It should be noted that the reference to the second document information and the reference to the third document information usually does not directly replace the first document information, but the reference to the second document information and the reference to the third document information can reflect which first document information is closest to the real information. Specifically, since the first document information extracted from the medical document photo using OCR text recognition technology is not unique (the model outputs each text arrangement and the corresponding probability), in step S103, the model can be selected to output 3-5 results with the highest probability (candidate first document information), and then the correlation between the reference to the second document information and the reference to the third document information and these 3-5 results is used to select the most similar result from these 3-5 results in logic (usually language logic) to the reference to the second document information and the reference to the third document information.

[0049] For example, if the result of limb movement recognition means a specific limb stretching movement, then the probability of the result that describes rehabilitation suggestions as the final result among multiple results can be increased. In some cases, due to different document attributes, the content recorded by the doctor in the document may not include rehabilitation movements (such as the content recorded in the document is for the doctor to see himself, and there is no need to record rehabilitation movements). Therefore, semantic recognition should be used to calculate the semantic relevance of different candidate first document information with the second document information and the third document information, and then determine which candidate first document information is the most accurate based on the semantic relevance. For example, if a candidate first document information records content related to the requirements for taking medicine in the rehabilitation stage, although the third document information only contains limb movements and no medication movements, it should be considered that there is a strong correlation between the candidate first document information and the third document information, because the objects described by both are content related to rehabilitation, and the probability of the candidate first document information being the final result should be increased.

[0050] Similarly, the processing method of the second document information is the same as that of the third document information, and the text content does not require to be completely related, but the two can be highly related, such as the objects described are related. It is precisely because there are not many types of communication content in hospitals that this method can only be used in hospital environments.

[0051] Specifically, by using the second document information and the third document information, in addition to selecting a candidate first document information as the final result, the second document information and the third document information can also be used to modify the final output result. However, since this modification is not a substantial change and is only a formal fine-tuning, it will not be explained in depth.

[0052] That is, step S1045 can be implemented as follows: Step 10451, calculating the first semantic relevance between each first document information (candidate first document information) and the reference second document information; Step 10452, calculating the second semantic relevance between each first document information (candidate first document information) and the reference third document information; Step 10453, selecting target first document information from a plurality of first document information according to the first semantic relevance and the second semantic relevance; Step 10454, correct the target first document information using the reference second document information and the reference third document information.

[0053] In step S105, the first document information entered into the database is the target first document information.

[0054] Of course, when performing OCR recognition, you can consider using an open source OCR engine, such as Tesseract, or a deep learning-based OCR model, such as CRNN or Attention OCR, to perform text recognition on the preprocessed images.

[0055] Specifically, the semantic relevance between each result and the reference second document information and the semantic relevance between each result and the reference third document information may be calculated respectively, and then the semantic similarity parameter of each result may be calculated by weighted average, and then the result with the largest semantic similarity parameter may be used as the correction result of step S104 (the first document information in step S104). Finally, the correction result may be input into the database.

[0056] However, in actual processing, there may be a situation where the semantic similarity parameters of the candidate first document information with the highest semantic similarity parameter and the candidate first document information with the second and third highest semantic similarity parameters are not much different. In this case, the following processing method can be considered, that is, step S105 can be implemented as follows: S1051: If there are multiple corrected first document information, retrieve the medical information of the patient corresponding to the medical document photo; S1052: Displaying the medical information and the plurality of corrected first document information in the doctor terminal, so that one of the plurality of corrected target first document information is selected as the final first document information according to the selection instruction received by the doctor terminal; S1053: The final first document information is stored in the database, and the historical speech recognition text-reference second document information comparison table and the historical body movement feature data-reference third document information comparison table are updated based on the corrected final first document information.

[0057] That is, when the first document information candidates with the highest semantic similarity parameters cannot be distinguished by referring to the second document information and the third document information, the patient's historical medical information can be used for reference. The medical information can include the patient's personal information (height, age, weight, etc.), historical medical information (what diseases he had in history, the treatment, etc.), and the current diagnosis and treatment information.

[0058] After obtaining the medical information, this information and the corrected first document information can be displayed on the doctor's terminal, so that the doctor can manually select / manually modify a certain first document information as the final first document information. After that, the first document information selected / modified by the doctor can be stored in the database. At the same time, the table in the database can also be updated to make the recognition more accurate next time. Of course, for practical considerations, the doctor's terminal should have certain permissions to view the tables in the database and correct the erroneous information in a timely manner.

[0059] The text finally stored in the database is preferably converted into a structured data format, such as JSON, to facilitate subsequent processing and application.

[0060] The method provided in this application has the following advantages as a whole: 1. High efficiency: The technology provided by this application uses OCR technology and big data model technology to enable users to complete form filling by uploading pictures (such as checklists, inspection sheets, etc.), which greatly saves users' time in filling out forms. Compared with the traditional manual filling method, our method can reduce the time of filling out forms by more than 70%, greatly improving the efficiency of data entry.

[0061] 2. Improve accuracy: Since the present invention uses speech recognition results and body recognition results for auxiliary recognition, and also uses a series of technical means such as image quality improvement, image pre-processing and post-processing, OCR recognition and data structuring, it can effectively improve the accuracy of recognition results. According to our experimental data, compared with traditional OCR technology, the present invention significantly improves the accuracy of data entry.

[0062] 3. User-friendly: The technology provided by this application also provides data storage and display functions, which displays the OCR results to users in the form of charts or forms, allowing users to intuitively see the OCR results, greatly improving the user experience. At the same time, the present invention can also directly store the OCR results in the database, which is convenient for users to query and use the data later.

[0063] 4. Environmental protection and reduced labor intensity: The traditional data entry method requires a lot of paper and manpower, which not only affects the environment but also has high labor intensity. The method of the present invention is completely electronic, which is not only environmentally friendly but also greatly reduces labor intensity.

[0064] Based on the same technical concept, the present application also provides a medical information input device, such as Figure 3 As shown, the device comprises: An acquisition module 301 is used to acquire a medical document photo obtained by photographing the text in the medical document through a camera located in the ward; A calling module 302 is used to call the audio data and video data when the medical document is generated according to the shooting time information and shooting location information of the medical document photo; An extraction module 303 is used to extract first document information from the medical document photo using OCR text recognition technology; A correction module 304, configured to correct the first document information based on the audio recording data and the video recording data; The input module 305 is used to input the corrected first document information into the database.

[0065] Optionally, the camera is a portable camera installed on the doctor; when the acquisition module 301 is used to acquire a medical document photo obtained by photographing the text in the medical document through a camera located in the ward, it is specifically used to: Taking photos at predetermined intervals to obtain candidate photos; Performing foreground extraction on the candidate photo to extract a current foreground image in the candidate photo; the current foreground image includes a handheld medical document; Calculating the foreground image similarity between the current foreground image and the previously acquired historical foreground image; If the foreground image similarity is less than the preset similarity, the current foreground image is used as the medical document photo.

[0066] Optionally, when the calling module 302 is used to call the audio data and video data when the medical document is generated according to the shooting time information and shooting location information of the medical document photo, it is specifically used to: Determine, based on the shooting location information of the medical document photo, a candidate room where the medical document photo was shot; Using visual positioning, the score of the reference image of each candidate room is calculated based on the medical document photos; Determine the candidate rooms whose scores exceed a preset value as target rooms; According to the shooting time information of the medical document photo, a target audio segment is selected from the video data of the target room as the audio data when calling the medical document generation, and a target video segment is selected as the video data when the medical document is generated.

[0067] Optionally, when the correction module 304 is used to correct the first document information based on the audio data and the video data, it is specifically used to: Performing speech recognition on the recorded data to obtain speech recognition text; Performing body movement analysis on the video data to obtain body movement feature data; Extracting reference second document information corresponding to the speech recognition text from a pre-established historical speech recognition text-reference second document information comparison table based on the speech recognition text; Extracting reference third document information corresponding to the body movement feature data from a pre-established historical body movement feature data-reference third document information comparison table based on the body movement feature data; The first document information is corrected based on the reference second document information and the reference third document information.

[0068] Optionally, when the input module 305 is used to input the corrected first document information into the database, it is specifically used to: If there are multiple corrected first document information, retrieve the medical information of the patient corresponding to the medical document photo; Displaying the medical information and the plurality of corrected first document information in a doctor terminal, so that one of the plurality of corrected first document information is selected as the final first document information according to a selection instruction received by the doctor terminal; The final first document information is stored in a database, and the historical speech recognition text-reference second document information comparison table and the historical body movement feature data-reference third document information comparison table are updated based on the corrected final first document information.

[0069] Figure 4 A structural diagram of an electronic device provided for an embodiment of the present application includes: a processor 401, a memory 402 and a bus 403, wherein the memory 402 stores machine-readable instructions executable by the processor 401. When the electronic device runs the above-mentioned information processing method, the processor 401 communicates with the memory 402 through the bus 403, and the processor 401 executes the machine-readable instructions to perform the method steps described in Example 1.

[0070] The embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method steps described in the first embodiment are executed.

[0071] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, electronic devices, and computer-readable storage media can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0072] In the several embodiments provided in the present application, it should be understood that the disclosed methods, devices, electronic devices and computer-readable storage media can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0073] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0074] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0075] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0076] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for entering medical information, characterized in that: include: Obtaining a photo of the medical document obtained by photographing the text in the medical document through a camera located in the ward; According to the shooting time information and shooting location information of the medical document photo, calling the audio data and video data when the medical document was generated; Extracting first document information from the medical document photo using OCR text recognition technology; Correcting the first document information based on the audio data and the video data; The corrected first document information is entered into the database.

2. The method according to claim 1, characterized in that: The camera is a portable camera installed on the doctor; the obtaining of the medical document photo obtained by photographing the text in the medical document with the camera located in the ward includes: Taking photos at predetermined intervals to obtain candidate photos; Performing foreground extraction on the candidate photo to extract a current foreground image in the candidate photo; the current foreground image includes a handheld medical document; Calculating the foreground image similarity between the current foreground image and the previously acquired historical foreground image; If the foreground image similarity is less than the preset similarity, the current foreground image is used as the medical document photo.

3. The method according to claim 1, characterized in that: The calling of the audio and video data when the medical document is generated according to the shooting time information and shooting location information of the medical document photo includes: Determine, based on the shooting location information of the medical document photo, a candidate room where the medical document photo was shot; Using visual positioning, the score of the reference image of each candidate room is calculated based on the medical document photos; Determine the candidate rooms whose scores exceed a preset value as target rooms; According to the shooting time information of the medical document photo, a target audio segment is selected from the video data of the target room as the audio data when calling the medical document generation, and a target video segment is selected as the video data when the medical document is generated.

4. The method according to claim 3, characterized in that: The correcting the first document information based on the audio data and the video data includes: Performing speech recognition on the recorded data to obtain speech recognition text; Performing body movement analysis on the video data to obtain body movement feature data; Extracting reference second document information corresponding to the speech recognition text from a pre-established historical speech recognition text-reference second document information comparison table based on the speech recognition text; Extracting reference third document information corresponding to the body movement feature data from a pre-established historical body movement feature data-reference third document information comparison table based on the body movement feature data; The first document information is corrected based on the reference second document information and the reference third document information.

5. The method according to claim 4, characterized in that: The step of entering the corrected first document information into a database includes: If there are multiple corrected first document information, retrieve the medical information of the patient corresponding to the medical document photo; Displaying the medical information and the plurality of corrected first document information in a doctor terminal, so that one of the plurality of corrected first document information is selected as the final first document information according to a selection instruction received by the doctor terminal; The final first document information is stored in a database, and the historical speech recognition text-reference second document information comparison table and the historical body movement feature data-reference third document information comparison table are updated based on the corrected final first document information.

6. The method according to claim 2, characterized in that: The step of performing foreground extraction on the candidate photo to extract the current foreground image in the candidate photo includes: Detect whether there is an identification mark in the current candidate photo; If it exists, extracting a sample image from the current candidate photo using the location information and shape information of the identification mark; The sampled image is input into a pre-trained foreground extraction model to obtain a current foreground image.

7. The method according to claim 4, characterized in that: The step of correcting the first document information based on the reference second document information and the reference third document information comprises: Calculating a first semantic relevance between each first document information and a reference second document information; Calculating a second semantic relevance between each first document information and reference third document information; Selecting target first document information from a plurality of first document information according to a first semantic relevance and a second semantic relevance; The target first document information is corrected using the reference second document information and the reference third document information.

8. A medical information input device, characterized in that: include: An acquisition module, used to acquire a photo of a medical document obtained by photographing the text in the medical document through a camera located in the ward; A calling module, used to call the audio data and video data when the medical document is generated according to the shooting time information and shooting location information of the medical document photo; An extraction module, used to extract first document information from the medical document photo using OCR text recognition technology; A correction module, used for correcting the first document information based on the audio data and the video data; The input module is used to input the corrected first document information into the database.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the method as described in any one of claims 1 to 7 are performed.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are executed.

Images