Electronic medical record generation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses an electronic medical record generation method and device, electronic equipment and a storage medium. The method comprises the following steps: in response to an electronic medical record generation instruction for a skin inquiry service, obtaining interaction content between a provider and an acquirer of the skin inquiry service and a deep learning model for realizing electronic medical record generation; acquiring skin data of the skin lesion part inquired by the acquirer; wherein the skin data comprises at least one of a skin image and a skin medical report; and generating an electronic medical record corresponding to the skin inquiry service according to the deep learning model, the interaction content and the skin data. According to the technical scheme provided by the embodiment of the invention, on the basis of the interactive content, the skin data capable of providing main auxiliary diagnosis information for a doctor is combined, so that the deep learning model can be utilized to process the multi-modal information (namely, the interactive content and the skin data) to generate the electronic medical record; therefore, the generation quality of the electronic medical record for the skin inquiry service is improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of Internet medical technology, and in particular, to a method, apparatus, electronic device, and storage medium for generating electronic medical records. Background Art

[0002] In recent years, with the rapid development of Internet technology and the increasing health demands of the people, the scale of Internet hospitals has been continuously growing, and the online consultations they support have become one of the main channels for medical consultations.

[0003] Through data statistics, it is found that skin consultations account for the highest proportion in the online consultations of each department, which makes the skin Internet hospital have a huge application demand. On this basis, it should be emphasized that electronic medical records are an important measure to ensure the service quality of the skin Internet hospital and help improve the quality of medical services.

[0004] In the process of implementing the present invention, the inventors found the following technical problems in the prior art: It is impossible to generate high-quality electronic medical records for the skin consultation services provided by the skin Internet hospital. Summary of the Invention

[0005] Embodiments of the present invention provide a method, apparatus, electronic device, and storage medium for generating electronic medical records, which can generate high-quality electronic medical records for skin consultation services.

[0006] According to one aspect of the present invention, a method for generating an electronic medical record is provided, which may include:

[0007] In response to an electronic medical record generation instruction for a skin consultation service, obtain the interaction content between the provider and the acquirer of the skin consultation service, and a deep learning model for implementing the generation of the electronic medical record;

[0008] Obtain the skin data of the skin lesion site consulted by the acquirer; wherein the skin data includes at least one of a skin image and a skin medical report;

[0009] Generate an electronic medical record corresponding to the skin consultation service according to the deep learning model, the interaction content, and the skin data.

[0010] According to another aspect of the present invention, an apparatus for generating an electronic medical record is provided, which may include:

[0011] A deep learning model acquisition module, configured to obtain the interaction content between the provider and the acquirer of the skin consultation service, and a deep learning model for implementing the generation of the electronic medical record in response to an electronic medical record generation instruction for the skin consultation service;

[0012] A skin data acquisition module, configured to acquire skin data of the skin lesion site inquired by the acquirer; wherein the skin data includes at least one of a skin image and a skin medical report;

[0013] An electronic medical record generation module, configured to generate an electronic medical record corresponding to the skin consultation service according to a deep learning model, interaction content, and skin data.

[0014] According to another aspect of the present invention, there is provided an electronic device, which may include:

[0015] At least one processor; and

[0016] A memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program executable by the at least one processor, and when the computer program is executed by the at least one processor, the at least one processor is caused to implement the electronic medical record generation method provided in any embodiment of the present invention.

[0018] According to another aspect of the present invention, there is provided a computer-readable storage medium, on which computer instructions are stored, and when the computer instructions are used to cause a processor to execute, the electronic medical record generation method provided in any embodiment of the present invention is implemented.

[0019] The technical solution of the embodiment of the present invention, by responding to an electronic medical record generation instruction for a skin consultation service, acquires the interaction content between the provider and the acquirer of the skin consultation service, and a deep learning model for implementing electronic medical record generation; acquires skin data of the skin lesion site inquired by the acquirer; wherein the skin data includes at least one of a skin image and a skin medical report; and generates an electronic medical record corresponding to the skin consultation service according to the deep learning model, interaction content, and skin data. The above technical solution, by combining skin data that can provide main auxiliary diagnostic information for doctors on the basis of interaction content, can thus use a deep learning model to process multi-modal information (i.e., interaction content + skin data) to generate an electronic medical record, thereby improving the quality of electronic medical record generation for skin consultation services.

[0020] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. Description of the Drawings

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for description in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0022] Figure 1 is a flowchart of an electronic medical record generation method provided according to an embodiment of the present invention;

[0023] Figure 2 is a flowchart of another electronic medical record generation method provided according to an embodiment of the present invention;

[0024] Figure 3 is a schematic diagram of the recognition process of the original image recognition model in another electronic medical record generation method provided according to an embodiment of the present invention;

[0025] Figure 4 is a flowchart of a skin medical report recognition example in another electronic medical record generation method provided according to an embodiment of the present invention;

[0026] Figure 5 is a flowchart of yet another electronic medical record generation method provided according to an embodiment of the present invention;

[0027] Figure 6 is a flowchart of a skin Internet hospital electronic medical record generation example of the multimodal large language model in yet another electronic medical record generation method provided according to an embodiment of the present invention;

[0028] Figure 7 is a flowchart of still another electronic medical record generation method provided according to an embodiment of the present invention;

[0029] Figure 8 is a schematic diagram of the interface of an electronic medical record interaction example in still another electronic medical record generation method provided according to an embodiment of the present invention;

[0030] Figure 9 is a structural block diagram of an electronic medical record generation device provided according to an embodiment of the present invention;

[0031] Figure 10 is a schematic diagram of the structure of an electronic device for implementing the electronic medical record generation method of the embodiments of the present invention. Detailed implementation manners

[0032] To enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0033] It should be noted that in the description and claims of the present invention and the above-mentioned drawings, the terms "first", "second", etc. are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. The same is true for "target", "original", etc., which will not be elaborated here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0034] It should be noted that in the technical solution of the present invention, in terms of the collection, gathering, updating, analysis, processing, use, transmission, storage, etc. of the user's personal information, it complies with the provisions of relevant laws and regulations, is used for legal purposes, and does not violate public order and good customs. Necessary measures are taken for the user's personal information to prevent illegal access to the user's personal information data and to maintain the security of the user's personal information and network security.

[0035] Before introducing the embodiments of the present invention, an exemplary description of the implementation process of the currently adopted electronic medical record generation solution is first given, which helps to understand the reason for the problem that high-quality electronic medical records cannot be generated for skin consultation services described in the background technology, and further helps to understand the reason why the embodiments of the present invention can generate high-quality electronic medical records for skin consultation services.

[0036] Exemplarily, currently, based on technologies such as keyword matching and regular expression matching, medical record content can be captured from the doctor-patient conversations generated during online consultations, and electronic medical records can be generated based on this medical record content. However, through practice, it is found that this implementation solution strongly depends on rule collection and essentially lacks the ability to understand context semantics. This makes it easy to mis-capture and / or miss-capture medical record content when the keywords and / or regular expressions are not fully enumerated, which will directly affect the quality of the generated electronic medical records.

[0037] In addition, electronic medical records can also be generated based on deep learning models with the ability of context semantic understanding, thus solving the problem that keyword matching and regular matching technologies cannot generate high-quality electronic medical records due to the lack of context semantic understanding ability. However, through further practice, it is found that compared with other Internet hospitals, during the online consultation process of the skin Internet hospital, doctors often obtain more than 95% of the auxiliary diagnosis information (i.e., information that can be used for auxiliary diagnosis) through skin data such as skin images and / or skin medical reports uploaded by patients. And these auxiliary diagnosis information cannot be achieved at all by the text summarization ability of the deep learning model alone. This makes the electronic medical record generation scheme based on a single deep learning model unable to generate high-quality electronic medical records for the skin consultation services provided by the skin Internet hospital.

[0038] In response to this, the embodiments of the present invention propose an implementation scheme that combines doctor-patient conversations and skin data, whereby a deep learning model can be used to process multimodal information to generate electronic medical records, thus ensuring the quality of electronic medical record generation. The following will elaborate on this in detail.

[0039] Figure 1 It is a flowchart of a method for generating electronic medical records provided by an embodiment of the present invention. This embodiment is applicable to the situation of generating electronic medical records, especially applicable to the situation of generating electronic medical records for the skin consultation services provided by the skin Internet hospital. This method can be executed by the electronic medical record generation device provided by the embodiments of the present invention. The device can be implemented in software and / or hardware, and the device can be integrated on an electronic device, which can be various user terminals or servers.

[0040] See Figure 1 , the method of the embodiment of the present invention specifically includes the following steps:

[0041] S110. In response to an electronic medical record generation instruction for a skin consultation service, obtain the interaction content between the provider and the acquirer of the skin consultation service, and a deep learning model for implementing electronic medical record generation.

[0042] Among them, the skin consultation service can be understood as the service provided by the skin Internet hospital for online consultation of skin diseases. The provider can be understood as the party providing the skin consultation service. Combining the application scenarios that the embodiments of the present invention may involve, for example, it can be a doctor in the skin Internet hospital. The acquirer can be understood as the party obtaining the skin consultation service, that is, the party that conducts online interaction with the provider during the skin consultation process. Combining the application scenarios that the embodiments of the present invention may involve, the acquirer can be, for example, a patient with skin diseases and their associated persons. The associated persons can be understood as persons associated with the patient, such as at least one of the patient's family members, friends, classmates, and colleagues, and no specific limitation is made here.

[0043] The interaction content can be understood as the content generated due to the interaction between the provider and the acquirer during the skin consultation process. Considering the application scenarios that the embodiments of the present invention may involve, optionally, the interaction content can be presented in at least one of the ways such as text, pictures, voice, and video; further optionally, figuratively speaking, the interaction content can also be referred to as a doctor-patient conversation. Combining the above examples, for example, it can be a text conversation, a picture conversation, a voice conversation, or a video conversation, etc., which is related to the actual situation and is not specifically limited herein.

[0044] The electronic medical record generation instruction can be understood as an instruction for generating an electronic medical record for the completed skin consultation service. Considering the application scenarios that the embodiments of the present invention may involve, the electronic medical record generation instruction can be triggered by the provider and / or the acquirer of the skin consultation service, especially by the provider.

[0045] The deep learning model can be understood as a model for realizing the generation of electronic medical records, especially a model for generating electronic medical records for skin consultation services.

[0046] In response to the electronic medical record generation instruction, the interaction content and the deep learning model are obtained.

[0047] S120. Obtain the skin data of the skin lesion site inquired by the acquirer; wherein, the skin data includes at least one of a skin image and a skin medical report.

[0048] Among them, the skin lesion site can be understood as the site where the skin is damaged inquired by the acquirer, and this site is the site of the patient corresponding to the acquirer suffering from skin diseases. For example, it can be the face, hands, or legs, etc., which is related to the actual situation and is not specifically limited herein.

[0049] The skin image can be understood as an image mainly taken for the skin lesion site to reflect the skin on the skin lesion site. The skin medical report can be understood as a medical report generated for the skin lesion site. Considering the application scenarios that the embodiments of the present invention may involve, for example, it can be at least one of a physical examination report, a test report, an inspection report, and a historical medical record, etc., which is related to the actual situation and is not specifically limited herein. Exemplarily, here taking the inspection report as an example, for example, it can be a fungal inspection report or a mite inspection report, etc.

[0050] Obtain the skin data of the skin lesion site, and the skin data can be at least one of a skin image and a skin medical report. According to the above description, compared with the interaction content, the skin data can provide more auxiliary diagnosis information for doctors, which is the key to generating high-quality electronic medical records.

[0051] S130. Generate an electronic medical record corresponding to the skin consultation service according to the deep learning model, interaction content, and skin data.

[0052] Among them, an electronic medical record corresponding to the skin consultation service is generated according to the deep learning model, interaction content, and skin data. Combining with the application scenarios that the embodiments of the present invention may involve, exemplarily, the interaction content and skin data can be directly input into the deep learning model, so that the deep learning model can be used to process the interaction content and skin data to generate an electronic medical record; alternatively, the interaction content can be first recognized to obtain the content text, and the skin data can be recognized to obtain the data text, and then the content text and the data text are input into the deep learning model, so that the deep learning model can be used to process the content text and the data text to generate an electronic medical record; of course, the electronic medical record can also be generated based on other methods, which are not specifically limited herein.

[0053] The technical solution of the embodiments of the present invention obtains the interaction content between the provider and the acquirer of the skin consultation service, and the deep learning model for realizing the generation of the electronic medical record in response to the electronic medical record generation instruction for the skin consultation service; obtains the skin data of the skin lesion site inquired by the acquirer; wherein, the skin data includes at least one of a skin image and a skin medical report; and generates an electronic medical record corresponding to the skin consultation service according to the deep learning model, interaction content, and skin data. The above technical solution combines the skin data that can provide the main auxiliary diagnosis information for doctors on the basis of the interaction content, so that the deep learning model can process multi-modal information (i.e., interaction content + skin data) to generate an electronic medical record, thereby improving the quality of generating the electronic medical record for the skin consultation service.

[0054] Figure 2 It is a flowchart of another method for generating an electronic medical record provided in the embodiments of the present invention. This embodiment is optimized based on the above technical solutions. In this embodiment, optionally, generating an electronic medical record for the skin consultation service according to the deep learning model, interaction content, and skin data includes: recognizing the interaction content to obtain the content text, and recognizing the skin data to obtain the data text; inputting the content text and the data text into the deep learning model to generate an electronic medical record corresponding to the skin consultation service. Among them, the explanations of the same or corresponding terms as those in the above embodiments are not repeated here.

[0055] See Figure 2 , the method of this embodiment may specifically include the following steps:

[0056] S210. In response to the electronic medical record generation instruction for the skin consultation service, obtain the interaction content between the provider and the acquirer of the skin consultation service, and the deep learning model for realizing the generation of the electronic medical record.

[0057] S220. Obtain skin data of the skin lesion site inquired by the acquirer; wherein, the skin data includes at least one of a skin image and a skin medical report.

[0058] S230. Identify the interaction content to obtain a content text, and identify the skin data to obtain a data text.

[0059] Among them, as in the above example, the interaction content can be presented in at least one of the ways of text, picture, voice, and video. It should be noted that compared with other ways, the deep learning model is more suitable for processing text and has powerful capabilities in text recognition and text summarization. Therefore, the interaction content can be identified to obtain a content text to utilize the deep learning model to process the content text.

[0060] Similarly, the skin data can be at least one of a skin image and a skin medical report, which indicates that the skin data is not necessarily in text form. Therefore, in order to better process the skin data, the skin data can be identified to obtain a data text, so as to utilize the deep learning model to process the data text.

[0061] S240. Input the content text and the data text into the deep learning model to generate an electronic medical record corresponding to the skin consultation service.

[0062] Among them, since the deep learning model can directly process text, the content text and the data text can be input into the deep learning model, so as to utilize the deep learning model to process both of them and generate an electronic medical record.

[0063] The technical solution of the embodiment of the present invention, by separately identifying the interaction content and the skin data to obtain a content text and a data text that can be better processed by the deep learning model, thereby utilizing the deep learning model to process these two texts, further improves the generation quality of the electronic medical record.

[0064] An optional technical solution, where the skin data includes a skin image, then identifying the skin data to obtain a data text includes:

[0065] Obtain a pre-trained skin image recognition model;

[0066] Input the skin image into the skin image recognition model to identify the skin image according to the skin image recognition model and obtain a data text describing the skin image.

[0067] Among them, the skin image recognition model can be understood as a pre-trained model with image recognition function. Input the skin image into the skin image recognition model to identify the skin image according to the skin image recognition model and obtain a data text describing the skin image.

[0068] In the above technical solution, by applying the skin image recognition model, accurate and rapid recognition of skin images is achieved.

[0069] On this basis, optionally, the skin image recognition model is pre-trained through the following steps:

[0070] Obtain the original image recognition model and training samples; among them, the original image recognition model includes a visual backbone network, a text feature extractor, and a text decoder, and the training samples include skin lesion images and descriptive texts describing the skin lesion images;

[0071] Input the training samples into the original image recognition model to process the skin lesion images according to the visual backbone network to obtain visual features, process the descriptive texts according to the text feature extractor to obtain text features, process the visual features according to the text decoder to obtain visual texts, and process the text features according to the text decoder to obtain text texts;

[0072] Based on the similarity between the visual text and the text text, train the original image recognition model, and based on the trained visual backbone network and text decoder, obtain the skin image recognition model.

[0073] Among them, the original image recognition model can be understood as a model for image recognition that has training requirements. In the embodiments of the present invention, the original image recognition model includes a visual backbone (Vision Transformer, ViT) network, a text feature extractor, and a text decoder. In practical applications, optionally, in order to accelerate the training speed, the ViT network and the text feature extractor can be pre-trained based on image-text pairs, that is, the pre-trained ViT network and text feature extractor are used as initial weights and applied to the subsequent model training process. Additionally, optionally, the text feature extractor can include a Roberta text feature extractor. Of course, it can also be other text feature extractors, which are not specifically limited herein.

[0074] The training samples can be understood as samples for training the original image recognition model. In the embodiments of the present invention, the training samples include skin lesion images and descriptive texts describing the skin lesion images. The skin lesion images can be understood as images obtained by photographing the skin lesion site, and the skin lesion site can be the same or different from the skin lesion site in the above text, which is related to the actual situation and is not specifically limited herein. In practical applications, optionally, in order to make the actual output format of the trained skin image recognition model conform to the expected output format, the descriptive text can be set according to the expected output format, which is similar to introducing the prompt strategy. Assuming that the expected output is the skin lesion site and skin lesion symptoms, the descriptive text can be set to something like "Possible site: [site], possible symptoms: [symptoms]" to further guide the model output.

[0075] Input the training samples into the original image recognition model, so that the original image recognition model can perform the following steps: process the skin lesion image using the ViT network (i.e., encode the skin lesion image) to obtain visual features, and process the description text using the text feature extractor (i.e., encode the description text) to obtain text features. Then, decode the visual features using the text decoder to obtain visual text, and decode the text features using the text decoder to obtain text text.

[0076] On this basis, further, since the description text is the text describing the skin lesion image, the visual text obtained by encoding and then decoding the skin lesion image should be highly similar to the text text obtained by encoding and then decoding the description text. Therefore, the parameters in the original image recognition model can be adjusted based on the similarity between the visual text and the text text to complete the training of the original image recognition model. Based on the trained ViT network and text decoder, a skin image recognition model can be obtained.

[0077] To more vividly understand the above technical solution, the following will be described by way of specific examples. Exemplarily, see Figure 3 , in terms of pre-trained weights, a ViT network and a Roberta text feature extractor trained on image-text pairs based on the CLIP architecture are used as the initial weights. On this basis, the skin lesion image passes through an Image Encoder to obtain a number of visual tokens, and each word in the description text can also pass through tokenize&embed to obtain a number of word tokens; subsequently, the visual tokens and word tokens are decoded separately in a unified Text Decoder for the subsequent parameter adjustment process. On this basis, a prompt strategy can also be introduced to further guide the model output through "possible part: [part], possible symptom: [symptom]".

[0078] Through the mutual cooperation of the ViT network, the text feature extractor and the text decoder, the above technical solution realizes the effective training of the skin image recognition model.

[0079] In another alternative technical solution, if the skin data includes a skin medical report, then identify the skin data to obtain data text, including:

[0080] In the case where the skin medical report is not in image format, convert the skin medical report into image format, and use the obtained skin medical image as the skin medical report;

[0081] Perform content analysis on the dermatological medical report to obtain the report content;

[0082] Perform structured extraction on the report content to obtain the data text.

[0083] Among them, considering the application scenarios that the embodiments of the present invention may involve, the format of the dermatological medical report may be an image format or a Portable Document Format (PDF). Compared with PDF, the image format is easier to recognize. Therefore, when the dermatological medical report is not in the image format, convert the dermatological medical report into the image format for recognition based on the subsequent dermatological medical report in the image format (i.e., the dermatological medical image). Update the dermatological medical image to the dermatological medical report. Further, perform content analysis on the dermatological medical report to obtain the report content in the dermatological medical report. Still further, perform structured extraction on the report content to obtain the data text.

[0084] To more vividly understand the above technical solution, the following provides an exemplary illustration with specific examples. Exemplarily, refer to Figure 4 , when the dermatological medical report is in PDF format, convert the dermatological medical report into an image format to obtain the dermatological medical image, and start recognition based on this. To improve the accuracy of dermatological medical image recognition, the dermatological medical image can be preprocessed. For example, image correction can be performed through perspective correction and horizontal correction; another example is to remove the noise in it; etc., which are not specifically limited here. Then, perform content analysis on the preprocessed dermatological medical image. Exemplarily, the DBNet model can be used for Optical Character Recognition (OCR) detection, the Convolutional Recurrent Neural Network (CRNN) model can be used for OCR recognition, and the DB+PP-YOLOv2 model can be used for table layout analysis. In addition, in terms of table recognition, the PP-LCNet model can be trained with real and expert-annotated real medical reports online to achieve accurate table recognition ability. After recognition, LayoutXLM can be used for Key Information Extraction (KIE) to obtain the report content. Then, perform structured extraction on the report content, such as extracting medical conclusion information (such as diagnosis conclusions, etc.) and table information (such as specific examination items, etc.) to obtain the data text. Thus, the recognition of the dermatological medical report is completed.

[0085] The above technical solution realizes the fast and accurate recognition of dermatological medical reports.

[0086] In another alternative technical solution, if the interaction content is a voice conversation, the interaction content is recognized to obtain content text, including:

[0087] Obtain a pre-trained speech recognition model;

[0088] Input the voice conversation into the speech recognition model to recognize the voice conversation according to the speech recognition model and obtain content text.

[0089] As can be seen from the above description, compared with speech, deep learning models are more suitable for processing text. Therefore, when the interaction content is a voice conversation, that is, the interaction content is presented in the form of voice, the voice conversation can be input into the speech recognition model to recognize the voice conversation according to the speech recognition model and obtain content text. The above technical solution converts the interaction content in the form of voice into a text form that can be better processed by the deep learning model, which helps to further improve the generation quality of electronic medical records.

[0090] Figure 5 It is a flowchart of another method for generating electronic medical records provided in an embodiment of the present invention. This embodiment is optimized based on the above technical solutions. In this embodiment, optionally, the deep learning model is a large language model. The explanations of the same or corresponding terms as those in the above embodiments are not repeated here.

[0091] See Figure 5 , the method of this embodiment may specifically include the following steps:

[0092] S310. In response to an electronic medical record generation instruction for a skin consultation service, obtain the interaction content between the provider and the acquirer of the skin consultation service, and a large language model for implementing electronic medical record generation.

[0093] It should be noted that, compared with deep learning models with a small number of parameters, deep learning models with a large number of parameters (i.e., large language models, or simply referred to as large models) can better ensure the generation quality of electronic medical records due to their powerful context semantic understanding ability and few-shot learning ability. Therefore, in the embodiments of the present invention, a large model is used to generate electronic medical records.

[0094] S320. Obtain the skin data of the skin lesion site questioned by the acquirer; where the skin data includes at least one of a skin image and a skin medical report.

[0095] S330. Recognize the interaction content to obtain content text, and recognize the skin data to obtain data text.

[0096] S340. Input the content text and the data text into the large language model to generate an electronic medical record corresponding to the skin consultation service.

[0097] The technical solution of the embodiment of the present invention further improves the generation quality of electronic medical records by using a large model with strong context semantic understanding ability and few-shot learning ability for text processing.

[0098] An optional technical solution is that the large language model is pre-learned based on prompts, and the prompts are pre-set according to the electronic medical record specifications.

[0099] Input the content text and data text into the deep learning model to generate an electronic medical record corresponding to the skin consultation service, including:

[0100] Input the content text and data text into the large language model to generate an electronic medical record corresponding to the skin consultation service based on the large language model, prompts, content text, and data text.

[0101] Among them, considering that compared with general text generation, the generation of electronic medical records has extremely high professional requirements. Therefore, in order to ensure that the electronic medical records generated by the large model have medical professionalism, the large model can be pre-learned with prompts with medical professionalism, so that the large model uses the prompts to generate electronic medical records with medical professionalism. It should be noted that in the large model, the main role of the prompt is to provide the context of the input information and the parameter information input into the model for the large model. The setting of the prompt has a great effect and influence on the content output by the large model. A suitable prompt can make the large model output more appropriate and accurate content, which is similar to giving an order. The more specific it is, the higher the output quality.

[0102] Specifically, the electronic medical record specifications can be understood as the specifications that electronic medical records with medical professionalism should comply with, especially the specifications that electronic medical records of skin Internet hospitals with medical professionalism should comply with. Therefore, the prompts can be pre-set according to the electronic medical record specifications to ensure the medical professionalism of the prompts. On this basis, optionally, in order to further refine the prompts, the content of each field in the electronic medical record specifications can be pre-combed, and the prompts can be pre-set according to these field contents, which helps to further improve the quality of the electronic medical records output by the large model. Further optionally, for fields with relatively low generation quality, such as the current medical history and diagnosis and treatment suggestions, the Few-shot learning scheme can be implemented, and the large model can be allowed to learn and imitate by providing excellent medical records written by real people to further improve the generation quality of the electronic medical records.

[0103] Since the large model is pre-trained based on prompts, after the content text and data text are input into the large model, the large model can be used to process the content text and data text, and based on the obtained text processing results, combined with prompts with medical professionalism, an electronic medical record that complies with the electronic medical record specification can be generated.

[0104] Exemplarily, the following table gives an example of a prompt.

[0105]

[0106]

[0107] On this basis, in order to better understand the above technical solutions as a whole, the following will be described by way of specific examples. Exemplarily, see Figure 6 , in the application scenario of the skin Internet hospital, obtain the doctor-patient dialogue generated during the online consultation. In the case where the doctor-patient dialogue is a text dialogue (i.e., in text form), the doctor-patient dialogue can be directly used as the content text; in the case where the doctor-patient dialogue is a voice dialogue (i.e., in voice form), a speech recognition model can be used to recognize the voice dialogue to obtain the content text. Obtain the skin images uploaded by the patient, and use a skin image recognition model to recognize the skin images to obtain image text. Obtain the skin medical reports uploaded by the patient, and then recognize the skin medical reports based on an OCR model and key information extraction technology, etc., to obtain report text. Obtain data text from the image text and report text. For example, the image text and report text can be used as data text respectively. Further, input the content text and data text into a large language model, so as to use the large language model to process the content text and data text and generate an electronic medical record for the skin Internet hospital.

[0108] In the above example, the skin images and skin medical reports during the skin consultation process can be recognized. The skin images and skin medical reports are the main information sources for doctors in the skin Internet hospital to collect the condition and make a diagnosis. Therefore, here, by combining the doctor-patient dialogue, skin images and skin medical reports to obtain multi-modal information, and then using a large language model to process this multi-modal information, a more complete and accurate electronic medical record can be generated, thus realizing the high-quality generation of electronic medical records.

[0109] Figure 7It is a flowchart of yet another electronic medical record generation method provided in an embodiment of the present invention. This embodiment is optimized based on the above technical solutions. In this embodiment, optionally, the electronic medical record at least includes key information in the data text. The above electronic medical record generation method further includes: sending the electronic medical record to the doctor's terminal so that the doctor's terminal displays the electronic medical record and modifies the electronic medical record in response to a modification operation input for the displayed electronic medical record; wherein, the modification operation is input when it is determined based on the key information that the electronic medical record needs to be modified. Among them, the explanations of the same or corresponding terms as those in the above embodiments are not repeated here.

[0110] See Figure 7 , the method of this embodiment may specifically include the following steps:

[0111] S410. In response to an electronic medical record generation instruction for a skin consultation service, obtain the interaction content between the provider and the acquirer of the skin consultation service, and a deep learning model for realizing electronic medical record generation.

[0112] S420. Obtain skin data of the skin lesion site questioned by the acquirer; wherein, the skin data includes at least one of a skin image and a skin medical report.

[0113] S430. Identify the interaction content to obtain a content text, and identify the skin data to obtain a data text.

[0114] S440. Input the content text and the data text into the deep learning model to generate an electronic medical record corresponding to the skin consultation service; wherein, the electronic medical record at least includes key information in the data text.

[0115] Among them, the key information can be understood as the information in the data text that can assist the doctor in reviewing whether the electronic medical record is accurate. Combining with the application scenarios that may be involved in the embodiments of the present invention, for example, it can be the skin lesion information obtained by identifying the skin image, and the skin lesion information can include the skin lesion site and / or skin lesion symptoms; it can also be the medical conclusion information obtained by identifying the skin medical report, such as the result of a fungal examination or a mite examination, etc.; of course, it can also be other key information, which is not specifically limited here.

[0116] The electronic medical record generated by the deep learning model based on the data text, specifically speaking, the medical record content in the electronic medical record, at least includes key information.

[0117] S450. Send the electronic medical record to the doctor's terminal so that the doctor's terminal displays the electronic medical record and modifies the electronic medical record in response to a modification operation input for the displayed electronic medical record; wherein, the modification operation is input when it is determined based on the key information that the electronic medical record needs to be modified.

[0118] Among them, the electronic medical record is sent to the doctor's end so that the doctor's end can display the electronic medical record. On this basis, since the electronic medical record at least includes key information, the doctor can, based on the key information, review whether the displayed electronic medical record is accurate, and modify the electronic medical record in the case where it is determined through the review that the electronic medical record has deviations. Then, from the perspective of the execution of the doctor's end, in the case where the doctor's end receives the modification operation input by the doctor for the displayed electronic medical record, the electronic medical record can be modified by responding to the modification operation, which can help ensure the accuracy of the electronic medical record.

[0119] Exemplarily, referring to Figure 8 , after completing the online consultation, the doctor of the skin Internet hospital can directly click the "AI Medical Record" button on the doctor's end, so that the background service automatically generates an electronic medical record by combining the doctor-patient conversation, skin images, and skin medical reports, and displays the electronic medical record on the doctor's end. Then, the doctor can review the electronic medical record based on the key information in the displayed electronic medical record, and modify the electronic medical record in the case where it is considered through the review that the electronic medical record is incorrect. In the case where the doctor confirms that the electronic medical record is correct, it can be sent to the patient with one click, thereby greatly improving the service efficiency and service quality.

[0120] The technical solution of the embodiment of the present invention, by sending the electronic medical record to the doctor's end for display, enables the doctor to review whether the electronic medical record is correct through the key information in the displayed electronic medical record, and modify it in the case of errors, thereby ensuring the accuracy of the generated electronic medical record, and further improving the generation quality of the electronic medical record.

[0121] Figure 9 It is a structural block diagram of an electronic medical record generation device provided in an embodiment of the present invention. The device is used to execute the electronic medical record generation method provided in any of the above embodiments. The device and the electronic medical record generation method in the above embodiments belong to the same inventive concept. For the details not described in detail in the embodiment of the electronic medical record generation device, reference can be made to the embodiments of the above electronic medical record generation method. Referring to Figure 9 , the device may specifically include: a deep learning model acquisition module 510, a skin data acquisition module 520, and an electronic medical record generation module 530.

[0122] Among them, the deep learning model acquisition module 510 is configured to obtain the interaction content between the provider and the acquirer of the skin consultation service, and the deep learning model for implementing the generation of the electronic medical record in response to the electronic medical record generation instruction for the skin consultation service;

[0123] The skin data acquisition module 520 is configured to obtain the skin data of the skin lesion site consulted by the acquirer; wherein, the skin data includes at least one of skin images and skin medical reports;

[0124] An electronic medical record generation module 530, configured to generate an electronic medical record corresponding to a skin consultation service according to a deep learning model, interaction content, and skin data.

[0125] Optionally, the electronic medical record generation module 530 may include:

[0126] A data text obtaining sub-module, configured to identify the interaction content to obtain content text, and identify the skin data to obtain data text;

[0127] An electronic medical record generation sub-module, configured to input the content text and the data text into the deep learning model to generate an electronic medical record corresponding to the skin consultation service.

[0128] On this basis, optionally, the deep learning model is a large language model.

[0129] On this basis, optionally, the large language model is pre-trained based on prompt words, and the prompt words are pre-set according to the electronic medical record specification;

[0130] The electronic medical record generation sub-module may include:

[0131] An electronic medical record generation unit, configured to input the content text and the data text into the large language model to generate an electronic medical record corresponding to the skin consultation service according to the large language model based on the prompt words, the content text, and the data text.

[0132] Another option is that the skin data includes skin images, and the data text obtaining sub-module may include:

[0133] A skin image recognition model obtaining unit, configured to obtain a pre-trained skin image recognition model;

[0134] A first data text obtaining unit, configured to input the skin image into the skin image recognition model to identify the skin image according to the skin image recognition model and obtain data text describing the skin image.

[0135] On this basis, optionally, the skin image recognition model is pre-trained through the following modules:

[0136] A training sample obtaining module, configured to obtain an original image recognition model and training samples; wherein, the original image recognition model includes a visual backbone network, a text feature extractor, and a text decoder, and the training samples include skin lesion images and description texts describing the skin lesion images;

[0137] A text obtaining module for inputting training samples into an original image recognition model to process skin lesion images according to a visual backbone network to obtain visual features, and process description texts according to a text feature extractor to obtain text features, and process visual features according to a text decoder to obtain visual texts, and process text features according to a text decoder to obtain text;

[0138] A skin image recognition model obtaining module for training the original image recognition model based on the similarity between visual texts and texts, and obtaining a skin image recognition model based on the trained visual backbone network and text decoder.

[0139] Another optional one is that the skin data includes a skin medical report. The data text obtaining sub-module may include:

[0140] A skin medical report updating unit for converting the skin medical report into an image format when the skin medical report is not in an image format, and using the obtained skin medical image after conversion as the skin medical report;

[0141] A report content obtaining unit for parsing the content of the skin medical report to obtain the report content;

[0142] A second data text obtaining unit for structurally extracting the report content to obtain the data text.

[0143] Another optional one is that the interaction content is a voice dialogue. The data text obtaining sub-module may include:

[0144] A speech recognition model obtaining unit for obtaining a pre-trained speech recognition model;

[0145] A content text obtaining unit for inputting the voice dialogue into the speech recognition model to recognize the voice dialogue according to the speech recognition model to obtain the content text.

[0146] Still another optional one is that the electronic medical record includes at least key information in the data text. The above-mentioned electronic medical record generating device may further include:

[0147] An electronic medical record sending module for sending the electronic medical record to the doctor side so that the doctor side can display the electronic medical record and modify the electronic medical record in response to a modification operation input for the displayed electronic medical record; wherein, the modification operation is input when it is determined based on the key information that the electronic medical record needs to be modified.

[0148] On this basis, optionally, the key information includes at least one of medical conclusion information and skin lesion information; wherein, the medical conclusion information is obtained by recognizing the skin medical report, and the skin lesion information is obtained by recognizing the skin image, and the skin lesion information includes at least one of the skin lesion site and skin lesion symptoms.

[0149] The electronic medical record generation device provided by the embodiment of the present invention, through the deep learning model acquisition module, in response to an electronic medical record generation instruction for a skin consultation service, acquires the interaction content between the provider and the acquirer of the skin consultation service, and a deep learning model that can be used to implement electronic medical record generation; through the skin data acquisition module, acquires the skin data of the skin lesion site questioned by the acquirer, and the skin data may include at least one of a skin image and a skin medical report; through the electronic medical record generation module, generates an electronic medical record corresponding to the skin consultation service according to the deep learning model, the interaction content, and the skin data. The above device, based on the interaction content, combines skin data that can provide main auxiliary diagnosis information for doctors, so as to utilize the deep learning model to process multi-modal information (i.e., interaction content + skin data) to generate an electronic medical record, thereby improving the quality of electronic medical record generation for skin consultation services.

[0150] The electronic medical record generation device provided by the embodiment of the present invention can execute the electronic medical record generation method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the method.

[0151] It should be noted that in the embodiments of the above electronic medical record generation device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of the functional units are only for the convenience of mutual distinction and do not limit the protection scope of the present invention.

[0152] Figure 10 FIG. shows a schematic structural diagram of an electronic device 10 that can be used to implement the embodiments of the present invention. The electronic device is intended to represent various forms of digital computers, such as, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, a personal digital processor, a cellular phone, a smart phone, a wearable device (such as a helmet, glasses, a watch, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples and are not intended to limit the implementation of the present invention described herein and / or claimed.

[0153] As Figure 10As shown, the electronic device 10 includes at least one processor 11 and a memory communicatively connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc. Among them, the memory stores a computer program executable by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0154] Multiple components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disc, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0155] The processor 11 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include but are not limited to a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the electronic medical record generation method.

[0156] In some embodiments, the electronic medical record generation method can be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the electronic medical record generation method described above can be executed. Alternatively, in other embodiments, the processor 11 can be configured to execute the electronic medical record generation method by any other appropriate means (e.g., by means of firmware).

[0157] The various embodiments of the systems and techniques described above in this specification can be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that receives data and instructions from, and transmits data and instructions to, a storage system, at least one input device, and at least one output device.

[0158] The computer programs for implementing the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions / operations specified in the flowchart and / or block diagram to be implemented. The computer programs can be executed entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

[0159] In the context of the present invention, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0160] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the electronic device. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0161] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected to each other by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), blockchain network, and the Internet.

[0162] The computing system can include a client and a server. The client and the server are generally far from each other and usually interact through a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system and solves the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services.

[0163] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved, and no limitation is imposed herein.

[0164] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. An electronic medical record generation method, characterized in that, Including: In response to an electronic medical record generation instruction for a skin consultation service, obtaining the interaction content between the provider and the acquirer of the skin consultation service, and a deep learning model for implementing electronic medical record generation; Obtaining skin data of the skin lesion site inquired by the acquirer; wherein, the skin data includes at least one of a skin image and a skin medical report; Generating an electronic medical record corresponding to the skin consultation service according to the deep learning model, the interaction content, and the skin data.

2. The method according to claim 1, wherein The generating the electronic medical record of the skin consultation service according to the deep learning model, the interaction content, and the skin data includes: Identifying the interaction content to obtain a content text, and identifying the skin data to obtain a data text; Inputting the content text and the data text into the deep learning model to generate an electronic medical record corresponding to the skin consultation service.

3. The method according to claim 2, wherein The deep learning model is a large language model.

4. The method according to claim 3, wherein The large language model is pre-trained based on prompt words, and the prompt words are preset according to electronic medical record specifications; The inputting the content text and the data text into the deep learning model to generate an electronic medical record corresponding to the skin consultation service includes: Inputting the content text and the data text into the large language model to generate an electronic medical record corresponding to the skin consultation service according to the large language model based on the prompt words, the content text, and the data text.

5. The method according to claim 2, wherein The skin data includes the skin image, and the identifying the skin data to obtain a data text includes: Obtaining a pre-trained skin image recognition model; Inputting the skin image into the skin image recognition model to identify the skin image according to the skin image recognition model and obtain a data text describing the skin image.

6. The method according to claim 5, characterized in that, The skin image recognition model is pre-trained through the following steps: Obtaining an original image recognition model and training samples; wherein, the original image recognition model includes a visual backbone network, a text feature extractor, and a text decoder, and the training samples include skin lesion images and description texts describing the skin lesion images; Inputting the training samples into the original image recognition model to process the skin lesion images according to the visual backbone network to obtain visual features, process the description texts according to the text feature extractor to obtain text features, process the visual features according to the text decoder to obtain visual texts, and process the text features according to the text decoder to obtain text texts; Based on the similarity between the visual text and the text text, implementing the training of the original image recognition model to obtain the skin image recognition model based on the trained visual backbone network and the text decoder.

7. The method according to claim 2, characterized in that, The skin data includes the skin medical report, and the identifying the skin data to obtain a data text includes: In the case where the skin medical report is not in an image format, convert the skin medical report into the image format, and use the resulting skin medical image as the skin medical report; Perform content analysis on the skin medical report to obtain the report content; Perform structured extraction on the report content to obtain the data text.

8. The method according to claim 2, wherein The interaction content is a voice conversation. The recognition of the interaction content to obtain the content text includes: Obtain a pre-trained speech recognition model; Input the voice conversation into the speech recognition model to recognize the voice conversation according to the speech recognition model and obtain the content text.

9. The method according to claim 2, wherein The electronic medical record at least includes the key information in the data text. The method further includes: Send the electronic medical record to the doctor side so that the doctor side displays the electronic medical record and modifies the electronic medical record in response to a modification operation input for the displayed electronic medical record; wherein, the modification operation is input when it is determined based on the key information that the electronic medical record needs to be modified.

10. The method according to claim 9, wherein, The key information includes at least one of medical conclusion information and skin lesion information; wherein, the medical conclusion information is obtained by recognizing the skin medical report, and the skin lesion information is obtained by recognizing the skin image. The skin lesion information includes at least one of the skin lesion site and the skin lesion symptom.

11. An electronic medical record generation device, characterized in that, Includes: A deep learning model acquisition module, configured to obtain the interaction content between the provider and the acquirer of the skin consultation service and the deep learning model for implementing the generation of the electronic medical record in response to an electronic medical record generation instruction for the skin consultation service; A skin data acquisition module, configured to obtain the skin data of the skin lesion site questioned by the acquirer; wherein, the skin data includes at least one of a skin image and a skin medical report; An electronic medical record generation module, configured to generate an electronic medical record corresponding to the skin consultation service according to the deep learning model, the interaction content, and the skin data.

12. An electronic device, characterized in that, Includes: At least one processor; And A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor. The computer program is executed by the at least one processor to cause the at least one processor to execute the electronic medical record generation method according to any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a processor to implement the electronic medical record generation method according to any one of claims 1-10 when executed.

Images