Information processing device, control method for information processing device, and program

Abstract

This invention provides an information processing device, a control method for the information processing device, and a program that can streamline the creation of record documents based on the details of surgeries performed by medical practitioners. [Solution] The information processing device 1 includes an image acquisition unit 31 that acquires medical image information captured during surgery or endoscopic examination, a medical information acquisition unit 32 that acquires audio information including findings spoken during surgery or endoscopic examination, and a record document creation unit 36 ​​that creates a record document based on the medical image information and audio information.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a control method for an information processing apparatus, and a program.

Background Art

[0002] Conventionally, a technique for assisting in creating a record document such as a report on treatment has been known. For example, Patent Document 1 describes this type of technique. Patent Document 1 relates to an endoscope examination report creation support system that supports the creation of an endoscope examination report.

[0003] Patent Document 1 describes an endoscope examination report creation support system including a recording means for recording a report template, a first control means for performing control to read out the template from the recording means and display it on a display device in response to a report creation instruction for an unperformed examination, and a first storage means for storing the report created by editing the template displayed on the display device by the first control means in a recording medium.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] By the way, due to the work style reform, it is desired to reduce the burden on medical practitioners. In a record document that records the content of a medical act such as surgery, in addition to the basic information of the surgery, it is necessary to describe the findings of medical practitioners, etc. The creation of findings records and surgery records has been one of the burdens on medical practitioners. Although the report creation support system in the prior art can improve the efficiency of report creation, the actual content of the surgery varies depending on the type of disease, the condition of the treatment target, etc., and there is room for improvement in reducing the labor of inputting findings.

[0006] This invention has been made in view of the above circumstances, and aims to provide an information processing device, a control method for the information processing device, and a program that can streamline the creation of record documents based on the content of surgeries and endoscopic procedures performed by medical practitioners. [Means for solving the problem]

[0007] To achieve the above objective, one aspect of the present invention is an information processing device comprising: an image acquisition means for acquiring medical image information of a surgical procedure; a medical information acquisition means for acquiring audio information including findings spoken during the medical procedure; and a record document creation means for creating a record document based on the medical image information and the audio information of the findings. [Effects of the Invention]

[0008] According to the present invention, the creation of record documents based on the content of surgeries and endoscopic procedures performed by medical practitioners can be streamlined. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows a report generation system to which an information processing device according to one embodiment of the present invention is applied. [Figure 2] This is a block diagram showing the hardware configuration of the information processing device according to this embodiment. [Figure 3] This is a functional block diagram showing an example of the functional configuration of the information processing device according to this embodiment. [Figure 4] This figure shows an example of bookmark information displayed on medical video footage. [Figure 5] This flowchart shows an example of the process of acquiring medical image information and medical information using an information processing device. [Figure 6] This flowchart shows an example of a series of processes used by an information processing device to create a record document. [Modes for carrying out the invention]

[0010] <System Configuration> One embodiment of the present invention will be described below with reference to the drawings. Figure 1 shows a report generation system 100 to which the information processing device 1 according to one embodiment of the present invention is applied. Figure 2 shows a schematic diagram of the configuration of the report generation system 100 to which the information processing device 1 according to one embodiment of the present invention is applied.

[0011] The report generation system 100 is configured by interconnecting an information processing device 1, a video camera 2, a procedure information acquisition device 3, and a medical practitioner terminal 4 via a predetermined network N such as the Internet.

[0012] Information processing device 1 is a report generation device that automatically creates record documents related to medical procedures such as surgery, and is a server having calculation processing functions and communication functions. This information processing device 1 can be implemented, for example, by electronic equipment such as a server device or a personal computer.

[0013] Video camera 2 is an imaging device (medical image acquisition device) for collecting medical video information captured during medical procedures in the operating room 10, and captures medical images during the medical procedure. The medical images captured by video camera 2 may be temporarily saved on a local computer and later transferred to remote storage, or they may be streamed to remote storage during the surgical procedure.

[0014] The data store where medical images are stored may include, in addition to the information processing device 1, one or more servers connected to a network N such as the Internet, generally referred to as cloud servers or cloud storage. In other words, the video signal emitted from the video camera 2 is streamed to a local computer or a remote server such as cloud storage and saved as a video file, and when the information processing device 1 performs editing of the medical image information, it retrieves the medical image information from a predetermined location where the medical image information is stored.

[0015] The procedure information acquisition device 3 is a device for collecting medical information related to medical procedures performed in the endoscopy room 10. In the example shown in Figure 1, the procedure information acquisition device 3 consists of a microphone 3A and a vital signs measurement device 3B.

[0016] Microphone 3A acquires medical information during medical procedures, such as the voice of the medical practitioner, including the attending physician D. Microphone 3A performs noise analysis on the acquired voice information to determine whether or not certain sounds should be excluded. In other words, it excludes sounds emitted when using medical equipment, such as the "beep" or "beep" sounds made when using an electrosurgical unit, and general background noise from speaking through a mask. It also removes sounds caused by the friction of the arm when the patient is wearing a gown.

[0017] While microphone 3A can be a type that is fixed to the wall or ceiling of the room, a headset wireless microphone that is worn (fixed) on the head, as shown in Figure 1, is preferable. This headset wireless microphone is suitable for medical procedures because it eliminates the need to hold microphone 3A in the hand, leaving both hands free, and because it is positioned near the mouth, it can reliably capture sound without dropping any. In the case of a headset wireless microphone, a transmitter is required separately. This transmitter is attached to the end of the cable extending from microphone 3A and transmits the sound input to microphone 3A to receiver R using radio waves or infrared rays. The audio information is then stored on a local computer or a remote server such as cloud storage, similar to medical video information, and the information processing device 1 retrieves the audio information from the designated location where the medical video information is stored when editing the audio information.

[0018] The biological information measurement device 3B is a device for checking the state of a patient during a procedure, and measures and acquires medical information during a medical act. Examples of the biological information measurement device 3B include a nerve function examination device that measures a patient's motor evoked potential (MEP), somatosensory evoked potential (SEP), auditory brainstem response (ABR), and visual evoked potential (VEP), and an electroencephalogram monitor (BIS) that measures an electroencephalogram. In addition, in order to check the accidental situation of the patient, for example, medical devices such as a biological information monitor, a respiratory function monitor, a circulatory dynamics monitor, and a sedation monitor may be used as the procedure information acquisition device 3. In the present embodiment, by using the biological information measurement device 3B as the procedure information acquisition device 3, an example of performing an editing that automatically adds bookmark information to the medical video information being photographed by the video camera 2 will be described as learning an important part of the procedure when dealing with an accidental situation. Examples of accidental situations include obtaining alarm information such as an alarm sound from the biological information measurement device of 3B, obtaining error information from the biological information measurement device of 3B, and the level of sound information output from the microphone 3A being equal to or higher than a predetermined value. In addition, an accidental situation is also assumed to include a sudden change (numerical increase or numerical decrease equal to or higher than a predetermined value within a predetermined period) in the biological information monitor. Examples of sudden changes include a drop in blood pressure, an increase in the pulse rate, and a decrease in a numerical value related to oxygenation (for example, blood oxygen concentration, etc.). In addition, as an accidental situation, there is a case where the number displayed on the nerve function monitor decreases from the beginning of the operation. In addition, an accidental situation includes a case where the amount of bleeding suddenly increases (equal to or higher than a predetermined value within a predetermined period).

[0019] The medical practitioner terminal 4 is a computer operated by a medical practitioner who creates a record document, and is a device having an arithmetic processing function and a communication function. This medical practitioner terminal 4 is realized by, for example, a desktop or laptop computer, a tablet computer, a smartphone, or the like.

[0020] <Hardware Configuration> FIG. 2 is a block diagram showing the hardware configuration of the information processing apparatus 1 according to the present embodiment. The information processing apparatus 1 includes a CPU (Central Processing Unit) 11 as a processor, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a bus 14, an input / output interface 15, an output unit 16, an input unit 17, a storage unit 18, a communication unit 19, and a drive 20.

[0021] The CPU 11 is a processor that controls the operation of the entire information processing apparatus 1 and executes various processes according to a program recorded in the ROM 12 or a program loaded from the storage unit 18 into the RAM 13. The program is, for example, a program such as an operating system or firmware for operating each part of the information processing apparatus 1, or application software for realizing the functional blocks described later. The RAM 13 also appropriately stores data and the like necessary for the CPU 11 to execute various processes. The CPU 11, ROM 12, and RAM 13 are interconnected via the bus 14. The input / output interface 15 is also connected to this bus 14.

[0022] The output unit 16, input unit 17, storage unit 18, communication unit 19, and drive 20 are connected to the input / output interface 15. The output unit 16 is composed of a display, a speaker, etc., and outputs various information as images and sounds. The input unit 17 is composed of a keyboard, a mouse, etc., and inputs various information. The storage unit 18 is composed of a hard disk, a DRAM (Dynamic Random Access Memory), an SSD (Solid State Drive), etc., and stores various data. The communication unit 19 communicates with other devices via a network N including the Internet.

[0023] The drive 20 is appropriately equipped with removable media 21, which may consist of a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory. Programs read from the removable media 21 by the drive 20 are installed in the storage unit 18 as needed. The removable media 21 can also store various types of data stored in the storage unit 18, just like the storage unit 18.

[0024] The hardware configuration described here is merely an example. The computer described in this embodiment, including the information processing device 1, may have the same configuration as that in Figure 2, or it may have a different configuration. Furthermore, the computer may consist of two or more computers. The medical practitioner terminal 4 in Figures 1 and 2 is, for example, a smartphone, tablet, or personal computer having a configuration similar to the hardware configuration shown in Figure 2.

[0025] <Functional configuration> Figure 3 is a functional block diagram showing an example of the functional configuration of the information processing device 1 according to this embodiment. As shown in Figure 3, in the CPU 11 of the information processing device 1, the following functions operate during operation: video acquisition unit 31, medical information acquisition unit 32, bookmark creation unit 33, surgical process identification unit 34, findings identification unit 35, record document creation unit 36, edited medical video transmission unit 37, video extraction unit 38, related information setting unit 39, record document confirmation unit 40, data management unit 41, etc. In other words, in this embodiment, the information processing device 1 functions as a device including the video acquisition unit 31, medical information acquisition unit 32, bookmark creation unit 33, surgical process identification unit 34, findings identification unit 35, record document creation unit 36, edited medical video transmission unit 37, video extraction unit 38, related information setting unit 39, record document confirmation unit 40, and data management unit 41, when various programs (OS, applications, etc.) stored in an auxiliary storage device (e.g., ROM) are loaded into the main memory device (RAM) and executed by the CPU 11.

[0026] The video acquisition unit 31 (video acquisition means) acquires medical video information related to the procedures of medical treatment. In other words, the video acquisition unit 31 acquires medical video information captured by the video camera 2.

[0027] The medical information acquisition unit 32 (medical information acquisition means) acquires medical information related to procedures that occur in conjunction with medical procedures. Medical information refers to, for example, voices emitted by the medical practitioner during a medical procedure, and biological information (measured values) such as evoked potentials and electroencephalograms that are measured by attaching a device to the patient during the medical procedure. In other words, the medical information acquisition unit 32 acquires voice information as medical information from the microphone 3A that acquires voice information, and also acquires biological information as medical information from the biological information measuring device 3B that acquires the patient's biological information.

[0028] The bookmark creation unit 33 (bookmark creation means) associates and adds bookmark information to a predetermined position in the medical video information associated with the medical information, when the medical information acquired by the medical information acquisition unit 32 meets specific conditions set in advance.

[0029] The specific conditions are, for example, specific words (trigger words), such as "findings," "important," and "recording." Medical practitioners can improve the accuracy of identifying the findings portion in audio information by uttering these specific words when stating their findings. For example, if the voice of a specific medical practitioner is recognized and the text of that voice contains the aforementioned specific words, the specific conditions are met. The voice of each medical practitioner may also be identified, and the voice information of medical practitioners may be registered in advance to enable voiceprint authentication. In addition, specific conditions also include cases where there is a large (above a predetermined value) fluctuation in monitor data (e.g., heart rate data). As a result, when comments from medical practitioners that meet the specific conditions, or voices of specific medical practitioners distinguished by voiceprints are acquired, bookmark information can be automatically added to the medical video information, or only comments uttered by specific medical practitioners can be added to the medical video information. The specific conditions are stored in the memory unit 18.

[0030] "Bookmark information" refers to tag data assigned to specific locations in medical video information over time. These are visually recognizable markers that enable mechanical searching. This chronological location can be rephrased as the time axis of the medical video information. "Assigning bookmark information" means associating searchable tag data that attracts the attention of the user viewing the document with a specific time; in other words, it means associating a marker (bookmark) with a specific location in the medical video information. Therefore, by assigning bookmark information to medical video information, in addition to user operation while viewing the medical video, it becomes possible to search for it within the software, providing a navigation function that allows users to find and instantly move (jump) to the desired location in the medical video. This also makes it possible to generate separate video data that extracts only the video near the locations to which the aforementioned bookmark information has been assigned.

[0031] Figure 4 shows an example of bookmark information displayed on medical video information. Figure 4 shows a medical video information editing screen on the monitor screen 50 of a medical practitioner terminal 4 displaying medical video information, which includes at least a video screen 51, a frame 52, and bookmarks 53. In Figure 4, three bookmarks 53 are shown, associated with the frame 52. Note that the example in Figure 4 is merely one example of a display method, and various methods can be used to display bookmark information.

[0032] Furthermore, when medical information includes biological information measured during a medical procedure, the bookmark creation unit 33 may associate and assign bookmark information to medical video information if the measured value of the biological information that meets specific conditions deviates from a predetermined range. This makes it possible to generate video data for confirming the response when the above-mentioned accidental situation occurs.

[0033] The bookmark creation unit 33 may also add bookmark information even if the trigger word described above is not used. For example, the bookmark creation unit 33 may recognize the start and end of an audio (utterance) and add bookmark information to that audio. Furthermore, the bookmark creation unit 33 may add bookmark information to multiple audios even if the audio is interrupted. In this case, if the audio is interrupted, one bookmark is added to these multiple audios based on the relationship between the first audio (utterance) and the second audio. Regarding the relationship, for example, the text obtained from the two audios may be analyzed by a predetermined analysis unit, and if the degree of similarity is higher than a predetermined value, it may be determined that there is a relationship, and if the degree of similarity is lower than a predetermined value, it may be determined that there is no relationship. In addition, if the time gap between the two audios is shorter than a predetermined time, it may be determined that there is a relationship.

[0034] The surgical and endoscopic treatment process identification unit 34 (surgical process identification means) identifies the surgical process based on the medical video information acquired by the video acquisition unit 31 and the medical information acquired by the medical information acquisition unit 32. The surgical and endoscopic treatment process identification unit 34 identifies the surgical process corresponding to the temporal position (predetermined range) of the medical video information. The surgical and procedural steps may include, for example, the following: In gastrointestinal endoscopic examination (procedure), first, the tip of the endoscope is inserted through a natural opening in the human body that is continuous with the digestive tract, such as the mouth, nose, or anus, to the deepest point of observation (the duodenum in the upper digestive tract, or the terminal ileum in small bowel endoscopy and colonoscopy, etc.) (step 1). Subsequently, while withdrawing the endoscope, the entire surface of the digestive tract is observed, taking into account the anatomical features to identify the observation area (Step 2). If a lesion is found, if hemostasis is necessary, a hemostatic device such as a clip or local injection needle is inserted; if it is for pathological diagnosis, forceps are inserted; and if it is for tumor removal, an electrosurgical unit is inserted and the procedure is carried out (Step 3). Furthermore, when observing the deep parts of the gastrointestinal wall or the vicinity of the outside, an ultrasound probe attached to the tip of the endoscope is used (endoscopic ultrasound) (step 4). In endoscopic ultrasound, a high-frequency ultrasound probe is inserted into the digestive tract to identify the observation site from the ultrasound image and endoscopic findings. When identifying a lesion, a tissue sampling needle is inserted for tissue diagnosis, or, in cases such as cyst drainage, a guidewire is placed and a stent is inserted (step 5). For gallstones in the bile duct and pancreatic duct, or strictures caused by cancer, an endoscope is inserted into the descending part of the duodenum, and a catheter is inserted into the main and accessory papillae, which are the openings of the bile duct and pancreatic duct, to perform procedures such as stone lithotripsy or stent placement. During this procedure, a contrast agent is used to identify the anatomical site under X-ray fluoroscopy before performing the procedure (step 6). Furthermore, after endoscopic diagnosis and treatment, the endoscopist should promptly review all aspects of the treatment process, record the details in the medical record using appropriate terminology, and, if tissue or bile samples are collected, prepare a request form for pathological diagnosis or bacterial analysis. In addition, the medical record should be saved with the recorded images linked to each finding.

[0035] The method for identifying surgical and endoscopic procedures is not particularly limited. The surgical and endoscopic procedure identification unit 34 may identify surgical and endoscopic procedures using a learning model 45 for identifying surgical and endoscopic procedures stored in the memory unit 18. For example, the learning model 45 for identifying surgical and endoscopic procedures is constructed by performing supervised learning on images contained in medical video information and audio information contained in medical information using a dataset in which surgical procedures are used as labels. Alternatively, the learning model 45 for identifying surgical and endoscopic procedures may be constructed by performing supervised learning on text converted from images and audio information contained in medical video information using a dataset in which surgical and endoscopic procedures are used as labels. In this case, the input data input to the learning model 45 for identifying surgical and endoscopic procedures is also text converted from audio information.

[0036] Furthermore, the surgical and endoscopic procedure identification unit 34 may perform preprocessing on the audio information included in the medical information to remove unnecessary audio that is not related to the identification of findings. In addition, the surgical and endoscopic procedure identification unit 34 may present the identified surgical procedure to the medical practitioner terminal 4 and accept confirmation or correction requests.

[0037] The findings identification unit 35 (finding identification unit means) performs a findings identification process to identify findings during a medical procedure for which medical video information and medical information are to be acquired.

[0038] The findings identification process will now be explained. In the findings identification process, the findings identification unit 35 identifies the location related to the findings from the medical information based on the identified surgical procedure. In this embodiment, the findings identification unit 35 identifies the portion where bookmark information is set within the range identified in the surgical and endoscopic treatment procedures as a finding. In the findings identification process, the bookmark information used to identify the findings is determined.

[0039] The record document creation unit 36 ​​(record document creation means) performs a text creation process to create identified findings (comments) and a record document creation process to create a record document of the medical procedure based on the comments. The record document includes information that can identify the surgical or endoscopic treatment record, such as the name of the physician who performed the surgery or endoscopy and the name of the patient, the date the surgery or endoscopy was performed, the start and end times of the surgery or endoscopy, the surgical procedure performed, the name of the disease, findings, and images extracted from medical imaging information.

[0040] The text creation process will now be described. In the text creation process, the record document creation unit 36 ​​generates text from audio information and biometric information corresponding to the findings using various speech recognition technologies. In this embodiment, the record document creation unit 36 ​​generates text from the portion of the audio information corresponding to the specified range of bookmark information, such as surgery and endoscopic treatment procedures, and also converts biometric information into text. The record document creation unit 36 ​​may also recognize the speaker based on the audio information. For example, the record document creation unit 36 ​​can distinguish the speaker from the vocal cord information in the audio information. Alternatively, speakers may be registered in advance, or predetermined attributes of the speaker (such as the surgeon, supervisor, approver, nurse, or laboratory technician) may be registered after acquiring medical video information to distinguish between speakers and create record documents. This makes it possible to distinguish between the physician who performed the procedure and the physician who approved the procedure (for example, a senior physician).

[0041] Furthermore, the document creation unit 36 ​​may generate text in real time, or it may generate text after surgery or endoscopic treatment using a predetermined speech processing engine (offline processing). This allows for correction of mistranslations or misrecognitions that may occur in real time. Also, when generating text in real time, the document creation unit 36 ​​may generate the text using a separate server in a cloud environment, or it may generate the text using a separate server in an on-premises environment. In this case, the text may be generated using a predetermined AI (Artificial Intelligence) in the on-premises environment. This allows confidential information such as medical information to be converted into text in a secure environment.

[0042] The process for creating a record document will now be described. In the record document creation process, the record document creation unit 36 ​​performs the process of creating a record document based on text. The method of creating the record document is not particularly limited. The record document creation unit 36 ​​may use a learning model 46 for record document creation stored in the memory unit 18 to identify surgical and endoscopic treatment procedures. For example, the learning model 46 for record document creation is constructed by performing supervised learning on text converted from audio information using a dataset in which record documents are used as labels. Alternatively, the record document creation unit 36 ​​may create a record document by rule-based text conversion processing without using the learning model 46 for record document creation. In rule-based text conversion processing, the record document creation unit 36 ​​performs the process of reflecting the text in a pre-set record document format. In rule-based text conversion processing, the record document creation unit 36 ​​may perform formatting processing to adjust the appearance and endings of sentences when reflecting the text.

[0043] The edited medical video transmission unit 37 stores the medical video information, to which bookmark information has been added, into an external storage device. An example of such an external storage device is a cloud server.

[0044] When bookmark information is added, the video extraction unit 38 extracts medical video information for a predetermined time (for example, 5 minutes before and after) the location where the bookmark information is added as edited medical video information. In other words, it generates medical video data that is different from the original medical video data edited by adding bookmark information, by extracting only the important parts. This allows users to efficiently view desired medical videos related to important surgical procedures without having to search the original medical video data.

[0045] The related information setting unit 39 (related information setting means) sets related information such as time information, image information, medical video information, and edited medical video information for the record document created by the record document creation unit 36.

[0046] Time information indicates the time a medical procedure was performed, the time corresponding to a finding contained in a record document, or the elapsed time in medical video information. If multiple findings are recorded in a record document, time information can be associated with each finding. By associating the time corresponding to a finding or the elapsed time in medical video information with the record document, the relevant section of the record document in the medical video information can be quickly identified.

[0047] Image information is an image corresponding to a finding contained in the record document. If a finding is described in multiple places in the record document, image information is associated with each finding. The image information may be a still image of the part of the medical imaging information corresponding to the finding, or it may be an image showing the biological information corresponding to the finding. Furthermore, the image information is associated with the record document in a format that is embedded in the record document or in a format that is displayed when the relevant part of the record document is selected.

[0048] Edited medical video information is an edited video corresponding to the findings contained in the document. If the document contains findings in multiple places, edited medical video information is associated with each finding. Edited medical video information is associated with the document, for example, by being embedded in the document or by being played when the relevant section of the document is selected.

[0049] Furthermore, the related information setting unit 39 may perform processing to reflect at least one of time information, image information, and video information in the recorded document. For example, the related information setting unit 39 can add the time at which a finding was stated to the recorded document, in association with the text indicating the finding. In addition, the related information setting unit 39 may embed thumbnail images of image information or edited medical video information, or embed links to display image information or edited medical video information, in the recorded document so that the image information or video information corresponding to the finding can be viewed.

[0050] The record document verification unit 40 (record document verification means) performs a presentation process that presents the record document to the medical practitioner terminal 4 of the medical practitioner who performed the medical procedure, a modification process that accepts modifications to the record document from the medical practitioner terminal 4, and a learning process that reflects the results of the modification process in the learning model. The record document verification unit 40 may output the record document and a prompt including commands to a trained model including a large-scale language model (so-called generative AI, etc.), execute an acquisition process to obtain the output data output by the trained model, and present the record document, which has been corrected based on the output data, to the medical practitioner terminal 4, etc.

[0051] In the presentation process, the document verification unit 40 presents the document to the medical practitioner terminal 4 via the network N, triggered, for example, by a viewing request from the medical practitioner terminal 4 or the completion of document creation. The method of presenting the document by the document verification unit 40 is not particularly limited. For example, the document verification unit 40 can present the document to a medical practitioner who is logged into an application running on the medical practitioner terminal 4 by sending and receiving chat-style messages and attaching data. Alternatively, the document verification unit 40 may send an email containing a link to view the document to the medical practitioner's pre-registered email address, so that the document is displayed when the medical practitioner clicks the link in the email.

[0052] In the correction process, the record document verification unit 40 accepts corrections made by the medical practitioner to the created record document. The method by which the record document verification unit 40 corrects the record document is not particularly limited. For example, the record document verification unit 40 can accept corrections from the medical practitioner through a chat application running on the medical practitioner terminal 4. Alternatively, the record document verification unit 40 may accept corrections to the record document by detecting that the medical practitioner has made corrections to the record document displayed on the medical practitioner terminal 4.

[0053] During the learning process, the document verification unit 40 saves the modifications made by the medical practitioner as learning data. This learning data is used to train learning models such as the surgical procedure identification learning model 45 and the document creation learning model 46. The timing of the learning is not particularly limited; it may be performed when a certain amount of learning data has been accumulated, or it may be performed sequentially. Furthermore, when surgical procedure identification or text conversion is performed using a rule-based system, the learning data may be stored and used during rule-based maintenance.

[0054] An example of how training data is used in a training model is described below. If the correction is to a surgical procedure in a recorded document, the recorded document verification unit 40 saves the corrected surgical procedure as training data, associating it with the medical video information and medical information that are input data for the surgical procedure identification training model 45. The saved training data is used to train the surgical and endoscopic procedure identification training model 45. If the correction is to text in a recorded document, the recorded document verification unit 40 saves the corrected text as training data, associating it with the audio information that are input data for the recorded document creation training model 46. The saved training data is used to train the recorded document creation training model 46.

[0055] The data management unit 41 (data management means) receives requests to view recorded documents or medical video information, and transmits the recorded documents and edited video information stored in the recorded document database 47. In other words, when a medical practitioner terminal 4 requests to view recorded documents, the data management unit 41 provides the requested recorded documents, or provides medical video information, image information, edited medical video information, time information, etc., associated with the recorded documents.

[0056] <Processing details> Figure 5 is a flowchart showing an example of the process of acquiring medical image information and medical information by the information processing device 1.

[0057] In step S11, the video acquisition unit 31 acquires medical video information related to the procedure of a medical treatment, and the medical information acquisition unit 32 performs processing to acquire audio information and biological information as important medical information related to the procedure that arises in connection with the medical treatment.

[0058] Next, in step S12, the bookmark creation unit 33 performs a process to determine whether or not the audio information and biometric information contain information that meets specific conditions (such as voice or measured values). If the bookmark creation unit 33 determines that the information meets the specific conditions (S12-YES), the process proceeds to step S13; otherwise (S12-NO), it proceeds to step S14.

[0059] In step S13, the bookmark creation unit 33 performs a process of associating and attaching bookmark information to a predetermined position in the time sequence of medical image information associated with medical information.

[0060] In step S14, the video acquisition unit 31 continues to perform a process to determine whether or not there is any medical video information (frames) to acquire. If the video acquisition unit 31 determines that there is no medical video information to acquire (S14-YES), the series of operations in the information processing device 1 ends (END). On the other hand, if the video acquisition unit 31 determines that there is medical video information to acquire (S14-NO), the process proceeds to step S11. The above is the flow of the process for acquiring medical video information and medical information by the information processing device 1.

[0061] Next, we will explain the process flow for creating record documents after acquiring medical image information and medical information. Figure 6 is a flowchart showing an example of a series of processes for creating record documents by the information processing device 1.

[0062] First, in step S101, the surgical and endoscopic treatment process identification unit 34 performs a process to remove unnecessary audio information from the audio information included in the medical information. For example, the surgical and endoscopic treatment process identification unit 34 removes portions of audio information that are clearly unrelated to the surgical process, endoscopic treatment process, or findings as unnecessary audio information.

[0063] In step S102, the surgical and endoscopic procedure identification unit 34 identifies surgical procedures corresponding to a predetermined time range of the medical video information based on the medical video information and medical information. The surgical and endoscopic procedure identification unit 34 inputs the medical video information and audio information contained in the medical information as input data into the surgical and endoscopic procedure identification learning model 45, and identifies surgical procedures within the medical video information.

[0064] In step S103, the finding identification unit 35 identifies the location of the finding from the medical information based on the identified surgical or endoscopic procedure. The finding identification unit 35 identifies the portion of the medical image information where bookmark information for the identified surgical or endoscopic procedure is set as the finding.

[0065] In step S104, the document creation unit 36 ​​generates text from the audio information and biometric information corresponding to the findings. The document creation unit 36 ​​generates text from the audio information and biometric information corresponding to the scope of the bookmark information identified as the surgical or endoscopic procedure.

[0066] In step S105, the document creation unit 36 ​​performs the process of creating a document based on the generated text. The document creation unit 36 ​​may create a document by inputting the text into the document creation learning model 46 stored in the storage unit 18, or it may create a document by embedding the text in a predetermined format.

[0067] In step S106, the related information setting unit 39 sets related information for the record document, such as time information, image information, and medical video information. The related information setting unit 39 also inputs time information indicating the time corresponding to the surgical or endoscopic procedure and findings into the record document. The related information setting unit 39 also associates image and medical video information corresponding to the surgical or endoscopic procedure and findings with the record document so that it can be viewed when the record document is viewed. The related information setting unit 39 may also embed thumbnails of the image information and medical video information within the record document, or it may make the image information and medical video information available for simultaneous display along with the record document.

[0068] In step S107, the record document verification unit 40 presents the created record document to the medical practitioner. The record document verification unit 40 displays the created record document on the screen of the medical practitioner's terminal 4. The record document is associated with time information, image information, and medical video information by the related information setting unit 39, and the medical practitioner can also verify the time information, image information, and medical video information along with the record document. The medical video information includes the edited medical video information mentioned above.

[0069] In step S108, the record document verification unit 40 performs a process to determine whether or not the record document has been modified on the medical practitioner terminal 4 by the medical practitioner. As a result, if the record document verification unit 40 determines that the document has been modified by the medical practitioner (S108-YES), the process proceeds to step S109; otherwise (S108-NO), the process proceeds to step S111.

[0070] In step S109, the record document verification unit 40 modifies the record document to reflect the modifications made on the medical practitioner terminal 4. Although not shown in the diagram, in the modification step, the record document verification unit 40 may output the record document and a prompt including commands to a trained model including a large-scale language model (so-called generative AI, etc.), execute an acquisition process to obtain the output data output by the trained model, and present the modified record document based on the output data to the medical practitioner terminal 4, etc. For example, the record document verification unit 40 presents the output data described above to the medical practitioner terminal 4 before accepting corrections from the medical practitioner. The presented content may include, for example, the contents of the record document (such as typographical errors or inaccuracies in the content), or alternative or additional item suggestions for each category of the record document presented to the medical practitioner terminal 4. The medical practitioner then reviews the presented content and sends whether additions or corrections are necessary. The record document verification unit 40 then corrects the record document to reflect the corrections made on the medical practitioner terminal 4. The processes in steps S108 and S109 may be repeated in this manner. This allows for interactive correction of the record document with the generating AI, etc., enabling the creation of accurate record documents without the medical practitioner having to unilaterally make corrections on the spot.

[0071] In step S110, the document verification unit 40 saves the modifications made by the medical practitioner as training data. This training data is used to train training models such as the training model 45 for identifying surgical and endoscopic procedures and the training model 46 for creating document records. As described above, the training data can be reflected at any time.

[0072] In step S111, the data management unit 41 registers the record document that has been confirmed or corrected by the medical practitioner in the record document database 47. This makes the record document accessible to the medical practitioner or any person authorized to confirm the record document.

[0073] <Other> (1) As described above, the information processing device 1 of this embodiment includes an image acquisition unit 31 that acquires medical image information of surgery, a medical information acquisition unit 32 that acquires audio information of findings spoken during surgery or endoscopic examination, and a record document creation unit 36 ​​that creates a record document based on the medical image information and the audio information of findings. This system effectively reduces the burden on medical practitioners in creating documentation by generating record documents that reflect the actual surgical procedures based on medical video information captured during the procedure and audio information of findings during the procedure. By having medical practitioners consciously speak about their findings during the procedure, the content of those findings can be accurately reflected in the documentation. Furthermore, it is possible to verify the content of the documentation from the medical video information, extract images and attach them to the documentation, or attach edited medical video information to the documentation.

[0074] (2) The information processing device 1 of this embodiment further includes a surgical process identification unit 34 that identifies a surgical process based on medical video information and audio information, and a findings identification unit 35 that identifies the portion of the audio information containing findings by the medical practitioner within the scope of the identified surgical and endoscopic procedures, and a record document creation unit 36 ​​that creates a record document based on the audio information corresponding to the identified portion of the findings. This allows medical practitioners to record findings made during surgery or endoscopic procedures in a way that corresponds to the surgical process.

[0075] (3) The information processing device 1 of this embodiment further includes a bookmark creation unit 33 that associates and adds bookmark information to a predetermined position in the medical video information associated with the audio information when the audio information meets specific conditions set in advance, and a finding identification unit 35 which includes a record document creation unit 36 ​​that identifies the part of the finding based on the position of the bookmark information. This allows for the identification of specific findings using bookmark information, and enables easy confirmation of the portion of the medical video information corresponding to the findings based on the location of the bookmark information. By registering medical video information together with the recorded documents in the record document database 47, medical practitioners can efficiently verify the contents of the recorded documents and review past cases using the medical video information in conjunction.

[0076] (4) The information processing device 1 of this embodiment further includes an associated information setting unit 39 that associates at least one of the following with a recorded document: medical video information, image information extracted from medical video information, edited medical video information obtained by editing medical video information, and time information indicating the time or duration during a medical procedure. This allows for easy verification of medical video information, image information, edited medical videos, and time information along with the recorded documents. Furthermore, by incorporating medical video information, image information, edited medical videos, and time information into the recorded documents so that they can be viewed within the documents themselves, the recorded documents can be created with greater precision.

[0077] (5) The information processing device 1 of this embodiment also includes a record document confirmation unit 40 that presents the record document to the medical practitioner's terminal 4 and can accept modifications to the record document from the medical practitioner's terminal 4. This prevents situations where unintended findings are recorded in the document and registered in the document database 47.

[0078] (6) In this embodiment, the document verification unit 40 stores the received corrections as learning data, and the document creation unit 36 ​​creates a document so that the learning data is reflected. As a result, the corrections made by the medical practitioner are reflected, and by continuing to use the information processing device 1, the accuracy of creating record documents can be improved.

[0079] (7) Furthermore, the control method of the information processing device 1 of this embodiment includes a medical image acquisition step of acquiring medical image information of surgery or endoscopic examination, a medical information acquisition step of acquiring audio information of findings spoken during surgery, and a record document creation step of creating a record document based on the medical image information and the audio information of findings. (8) Furthermore, the program of this embodiment causes the computer to perform steps similar to those of the control method for the information processing device 1.

[0080] (2-1) Furthermore, the information processing device 1 of this embodiment also comprises: an image acquisition means for acquiring medical image information of surgery or endoscopic examination; a medical information acquisition means for acquiring audio information including findings spoken during the surgery; a record document creation means for creating a record document based on the medical image information and the audio information of the findings; and a record document confirmation means for presenting the record document to the medical practitioner's terminal and accepting corrections to the record document from the medical practitioner's terminal, wherein the record document confirmation means outputs the record document and a prompt including commands to a trained model including a large-scale language model, executes an acquisition process to acquire output data output by the trained model, and presents the record document corrected based on the output data. This allows for appropriate corrections to be made to the generated record documents using a large-scale language model, without requiring any revisions from doctors or other medical professionals.

[0081] (2-2) The record document verification means may store the received corrections as learning data, and the record document creation means may create the record document so that the learning data is reflected. As a result, the corrections made by the medical practitioner are reflected, and by continuing to use the information processing device 1, the accuracy of creating record documents can be improved.

[0082] (2-3) The present invention also includes a control method for an information processing device, comprising: a medical image acquisition step of acquiring medical image information of a surgery; a medical information acquisition step of acquiring audio information including findings spoken during the surgery; a record document creation step of creating a record document based on the medical image information and the audio information of the findings; and a record document confirmation step of presenting the record document to a medical practitioner's terminal and accepting modifications to the record document from the medical practitioner's terminal, wherein the record document confirmation step outputs a prompt including the record document and commands to a trained model including a large-scale language model, executes an acquisition process to acquire output data output by the trained model, and presents the record document modified based on the output data.

[0083] (2-4) The present invention is a program for causing a computer to execute the following steps: a medical image acquisition step of acquiring medical image information of a surgery; a medical information acquisition step of acquiring audio information including findings spoken during the surgery; a record document creation step of creating a record document based on the medical image information and the audio information of the findings; and a record document confirmation step of presenting the record document to a medical practitioner's terminal and accepting modifications to the record document from the medical practitioner's terminal, wherein the record document confirmation step can also be considered as a program that outputs a prompt including the record document and commands to a trained model including a large-scale language model, executes an acquisition process to acquire output data output by the trained model, and presents the record document modified based on the output data.

[0084] Although one embodiment of the present invention has been described above, the above-described embodiment is merely an example to facilitate understanding of the present invention and is not intended to limit the interpretation of the present invention. Furthermore, the present invention is not limited to the above-described embodiment, and modifications, improvements, etc., to the extent that the objectives of the present invention can be achieved are included in the present invention.

[0085] For example, in the above embodiment, the document creation unit 36 ​​targets voice information and biometric information for text conversion, but the configuration is not limited to this. For example, biometric information may be excluded from the text conversion target.

[0086] Furthermore, the information processing device 1 may also include a timestamp acquisition unit in addition to the configuration of the above embodiment. When bookmark information is assigned, the timestamp acquisition unit requests the issuance of a timestamp token with respect to the hash value of the edited video information, and stores the issued timestamp token together with the edited video information in the recorded document database 47. The request for a timestamp token is made, for example, to a Time Stamping Authority (TSA), and the timestamp token issued by this TSA is obtained. In other words, the timestamp acquisition unit sends a timestamp request to the timestamp server of the time stamping authority operated by the TSA, specifying the hash value of the edited video information. The time authentication authority (timestamp server) adds the time received by the time authentication authority to the hash value of the edited video information and returns a timestamp token with the time authentication authority's digital certificate attached. The timestamp acquisition unit then acquires the timestamp token returned from the time authentication authority's timestamp server. This allows the timestamp token to prove when the data was processed, making it a highly reliable source of medical video information that can be used against third parties in the event of a dispute. Furthermore, it can prove that the data has not been tampered with, allowing for the identification of speakers from audio recordings to determine who made which comments, thus ensuring credibility is not questioned in any dispute. Furthermore, it is preferable to add an electronic signature to demonstrate the authenticity of this information. An electronic signature is a technique used to prove that a file has not been tampered with since it was verified by the signer. Therefore, by combining a timestamp and an electronic signature, it becomes possible to verify that an electronic file was created by the signer at a certain time and has not been forged or tampered with since.

[0087] Furthermore, the series of processes described above can be executed by hardware or implemented by software. In other words, the functional configuration described above is merely illustrative and not particularly limiting. That is, it is sufficient for the information processing system to have the functionality to execute the series of processes described above as a whole, and the functional blocks used to realize this functionality are not particularly limited to the examples given above.

[0088] Furthermore, the location of the functional blocks is not particularly limited to Figure 3 and can be arbitrary. For example, the functional blocks of a server may be transferred to other devices, or conversely, the functional blocks of other devices may be transferred to a server. Also, a single functional block may consist of hardware alone, software alone, or a combination of both.

[0089] When a series of processes are executed by software, the programs that make up that software are installed on a computer or other device from a network N or storage medium. The computer may be a computer built into dedicated hardware. Alternatively, the computer may be a computer capable of performing various functions by installing various programs, such as a server, a general-purpose smartphone, or a personal computer.

[0090] Such a recording medium containing a program may consist not only of removable media (not shown) distributed separately from the main device to provide the program to the user, but also of a recording medium provided to the user in a state where it is pre-installed in the main device. Since the program can be distributed via network N, the recording medium may be installed on or accessible from a computer connected to or capable of connecting to network N.

[0091] In this specification, the step of describing a program to be recorded on a recording medium includes not only processes that are performed chronologically in that order, but also processes that are not necessarily performed chronologically, but are executed in parallel or individually. Furthermore, in this specification, the term "system" refers to an overall system composed of multiple devices, means, etc. [Explanation of Symbols]

[0092] 1. Information Processing Device 31 Video Acquisition Unit 32 Medical Information Acquisition Department 33 Bookmark Creation Department 34. Department for Specific Procedures in Surgery and Endoscopic Procedures 35 Findings Identification Department 36. Document Creation Department 37 Editing Medical Video Transmission Department 38. Video Extraction Unit 39 Related Information Setting Section 40. Document Verification Department 41 Data Management Department 100 Report Generation System

Claims

1. A means of acquiring medical image information obtained by imaging surgical procedures and endoscopic examinations, A medical information acquisition means for acquiring audio information including findings spoken during the aforementioned surgery, A means for creating a record document that creates a record document based on the medical video information and the audio information of the findings, A record document verification means that displays the record document on the medical practitioner's terminal and accepts modifications to the record document from the medical practitioner's terminal, Equipped with, The record document verification means outputs the record document and a prompt including commands to a trained model including a large-scale language model, executes an acquisition process to acquire output data output by the trained model, and presents the record document modified based on the output data. Information processing device.

2. The aforementioned document verification means stores the received corrections as learning data, The record document creation means creates the record document so that the learning data is reflected. The information processing apparatus according to claim 1.

3. The medical image acquisition step involves obtaining medical image information of a surgical procedure, A medical information acquisition step involves acquiring audio information, including findings spoken during the aforementioned surgery, A record document creation step, which creates a record document based on the medical video information and the audio information of the findings, A record document confirmation step is provided, which involves presenting the record document to the medical practitioner's terminal and accepting modifications to the record document from the medical practitioner's terminal. Includes, In the aforementioned document verification step, the document is output to a trained model including a large-scale language model, an acquisition process is performed to obtain the output data output by the trained model, and the document is presented that has been corrected based on the output data. A method for controlling an information processing device.

4. The medical image acquisition step involves obtaining medical image information of a surgical procedure, A medical information acquisition step involves acquiring audio information, including findings spoken during the aforementioned surgery, A record document creation step, which creates a record document based on the medical video information and the audio information of the findings, A record document confirmation step is provided, which involves presenting the record document to the medical practitioner's terminal and accepting modifications to the record document from the medical practitioner's terminal. A program to be executed by a computer, In the aforementioned document verification step, the document is output to a trained model including a large-scale language model, an acquisition process is performed to obtain the output data output by the trained model, and the document is presented that has been corrected based on the output data. program.

Images