Assistance system, server device, and assistance method

Abstract

An assistance system according to the present invention comprises: a collection processing unit that collects sound information and image information associated with a medical care action by a user; a recording processing unit that extracts a first keyword, which is a keyword included in the sound information collected by the collection processing unit, and causes a recording information storage unit to store, as recording information, information in which the first keyword, and date and time information about the date and time at which the first keyword was uttered are associated with each other, and information in which the image information collected by the collection processing unit, and the date and time information about the date and time at which the image information was acquired are associated with each other; and an output control unit that causes a display unit to display a timeline display in which the first keyword and the image information are displayed side by side in chronological order on the basis of the recording information stored in the recording information storage unit.

Description

Support system, server device, and support method

[0001] The present disclosure relates to a support system, a server device, and a support method.

[0002] There is known a technique for reducing the burden on medical personnel such as doctors by making it easy to check data obtained during medical procedures (see, for example, Patent Document 1).

[0003] JP 2015-97065 A

[0004] The above-mentioned conventional technology makes it easy for medical professionals to check the progress of medical procedures by displaying text output extracted from keywords in the attending physician's voice instructions along with the time. However, in order to further reduce the burden on medical professionals, there is a need for technology that can record data obtained during medical procedures in more detail.

[0005] The present disclosure has been made to solve the above problems, and its purpose is to provide a support system, server device, and support method that can further reduce the burden on medical professionals and improve work efficiency and accuracy.

[0006] In order to solve the above problems, one aspect of the present disclosure is an assistance system including: a collection processing unit that collects sound information and image information associated with a medical procedure performed by a user; a recording processing unit that extracts a first keyword that is a keyword included in the sound information collected by the collection processing unit, and stores, as recorded information, information that associates the first keyword with date and time information when the first keyword was uttered, and information that associates the image information collected by the collection processing unit with date and time information when the image information was acquired, in a recorded information storage unit; and an output control unit that displays, on a display unit, a timeline display that displays the first keyword and the image information in chronological order based on the recorded information stored in the recorded information storage unit.

[0007] Furthermore, one aspect of the present disclosure is a server device that includes a collection processing unit that collects sound information and image information associated with a medical procedure performed by a user, and a recording processing unit that extracts a first keyword that is a keyword included in the sound information collected by the collection processing unit, and stores, as recorded information, information that associates the first keyword with date and time information when the first keyword was uttered, and information that associates the image information collected by the collection processing unit with date and time information when the image information was acquired, in a recorded information storage unit, wherein the recording processing unit displays, on a display device, a timeline display in which the first keyword and the image information are arranged in chronological order based on the recorded information stored in the recorded information storage unit.

[0008] Another aspect of the present disclosure is an assistance method in which a collection processing unit collects sound information and image information associated with a medical procedure performed by a user, a recording processing unit extracts a first keyword that is a keyword included in the sound information collected by the collection processing unit, and stores, as recorded information, information associating the first keyword with date and time information when the first keyword was uttered, and information associating the image information collected by the collection processing unit with date and time information when the image information was acquired, in a recorded information storage unit, and output control causes a display unit to display a timeline display in which the first keyword and the image information are arranged in chronological order based on the recorded information stored in the recorded information storage unit.

[0009] According to the present disclosure, the burden on medical professionals can be further reduced and work efficiency and accuracy can be improved.

[0010] 1 is a functional block diagram showing an example of a medical support system according to a first embodiment. FIG. 1 is a diagram showing an example of data in a terminal information storage unit according to the first embodiment. FIG. 2 is a diagram showing an example of data in a user information storage unit according to the first embodiment. FIG. 3 is a diagram showing an example of data in a collected sound information storage unit according to the first embodiment. FIG. 4 is a diagram showing an example of data in a collected image information storage unit according to the first embodiment. FIG. 5 is a diagram showing an example of data in a recorded information storage unit according to the first embodiment. FIG. 6 is a diagram showing an example of display of recorded information on a mobile terminal according to the first embodiment. FIG. 7 is a diagram showing an example of setting the display position and type of biometric data according to the first embodiment. FIG. 8 is a diagram showing an example of collection and recording processing of recorded information in the medical support system according to the first embodiment. FIG. 9 is a diagram showing an example of support processing of the medical support system according to the first embodiment. FIG. 10 is a flowchart showing an example of abnormality warning processing of the medical support system according to the first embodiment. FIG. 11 is a diagram showing an example of editing and processing processing of recorded information in the medical support system according to the first embodiment. FIG. 12 is a diagram showing an example of an archive screen of recorded information in the medical support system according to the first embodiment. FIG. 13 is a diagram showing an example of a summary text screen of recorded information in the medical support system according to the first embodiment. FIG. 14 is a flowchart showing an example of medical document generation processing of the medical support system according to the first embodiment. FIG. 15 is a functional block diagram showing an example of a medical support system according to a second embodiment. FIG. 16 is a diagram showing an example of operation of the medical support system according to the second embodiment. FIG. 10 is a diagram showing another example of the operation of the medical support system according to the second embodiment. FIG. 11 is a functional block diagram showing an example of a medical support system according to the third embodiment. FIG. 12 is a flowchart showing an example of a learning process of the medical support system according to the third embodiment. FIG. 13 is a flowchart showing an example of a personnel allocation process of the medical support system according to the third embodiment. FIG. 14 is a flowchart showing an example of a labor management process of the medical support system according to the third embodiment. FIG. 15 is a flowchart showing an example of a proposal process of the medical support system according to the third embodiment. FIG. 16 is a flowchart showing an example of an external provision process of the medical support system according to the third embodiment. FIG. 17 is a diagram explaining an example of the hardware configuration of each server device of the medical support system according to the embodiment.FIG. 2 is a diagram illustrating an example of the hardware configuration of each terminal device of the medical support system according to the embodiment.

[0011] A support system, a server device, and a support method according to an embodiment of the present disclosure will be described below with reference to the drawings.

[0012] (First embodiment) Fig. 1 is a functional block diagram showing an example of a medical support system 1 according to this embodiment. The medical support system 1 is a system that supports medical procedures by medical professionals in medical settings such as treatment rooms, operating rooms, and ambulances. Medical professionals who are users of the medical support system 1 include doctors, nurses, emergency personnel, etc.

[0013] 1, the medical support system 1 includes a mobile terminal 10, a headset device 20, an external image capture device 25, an information collection server 30, a management terminal 40, and a hospital management server 50. In this embodiment, the mobile terminal 10, the external image capture device 25, the information collection server 30, the management terminal 40, and the hospital management server 50 are connected to a network NW1 and can communicate with each other.

[0014] The medical support system 1 is an example of a support system. The medical support system 1 may include a plurality of mobile terminals 10, headset devices 20, and external image capture devices 25.

[0015] The mobile terminal 10 is a terminal device carried by a medical professional who is a user, and is, for example, a smartphone, a tablet terminal, or the like.

[0016] The mobile terminal 10 includes a network communication unit 11, an input unit 12, a display unit 13, a microphone 14, a speaker 15, an imaging unit 16, a wireless communication unit 17, a terminal memory unit 18, and a terminal control unit 19.

[0017] The NW communication unit 11 is a communication device that can be connected to the network NW1 via, for example, a wireless local area network (LAN). The NW communication unit 11 performs data communication between the information collection server 30 and the external image capture device 25 via the network NW1. For example, the NW communication unit 11 receives image data captured by the external image capture device 25 from the external image capture device 25. Furthermore, for example, the NW communication unit 11 transmits collected information collected by a collection processing unit 191 of the mobile terminal 10 (described later) to the information collection server 30.

[0018] The input unit 12 is an input device such as a keyboard or a touch sensor of a touch screen, and receives various information input by a user. The user can input, via the input unit 12, various setting information for using the medical support system 1, which is stored in a setting information storage unit 181 (described later), for example.

[0019] The display unit 13 is, for example, a liquid crystal display, and displays (outputs) various information used by the mobile terminal 10. The display unit 13 displays, for example, sound data (sound information) collected in connection with a medical procedure and recorded information based on image data.

[0020] The microphone 14 collects sounds around the mobile terminal 10 and outputs sound data (sound information) to the terminal control unit 19. The microphone 14 is a built-in microphone of the mobile terminal 10 and is an example of a sound collection unit.

[0021] The speaker 15 is a built-in speaker of the mobile terminal 10 and outputs various sound data. For example, when a user instructs the speaker 15 to check the sound data corresponding to the timeline display of recorded information described below, the speaker 15 outputs (emits sound) the sound data included in the recorded information (sound data corresponding to the original data). The speaker 15 also outputs (emits sound), for example, a warning sound from the information collection server 30, sound results of various information searches, and various sound suggestions.

[0022] The imaging unit 16 is, for example, a camera device that captures still images and video images, and is a built-in camera of the mobile terminal 10. The imaging unit 16 captures various images and outputs image data (image information) to the terminal control unit 19. Note that the image information (image data) includes still image data and video data.

[0023] The wireless communication unit 17 is, for example, a Bluetooth (registered trademark) communication device, and performs data communication with the headset device 20. The wireless communication unit 17 receives, for example, audio data and image data acquired by the headset device 20, and outputs the data to the terminal control unit 19.

[0024] The terminal storage unit 18 is a storage unit realized by a memory (not shown) included in the mobile terminal 10, and stores various information used by the mobile terminal 10. The terminal storage unit 18 includes a setting information storage unit 181.

[0025] The setting information storage unit 181 stores various setting information of the mobile terminal 10 for using the medical support system 1. The setting information storage unit 181 stores, for example, setting information for hands-free operations that perform various operations without contact in the medical support system 1. The setting information related to hands-free operations includes, for example, information on actions corresponding to various operations such as a start operation for starting the collection of sound data and image data, a stop operation for stopping the collection of sound data and image data, an operation for requesting an information search, an operation for requesting support, and the like.

[0026] Here, the hands-free operation includes, for example, an operation by the user's voice, an operation by the user's gaze, a motion operation using the hand or finger, etc. The setting information storage unit 181 also stores, for example, information on voice keywords corresponding to various operations as setting information related to voice-based operations.

[0027] The terminal control unit 19 is a functional unit realized by, for example, causing a processor (not shown) including a CPU (Central Processing Unit) to execute a program stored in the terminal storage unit 18. The terminal control unit 19 comprehensively controls the mobile terminal 10 and executes various processes.

[0028] The terminal control unit 19 includes a collection processing unit 191, an output control unit 192, and a setting processing unit 193. The collection processing unit 191, the output control unit 192, and the setting processing unit 193 are realized by causing a processor (not shown) to execute, for example, an application program for using the medical support system 1.

[0029] The collection processing unit 191 collects sound data (sound information) and image data (image information) associated with a medical procedure performed by a user. Here, the sound data includes, for example, voice data uttered by the user (medical worker), and the image data includes still image data or video data captured by the user using a headset device 20 (described later) and still image data or video data captured by an external image capture device 25.

[0030] The collection processing unit 191 starts collecting sound data or image data when, for example, a start operation to start collecting sound data or image data is detected by a hands-free operation by a medical professional. The collection processing unit 191 starts collecting sound data or image data when, for example, sound data acquired from the microphone 14 or the headset device 20 is converted into text by a voice recognition processing unit 331 of the information collecting server 30 (described later), a keyword is extracted by a recording processing unit 333, and the extracted keyword matches a keyword corresponding to the start operation stored in the setting information storage unit 181. Alternatively, the collection processing unit 191 starts collecting sound data or image data when, for example, image data acquired from the imaging unit 16, the headset device 20, or the external imaging device 25 is analyzed by an image recognition processing unit 332 of the information collecting server 30 (described later), a predetermined action of the user is detected, and the action matches the action corresponding to the start operation stored in the setting information storage unit 181.

[0031] The collection processing unit 191 transmits the collected sound data or image data to the information collecting server 30 via the NW communication unit 11. The sound data transmitted to the information collecting server 30 is associated with date and time information and stored in a collected sound information storage unit 324 (described later). The image data transmitted to the information collecting server 30 is associated with date and time information and stored in a collected image information storage unit 325 (described later).

[0032] Furthermore, the collection processing unit 191 terminates the collection of sound data or image data when a termination operation for terminating the collection of sound data or image data is detected by a hands-free operation. For example, the collection processing unit 191 terminates the collection of sound data or image data when sound data acquired from the microphone 14 or the headset device 20 is converted into text by a voice recognition processing unit 331 of the information collecting server 30 (described later) to extract a keyword, and the extracted keyword matches a keyword corresponding to the termination operation stored in the setting information storage unit 181. Alternatively, the collection processing unit 191 terminates the collection of sound data or image data when, for example, image data acquired from the imaging unit 16, the headset device 20, or the external imaging device 25 is analyzed by an image recognition processing unit 332 of the information collecting server 30 (described later) to extract a predetermined user action, and the extracted action matches the action corresponding to the termination operation stored in the setting information storage unit 181.

[0033] In addition, when the collection processing unit 191 detects, in addition to data collection, for example, an operation to request an information search or an operation to request assistance through hands-free operation, the collection processing unit 191 transmits a processing request corresponding to the detected request operation to the information collection server 30 via the NW communication unit 11.

[0034] The output control unit 192 controls the operation of the display unit 13 and causes the display unit 13 to display various information. Specifically, the output control unit 192 causes the display unit 13 to display a timeline display in which keywords and image information are arranged in chronological order based on the record information stored in the record information storage unit 327 of the information collecting server 30 (described later). The output control unit 192 causes the display unit 13 to display date and time information (e.g., time information) associated with keywords and image data based on the record information. Here, the keywords are keywords extracted from sound data or image data. Details of the keywords and the display content of the output control unit 192 will be described later. The output control unit 192 also controls the operation of the speaker 15 and causes the speaker 15 to output (emit sound) various information. Specifically, the output control unit 192 causes the speaker 15 to output (emit sound) the sound data included in the record information. The output control unit 192 also causes the speaker 15 to output (emit) sounds such as warning sounds from the information collection server 30, sounds indicating the results of various information searches, and sounds indicating various suggestions.

[0035] The setting processing unit 194 executes various setting processes and various setting change processes for using the medical support system 1 in response to, for example, operation input by the user via the input unit 12. The setting processing unit 194 acquires various setting information via the input unit 12 and stores it in the setting information storage unit 181. The setting information is, for example, setting information for hands-free operation that performs various operations in the medical support system 1 without contact.

[0036] The headset device 20 is, for example, a device worn on the head of a medical professional, and mainly transmits, via wireless communication, voice data of the medical professional and image data of an object that the medical professional is looking at to the mobile terminal 10. The headset device 20 includes a wireless communication unit 21, a microphone 22, and an imaging unit 23.

[0037] The wireless communication unit 21 is, for example, a Bluetooth (registered trademark) communication device, and performs data communication via wireless communication with the wireless communication unit 17 of the mobile terminal 10. The wireless communication unit 21 transmits, for example, audio data and image data to the mobile terminal 10 via wireless communication.

[0038] The microphone 22 mainly collects sounds made by medical personnel during medical procedures and generates sound data. The microphone 22 is an example of a sound collection unit, and the sound data generated by the microphone 22 is transmitted to the mobile terminal 10 via the wireless communication unit 21.

[0039] The imaging unit 23 is, for example, a camera device that captures still and video images, and captures an image including an object that a medical professional is looking at during a medical procedure, for example, to generate image data. The image data generated by the imaging unit 23 is transmitted to the mobile terminal 10 via the wireless communication unit 21.

[0040] The external imaging device 25 is installed in a medical setting (such as a treatment room, operating room, or ambulance) and captures images of the situation at the medical setting. For example, the external imaging device 25 is placed in front of a medical device such as a measuring device or a medication device, and captures images of the screen of the measuring device, images showing the medication status, etc. during a medical procedure, and generates (collects) various image data. Alternatively, the external imaging device 25 is placed in a location such as a ceiling that provides a bird's-eye view of the medical setting (particularly, a treatment table, an operating table, or inside an ambulance), and is a fixed-point camera that captures images of the situation of medical personnel and patients (including affected areas) during medical procedures. The external imaging device 25 is an example of an imaging unit. The image data generated (collected) by the external imaging device 25 is transmitted to the mobile terminal 10 via the network NW1.

[0041] The management terminal 40 is a terminal device for managing the medical support system 1, and may be, for example, a personal computer, a smartphone, or a tablet terminal. The management terminal 40 confirms, edits, and processes recorded information recorded during medical procedures, and controls the creation of medical documents such as electronic medical records, medical receipts (medical fee statements), and prescriptions. The management terminal 40 includes a network communication unit 41, an input unit 42, a display unit 43, a terminal storage unit 44, and a terminal control unit 45.

[0042] The NW communication unit 41 is a communication device connectable to the network NW1 via, for example, a wireless LAN or a wired LAN. The NW communication unit 41 performs data communication with, for example, the information collecting server 30 via the network NW1. The NW communication unit 41 transmits, for example, various control information (various processing requests) for managing the medical support system 1 to the information collecting server 30. The NW communication unit 41 receives, for example, various display information based on the recorded information from the information collecting server 30.

[0043] The input unit 42 is an input device such as a keyboard, a touch sensor of a touch screen, a mouse, or the like, and accepts various information input by operations of a user or administrator of the medical support system 1. For example, the input unit 42 accepts a request from a user to edit record information and a request from a user to create medical documents.

[0044] The display unit 43 is, for example, a liquid crystal display, and displays (outputs) various information used by the management terminal 40. The display unit 43 displays, for example, recording information, search information for recorded information, editing information, archive information, and the like.

[0045] The terminal storage unit 44 is a storage unit realized by a memory (not shown) provided in the management terminal 40, and stores various information used by the management terminal 40. The terminal control unit 45 is a functional unit realized, for example, by causing a processor (not shown) including a CPU to execute a program stored in the terminal storage unit 44. The terminal control unit 45 comprehensively controls the management terminal 40 and executes various processes. The terminal control unit 45 includes a display control unit 451.

[0046] The display control unit 451 displays various information received from the information collection server 30 via the NW communication unit 41 on the display unit 43. The display control unit 451 displays, for example, a screen for viewing, searching, correcting, processing, etc., of recorded information recorded during medical procedures on the display unit 43. The display control unit 451 also displays, for example, an archive screen, a summary screen, etc., which are a list of medical procedures for the patient, on the display unit 43.

[0047] The hospital management server 50 is a server device that, for example, registers and manages patient information, manages electronic medical records, outputs prescriptions and medical fee statements, etc., and is an existing hospital management system. The hospital management server 50 is connected to the network NW1 and is capable of data communication with, for example, the information collection server 30 and the management terminal 40.

[0048] The information collection server 30 is a server device that performs support processing to reduce the burden on medical professionals, such as recording various information (recorded information) associated with medical procedures performed by medical professionals and displaying display information based on the recorded information on the display unit 13 of the mobile terminal 10. The information collection server 30 is capable of communicating with the mobile terminal 10, the management terminal 40, and the hospital management server 50 via the network NW1. The information collection server 30 includes a NW communication unit 31, a server storage unit 32, and a server control unit 33.

[0049] The NW communication unit 31 is a communication device that can be connected to the network NW1 via, for example, a wireless LAN or a wired LAN. The NW communication unit 31 performs data communication between, for example, the mobile terminal 10 and the management terminal 40 via the network NW1. The NW communication unit 31 receives collected information such as audio data and image data from the mobile terminal 10. The NW communication unit 31 also transmits display information to the mobile terminal 10 or the management terminal 40.

[0050] The server storage unit 32 is a storage unit realized by a memory (not shown) provided in the information collecting server 30 , and stores various information used by the information collecting server 30 .

[0051] The server memory unit 32 includes a terminal information memory unit 321, a user information memory unit 322, a learning result memory unit 323, a collected sound information memory unit 324, a collected image information memory unit 325, a keyword information memory unit 326, a recording information memory unit 327, a format memory unit 328, and a medical document memory unit 329.

[0052] The terminal information storage unit 321 stores information on the mobile terminal 10 and the external image capture device 25 that can be used in the medical support system 1. An example of data stored in the terminal information storage unit 321 will now be described with reference to FIG.

[0053] 2 is a diagram showing an example of data stored in the terminal information storage unit 321 according to the present embodiment. As shown in FIG. 2, the terminal information storage unit 321 stores terminal information in which a terminal ID, a MAC address, a user ID, a device type, and collected information are associated with each other.

[0054] Here, the terminal ID is identification information that identifies the mobile terminal 10 or the external imaging device 25. The MAC address is address information that identifies a device on the network NW1. The user ID is user identification information that identifies the user (healthcare worker) who uses the mobile terminal 10 or the external imaging device 25. The device type indicates the type of device (whether it is the mobile terminal 10 or the external imaging device 25), and the collection information indicates the type of data that can be collected.

[0055] In the example shown in Figure 2, the device with terminal ID "MN001" has a MAC address of "XX:XX:XX:XX:XX:XX:XX" and a user ID of "U0001". It also shows that the device type is a "mobile terminal" and the collected information is "sound, images".

[0056] 1, the user information storage unit 322 stores information about users (medical workers) who use the medical support system 1. Here, an example of data in the user information storage unit 322 will be described with reference to FIG.

[0057] FIG. 3 is a diagram showing an example of data stored in the user information storage unit 322 in this embodiment. As shown in FIG. 3, the user information storage unit 322 stores a user ID, a name, job type information, and a status in association with each other. Here, the job type information indicates the user's job type, such as nurse, doctor, radiologist, or paramedic. The status indicates the user's current working status. The status is used when requesting assistance from a waiting user or when requesting opinions on medical procedures, patient symptoms, etc.

[0058] 3, the user (healthcare worker) with user ID "U0001" has the name "△△Ichiro" and the occupation information is "nurse." The example also shows that the user's condition is "medication / treatment in progress."

[0059] It also shows that the user (healthcare worker) with user ID "U0002" has the name "Taro XX" and the occupation information is "doctor." It also shows that the status of this user is "on standby."

[0060] 1 , the learning result storage unit 323 stores the learning results of various machine learning processes performed by the learning processing unit 337. The learning result storage unit 323 stores, as learning results, for example, a voice recognition model, an image recognition model, an anomaly detection model for treatments and medications by medical professionals, and a proposed model for medical procedures.

[0061] The collected sound information storage unit 324 stores collected sound data (raw data). An example of data stored in the collected sound information storage unit 324 will now be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of data stored in the collected sound information storage unit 324 in this embodiment.

[0062] 4, the collected sound information storage unit 324 stores a data ID, a terminal ID, date and time information, and sound information in association with each other. Here, the data ID is data identification information assigned to a series of sound data collected by the collection processing unit 191 during the period from when the start operation of sound data collection is detected to when the end operation is detected, and the terminal ID is identification information that identifies the mobile terminal 10 that collected the data. The date and time information indicates the date and time when the sound data was collected (e.g., the date and time when collection started). The sound information is, for example, information that indicates the sound data, such as the file name of the sound data.

[0063] 4 shows that sound data with a data ID of "DT00001" was acquired by the mobile terminal 10 with a terminal ID of "MT001" at the date and time information of "20XX / 11 / 01 10:15" (10:15 on November 1, 20XX). The example also shows that the sound data is stored as sound information "SXXX1."

[0064] Returning to the explanation of Fig. 1, the collected image information storage unit 325 stores collected image data (raw data). Here, an example of data stored in the collected image information storage unit 325 will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of data stored in the collected image information storage unit 325 in this embodiment.

[0065] 5, the collected image information storage unit 325 stores a data ID, a terminal ID, date and time information, and image information in association with each other. Here, the data ID is data identification information assigned to a series of image data collected by the collection processing unit 191 during the period from when the start operation of image data collection is detected to when the end operation is detected, and the terminal ID is identification information that identifies the mobile terminal 10 (or external imaging device 25) that collected the image data. The date and time information indicates the date and time when the image data was collected (e.g., the date and time when collection started). The image information is, for example, information that indicates the image data, such as the file name of the image data (e.g., video data).

[0066] 5 shows that image data with a data ID of "DT10001" was acquired by a mobile terminal 10 with a terminal ID of "MT001" at date and time information of "20XX / 11 / 01 10:15" (10:15 on November 1, 20XX). The image data is also stored as image information "MVXXX1."

[0067] 1 , the keyword information storage unit 326 stores information indicating keywords extracted from the sound data or image data. Here, the keywords include, for example, at least one of terms related to the treatment (e.g., “hemostasis,” “infusion,” etc.), terms related to the tools used in the treatment (e.g., “scalpel,” “forceps,” etc.), terms related to medication, and terms related to the patient's biological information (e.g., “blood pressure,” “pulse rate,” “respiratory rate,” etc.).

[0068] The recording information storage unit 327 stores recording information based on the collected sound data and image data. The recording information is stored in the recording information storage unit 327 by a recording processing unit 333 (described later). Here, an example of data stored in the recording information storage unit 327 will be described with reference to FIG. 6 .

[0069] FIG. 6 is a diagram showing an example of data stored in the record information storage unit 327 in this embodiment. As shown in FIG. 6, the record information storage unit 327 stores record information in which a patient name, date and time information, a user ID, medical procedure information, an attached image, and original data are associated with each other. Here, the patient name indicates the name of the patient who underwent the medical procedure, and the user ID is identification information of the medical professional. The patient name is an example of identification information for identifying a patient. The medical procedure information indicates, for example, keywords extracted from sound data or image data and indicates the details of the medical procedure. The attached image indicates image data displayed together with the keywords. The original data indicates the sound data (audio data) and image data (video data, etc.) that are the source of the record information.

[0070] For example, in the example shown in Figure 6, the recorded information for a patient named "XX Hanako" shows that the medical treatment information was "consciousness level clear" spoken by a user with a user ID of "U0001" at date and time information 20XX / 11 / 01 21:25 (21:25 on November 1, 20XX), and that an image of "G0001" was recorded as an attached image. It also shows that the original data of this recorded information is "DT00001, DT10001." Note that the original data is the data IDs of the collected sound information storage unit 324 and the collected image information storage unit 325.

[0071] In this way, the record information storage unit 327 stores, as record information, information associating a keyword with date and time information when the keyword was uttered, and information associating collected image information with date and time information when the image information was acquired. The record information storage unit 327 may also store, as record information, information associating image data with date and time information and keywords detected from the image data.

[0072] The recorded information storage unit 327 may also store information such as patient information, information about the medical personnel (doctors, nurses) who provided the care, the details of the medical procedure (time-series data linked to time information), the time required for the treatment (the time required for the treatment), and the progress after the treatment (the effects of the treatment).

[0073] 1, the format storage unit 328 stores format information (format information) for creating medical documents (e.g., electronic medical records, prescriptions, etc.). The format storage unit 328 stores, for example, identification information indicating the medical document and the format information in association with each other.

[0074] The medical document storage unit 329 stores medical documents generated from record information by a medical document generation unit 336 (described later). The medical documents include, for example, electronic medical records, prescriptions, and medical fee statements.

[0075] The server control unit 33 is a functional unit realized by, for example, causing a processor (not shown) including a CPU to execute a program stored in the server storage unit 32. The server control unit 33 comprehensively controls the information collection server 30 and executes various processes. The server control unit 33 includes a voice recognition processing unit 331, an image recognition processing unit 332, a recording processing unit 333, a support processing unit 334, an editing processing unit 335, a medical document generation unit 336, and a learning processing unit 337.

[0076] The voice recognition processing unit 331 converts the sound data (voice data) collected by the collection processing unit 191 into text data (character data). Various known methods may be used to convert sound data into text data. For example, the voice recognition processing unit 331 may convert sound data into text data based on the learning result (voice recognition model) stored in the learning result storage unit 323.

[0077] The image recognition processing unit 332 detects text data (character data) included in the image data (video data, etc.) collected by the collection processing unit 191. For example, the image recognition processing unit 332 extracts text data indicating the patient's biometric data (heart rate, blood pressure, etc.) and the type or amount / flow rate of the medication used from an image of the screen of a medical device such as a measuring device or a medication administration device captured by the external imaging device 25 through image recognition. The image recognition processing unit 332 also detects a preset motion (hands-free operation) from the image data collected by the collection processing unit 191. Various known methods may be used to detect the text data and the preset motion from the image data. For example, the image recognition processing unit 332 may detect the text data and the preset motion from the image data based on the learning results (image recognition model) stored in the learning result storage unit 323.

[0078] The recording processing unit 333 extracts keywords included in the sound data collected by the collection processing unit 191, and stores, as record information, information associating the keywords with date and time information (e.g., time information) when the keywords were uttered, and information associating the image data collected by the collection processing unit 191 with date and time information (e.g., time information) when the image data was acquired, in the record information storage unit 327. At this time, the recording processing unit 333 creates record information including information such as the patient name and user ID (ID of the medical professional), and stores the record information in the record information storage unit 327 in the format shown in FIG. 6 described above. Specifically, the recording processing unit 333 extracts keywords included in the sound data by checking whether the text data converted from the sound data by the voice recognition processing unit 331 includes preset keywords stored in the keyword information storage unit 326 (e.g., the above-mentioned terms related to treatment, terms related to jigs used for treatment, terms related to medication, and terms related to the patient's biological information). In this specification, the keyword extracted from the sound data is also referred to as the first keyword.

[0079] The recording processing unit 333 may also extract keywords contained in the image data collected by the collection processing unit 191 and store information associating the image data with date and time information (e.g., time information) at which the image data was acquired and the keywords detected from the image data as recorded information in the recorded information storage unit 327. In this case, the recording processing unit 333 extracts keywords contained in the image data by checking whether text data (character data) detected from the image data by the image recognition processing unit 332 contains pre-set keywords stored in the keyword information storage unit 326 (e.g., the above-mentioned terms related to treatment, terms related to jigs used in treatment, terms related to medication, and terms related to the patient's biological information). Note that in this specification, keywords extracted from image data are also referred to as second keywords.

[0080] The recording processing unit 333 also creates record information for each period during which sound data and image data are collected, and stores the created record information in the record information storage unit 327. The recording processing unit 333 may also identify a speaker who uttered a keyword from among multiple medical professionals based on the sound data (voice data), and store information in which identification information identifying the speaker is associated with the keyword and date and time information as record information in the record information storage unit 327. That is, the recording processing unit 333 stores record information such as that shown in FIG. 6 in the record information storage unit 327.

[0081] In addition, when the recording information is created by aggregating the collected information collected by the collection processing unit 191 from multiple mobile terminals 10, multiple headset devices 20, and multiple external image capture devices 25 arranged in the same medical site (treatment room, operating room, etc.), the recording processing unit 333 may create the recording information including the identification information of each of the multiple mobile terminals 10, multiple headset devices 20, and multiple external image capture devices 25 (in other words, create the recording information so that it can be identified which mobile terminal 10, headset device 20, and external image capture device 25 the sound data and image data that are the source of the recording information were collected from), and store the recording information in the recording information storage unit 327.

[0082] 7 on the mobile terminal 10 via the output control unit 192 based on the recording information stored in the recording information storage unit 327. Specifically, the recording processing unit 333 transmits the recording information stored in the recording information storage unit 327 to the mobile terminal 10 via the NW communication unit 31, and the output control unit 192 of the mobile terminal 10 displays a timeline display such as screen G1 shown in FIG. 7 on the display unit 13 based on the recording information. Note that the recording processing unit 333 may display the timeline display of the recording information on a display device other than the mobile terminal 10.

[0083] 7 is a diagram showing an example of display of recorded information on the mobile terminal 10 in this embodiment. As shown in FIG. 7, the mobile terminal 10 displays keywords and image data in chronological order on the display unit 13.

[0084] 7, the image data PH1 is an image of the burned area of ​​the patient captured by the imaging unit 23 of the headset device 20. Here, the recording processing unit 333 and the output control unit 192 may superimpose, for example, a line surrounding the burned area or text (e.g., "depth I," "depth III," etc.) on the image. In other words, the recording processing unit 333 and the output control unit 192 may add and display any object based on a user's instruction (e.g., a line surrounding the burned area, text input by the user's voice or any input device, etc.) at a specified position in the image data displayed on the display unit 13.

[0085] At this time, the position where the object is to be added may be specified by any method. For example, the user may specify the position by moving a cursor on the screen via a pointing device included in the input unit 12. For example, the user may perform a so-called drag operation to draw a line at a position corresponding to the trajectory of the cursor movement or to specify an area where text is to be displayed. Alternatively, if the display unit 13 is a touch screen (i.e., if the input unit 12 is a touch sensor included in the touch screen and input is possible by touch operation on the screen of the display unit 13), the user may draw a line or specify a position where text is to be displayed by tracing the screen.

[0086] When the text is displayed, a so-called "talking and drawing" technique may be used, in which text based on the user's voice is superimposed on the path of moving the cursor or tracing with a finger. For "talking and drawing," the techniques described in, for example, JP 2018-22528 A and JP 2019-71104 A can be used.

[0087] Furthermore, the recording processing unit 333 and the output control unit 192 may automatically suggest, for example, the extent of the burn area, the depth of each burn area, areas to add annotations to, etc., using the machine learning results (i.e., the image recognition model stored in the learning result storage unit 323) by the image recognition processing unit 332. Furthermore, the recording processing unit 333 and the output control unit 192 may mask the patient's face in the image to be output from the perspective of privacy.

[0088] Also, in Figure 7, when it is detected that the user has pressed button BT1, the recording processing unit 333 outputs sound data (audio data) corresponding to the original data among the sound data stored in the collected sound information storage unit 324 to the speaker 15 of the mobile terminal 10 via the output control unit 192.

[0089] 7, the image data PH2 is an image of the display screen of a medical device captured by the external imaging device 25 (an example of an imaging unit). As described above, the image recognition processing unit 332 can extract text data indicating the patient's biometric data from the image data PH2 of the display screen of such a medical device. In this case, the image recognition processing unit 332 associates the display position and type (pulse, blood pressure, etc.) of the biometric data on the screen of each medical device, and extracts text data (keywords), such as pulse, blood pressure, etc., from the image data PH2 based on this information (details will be described later with reference to FIG. 8). The extracted text data is associated with the original image data PH2 and recorded as record information by the recording processing unit 333, and is displayed together with the original image data PH2 in the timeline display, as shown in FIG. 7.

[0090] Furthermore, if the recording processing unit 333 detects an abnormality from the recorded information, it outputs a warning from the display unit 13 or speaker 15 of the mobile terminal 10 via the output control unit 192. For example, rules for detecting abnormalities may be set in advance based on expected abnormality patterns, and the recording processing unit 333 detects the abnormality according to these rules. Specifically, for example, the recording processing unit 333 references the recorded information and determines whether the contents of text data indicating the healthcare professional's voice drug order extracted from the audio data match those of text data indicating the type, amount, and flow rate of the drug actually used extracted from the screen data of the medication device. Then, if the voice drug order does not match the type, amount, and flow rate of the drug actually used, the recording processing unit 333 outputs a warning from the display unit 13 or speaker 15 of the mobile terminal 10 via the output control unit 192. The recording processing unit 333 may also refer to the recorded information and, based on text data indicating the remaining amount of medication, such as an intravenous drip, extracted from the image data, determine whether the remaining amount is appropriate for the instructions given by the medical professional. If the remaining amount is determined to be inappropriate, the recording processing unit 333 may output a warning from the display unit 13 or speaker 15 of the mobile terminal 10 via the output control unit 192. Note that the warning does not have to be output from the display unit 13 or speaker 15 of the mobile terminal 10, but may be issued in the form of a display, voice, or the like from another device installed in the medical field (treatment room, operating room, etc.). The recording processing unit 333 may also detect an abnormality and issue a warning using the results of machine learning, i.e., an anomaly detection model for medical professional treatment and medication stored in the learning result storage unit 323.

[0091] Furthermore, when there is a sudden change in the patient's biometric data such as heart rate or blood pressure, the recording processing unit 333 may automatically cause the collection processing unit 191 to start recording audio and images. Specifically, the recording processing unit 333 determines whether there is a sudden change in the patient's biometric data such as heart rate or blood pressure that exceeds a predetermined reference value, based on text data indicating the patient's biometric data extracted from the screen data of the measurement device, and can cause the collection processing unit 191 to start recording audio and images based on the result of the determination.

[0092] 8 is a diagram showing an example of setting the display position and type of biometric data in this embodiment. In FIG. 8, it is assumed that the numeric data DT1 second from the left on the top line is "pulse" and the numeric data DT2 on the middle line is "blood pressure." After these settings are made, the image recognition processing unit 332 extracts text data indicating the heart rate and blood pressure values ​​from the image data collected by the collection processing unit 191.

[0093] 1 , the support processing unit 334 executes various support processes. When the support processing unit 334 detects a support request from a user regarding a medical procedure, it executes support processing according to the support request and outputs the support processing results to the user. For example, the support request may be a search for information regarding a medical procedure, a suggestion regarding a medical procedure, or a request for help from another user.

[0094] For example, when the support processing unit 334 receives a request from a mobile terminal 10 to search for information on a medical procedure by a user, it searches for information such as terms, treatment, medication, and procedures, sends the search results to the mobile terminal 10, and outputs them as support processing results from the display unit 13 or speaker 15 via the output control unit 192.

[0095] Furthermore, for example, when the support processing unit 334 receives a user's request for a medical procedure proposal from the mobile device 10, the support processing unit 334 generates a medical procedure proposal based on past record information, transmits the generated medical procedure proposal to the mobile device 10, and outputs the medical procedure proposal as a support processing result from the display unit 13 or the speaker 15 via the output control unit 192. For example, the medical procedure proposal is a proposal for an optimal treatment (treatment, medication, etc.) for the patient. For example, the support processing unit 334 searches past record information, extracts record information that is highly similar to the current record information, and proposes a treatment method, the type and dosage of medication to be administered, etc. based on the highly similar record information. The support processing unit 334 may make a proposal for a treatment, medication, etc. using, for example, the machine learning results of the learning processing unit 337 (i.e., the medical procedure proposal model stored in the learning result storage unit 323).

[0096] Furthermore, for example, when the support processing unit 334 receives a rescue request from a user to another user from the mobile terminal 10, it may check the status of the users stored in the user information storage unit 322, extract users who can respond to the rescue request, and arrange for rescue (specifically, a call to a treatment room or a request for advice remotely) for the available users (such as doctors). Note that when arranging rescue, the support processing unit 334 may send information related to a timeline display of recorded information to available users (such as doctors) to share information about the patient and the details of medical procedures performed up to that point.

[0097] The assistance request from the user does not have to be input from the mobile device 10, but may be input from another device. The method for inputting the assistance request from the user is not limited, and may be, for example, voice input, character input, gesture input, etc. The result of the assistance processing by the assistance processing unit 334 does not have to be output to the mobile device 10, but may be output to the user via another device.

[0098] Returning to the description of FIG. 1 , the editing processing unit 335 executes editing processes such as browsing, searching, editing, and processing the record information stored in the record information storage unit 327. Specifically, the editing processing unit 335 edits the record information stored in the record information storage unit 327 and outputs the edited record information to the user. For example, the editing processing unit 335 displays the record information in a timeline format on the display unit 43 of the management terminal 40 (terminal device) via the display control unit 451. The user can perform editing operations, such as modifying keywords, while referring to the display screen. The editing processing unit 335 receives an editing request corresponding to the editing operation from the management terminal 40, edits the record information stored in the record information storage unit 327 in response to the editing request, and outputs the edited record information to the management terminal 40. This allows for easy correction even if there is an error in keyword extraction by the voice recognition processing unit 331 and the image recognition processing unit 332. For convenience in the editing process, the editing processing unit 335 can also display only the recording information that includes the specified keyword on the timeline (search, narrowing down process).

[0099] Furthermore, when record information is created using information aggregated from multiple mobile terminals 10, multiple headset devices 20, and multiple external image capture devices 25, the editing processing unit 335 may allow a user (or administrator) to select keywords to be used in the medical document (or keywords not to be used) via the management terminal 40, and reflect these in the medical document created by the medical document generation unit 336 (described later). For example, the editing processing unit 335 may integrate information by prioritizing information collected by the doctor's mobile terminal 10 over information collected by the nurse's mobile terminal 10.

[0100] The editing processing unit 335 may also generate a summary of the medical procedure using the record information stored in the record information storage unit 327 and display it on the display unit 43 of the management terminal 40 via the display control unit 451, for example. The editing processing unit 335 may also generate a list (archive information) including, for example, the date, attending physician, attending nurse, and symptoms from the record information stored in the record information storage unit 327 and display it on the display unit 43 of the management terminal 40 via the display control unit 451. The generation of the summary and archive information may be performed automatically by the editing processing unit 335 or in response to a user's instruction. When the generation of the summary and archive information is performed automatically, various AI generation tools may be utilized. When the generation of the summary and archive information is performed in response to a user's instruction, the user may specify the content to be reflected in the summary and archive information from the record information and its format. That is, in response to a user's instruction, the editing processing unit 335 may extract only the information deemed necessary from the record information and output it in any format. In this case, the editing processing unit 335 may provide a search function that allows the user to search for desired information from among the recorded information, and the user may use this search function to search for and specify the desired information to be output from among the recorded information. In this case, the user may be able to specify the period of recorded information to be searched (today, the past year, etc.). Note that the terminal from which the editing request is input and the terminal from which the results of the editing process are output are not limited to the management terminal 40 and may be other terminals.

[0101] The medical document generation unit 336 generates medical documents in a preset format based on the record information stored in the record information storage unit 327, either in response to a user request or automatically in accordance with preset instructions. The method by which the user inputs a request for medical document generation is not limited; for example, the request may be input via the mobile terminal 10 or another terminal. Furthermore, when medical documents are generated automatically, the user or an administrator of the medical support system 1 may appropriately set the medical document to be automatically generated. The medical document generation unit 336 generates medical documents from record information based on the medical document format stored in the format storage unit 328. Specifically, the medical document generation unit 336 generates medical documents by referencing the medical document format and filling in each blank field with the corresponding content contained in the record information. For example, a numerical value indicating blood pressure in the record information is automatically transcribed into the blood pressure numerical value input field in the medical document. Here, medical documents include, for example, electronic medical records, medical receipts (medical fee statements), prescriptions, and emergency team activity reports. In addition, various AI generation tools may be used to generate medical documents.

[0102] The medical document generation unit 336 also stores the generated medical documents in the medical document storage unit 329 and transmits them to the hospital management server 50 via the NW communication unit 31. The hospital management server 50 manages electronic medical records, medical receipts (medical fee statements), and prescriptions, and outputs medical receipts (medical fee statements) and prescriptions to patients.

[0103] The learning processing unit 337 executes various types of machine learning processing. Specifically, the learning processing unit 337 performs machine learning using past record information as training data to generate a voice recognition model, an image recognition model, a model for detecting abnormalities in medical treatments and medication by medical professionals, a proposed model for medical procedures, and the like. The learning processing unit 337 stores various models that are the results of the learning processing in the learning result storage unit 323. Note that not only record information but also various types of information related to past medical procedures may be used as training data for machine learning.

[0104] Next, the operation of the medical support system 1 according to this embodiment will be described with reference to the drawings. Fig. 9 is a diagram showing an example of the process of collecting and recording record information in the medical support system 1 according to this embodiment.

[0105] 9 , the mobile terminal 10 of the medical support system 1 first detects sounds and images (step S101). The collection processing unit 191 of the mobile terminal 10 collects sound data and image data from the mobile terminal 10, the headset device 20, and the external image capture device 25.

[0106] Next, the collection processing unit 191 transmits the sound data and image data to the information collection server 30 (step S102). The collection processing unit 191 transmits the sound data and image data via the NW communication unit 11 and requests voice recognition processing and image recognition processing.

[0107] Next, the information collecting server 30 executes a voice recognition process and an image recognition process (step S103). The voice recognition processing unit 331 of the information collecting server 30 converts the received sound data (voice data) into text data. The image recognition processing unit 332 of the information collecting server 30 extracts hands-free operations included in the received image data.

[0108] Next, the information collecting server 30 transmits the recognition information, which is the processing result of the voice recognition processing and the image recognition processing, to the mobile terminal 10 (step S104). The server control unit 33 of the information collecting server 30 transmits the recognition information (text data, hands-free operation, etc.) to the mobile terminal 10 via the NW communication unit 31.

[0109] Next, the collection processing unit 191 determines whether recording is started (step S105). Based on the setting information in the setting information storage unit 181, the collection processing unit 191 determines whether the received recognition information (text data, hands-free operation, etc.) is a start operation. If it is a start operation (step S105: YES), the collection processing unit 191 proceeds to step S106. If it is not a start operation (step S105: NO), the collection processing unit 191 returns the process to step S101.

[0110] In step S106, the collection processing unit 191 starts the collection process and transmits, for example, collected information (raw data) and date and time information acquired from the mobile terminal 10, the headset device 20, and the external image capture device 25 to the information collection server 30. Here, the collected information is, for example, sound data and image data.

[0111] Next, the recording processing unit 333 of the information collecting server 30 stores the collected information (step S107). The recording processing unit 333 associates the sound data with the date and time information and stores the associated data in the collected sound information storage unit 324 (see, for example, FIG. 4), and also associates the image data with the date and time information and stores the associated data in the collected image information storage unit 325 (see, for example, FIG. 5).

[0112] Next, the recording processing unit 333 extracts keywords from the collected information (step S108). For example, the recording processing unit 333 checks whether the text data converted from the sound data by the voice recognition processing unit 331 contains any of the preset keywords stored in the keyword information storage unit 326, thereby extracting the keywords contained in the sound data.

[0113] Next, the recording processing unit 333 determines whether or not a keyword is present (whether or not it has been extracted) (step S109). If the keyword is present (step S109: YES), the recording processing unit 333 proceeds to step S110. If the keyword is not present (step S109: NO), the recording processing unit 333 proceeds to step S113.

[0114] In step S110, the recording processing unit 333 records the keyword-based medical treatment information as record information. Specifically, using the information stored in the collected sound information storage unit 324 and the collected image information storage unit 325 in step S107, the recording processing unit 333 stores, as record information, information associating the keyword extracted in step S109 with date and time information when the keyword was uttered, and information associating the image data collected by the collection processing unit 191 with date and time information when the image data was acquired, in the record information storage unit 327. The recording processing unit 333 stores the record information in the record information storage unit 327 in a format such as that shown in FIG. 6 , for example.

[0115] Next, the recording processing unit 333 transmits the recording information to the mobile terminal 10 (step S111). The recording processing unit 333 transmits the recording information stored in the recording information storage unit 327 to a terminal that outputs the recording information, such as the mobile terminal 10. Note that the recording processing unit 333 may generate timeline display information based on the recording information and transmit the timeline display information to a terminal, such as the mobile terminal 10.

[0116] The output control unit 192 of the mobile terminal 10 displays the recorded information in a timeline (step S112). For example, the output control unit 192 causes the display unit 13 of the mobile terminal 10 to display a timeline based on the recorded information, as shown in FIG.

[0117] Next, in step S113, the recording processing unit 333 determines whether recording has ended. The recording processing unit 333 determines whether a hands-free end operation has been detected using the voice recognition processing unit 331 and the image recognition processing unit 332. If the end operation has been detected and recording has ended (step S113: YES), the recording processing unit 333 proceeds to step S114. On the other hand, if the end operation has not been detected and recording has not ended (step S113: NO), the recording processing unit 333 returns the process to step S107.

[0118] In step S114 , the recording processing unit 333 transmits an end notification to the mobile terminal 10 .

[0119] Next, the collection processing unit 191 determines whether recording has ended (step S115). The collection processing unit 191 determines whether recording has ended (collection has ended), for example, by receiving an end notification. If recording has ended (step S115: YES), the collection processing unit 191 returns the process to step S101. If recording has not ended (step S115: NO), the collection processing unit 191 returns the process to step S106. Note that in step S110, the recording processing unit 333 may create recorded information that further includes text data extracted from the image data by the image recognition processing unit 332.

[0120] Next, the support process of the medical support system 1 according to this embodiment will be described with reference to Fig. 10. Fig. 10 is a diagram showing an example of the support process of the medical support system 1 according to this embodiment. Here, an example in which a user makes a support request by voice will be described.

[0121] 10 , first, the mobile terminal 10 transmits sound data to the information collecting server 30 (step S201). The terminal control unit 19 of the mobile terminal 10 acquires sound data detected by, for example, the microphone 14 of the mobile terminal 10 and the microphone 22 of the headset device 20, and transmits the sound data to the information collecting server 30 via the NW communication unit 11.

[0122] Next, the voice recognition processing unit 331 of the information collecting server 30 executes voice recognition processing (step S202). The voice recognition processing unit 331 converts the sound data into text data.

[0123] Next, the support processing unit 334 determines whether or not there is a support request (step S203). The support processing unit 334 determines whether or not there is a support request, such as an information search, in the text data converted from the audio data. If there is a support request (step S203: YES), the support processing unit 334 proceeds to step S204. If there is no support request (step S203: NO), the support processing unit 334 returns the process to step S202.

[0124] In step S204, the support processing unit 334 executes processing in response to the support request. The processing in response to the support request here includes, for example, searching for information on medical procedures, proposing medical procedures, requesting assistance from other users, etc.

[0125] Next, the support processing unit 334 transmits the support processing result to the mobile device 10 (step S205).

[0126] Next, the output control unit 192 of the mobile terminal 10 displays the support processing result on the display unit 13 or outputs it as voice (step S206). When outputting the support processing result as voice, the output control unit 192 performs voice synthesis processing on the support processing result and outputs it as voice from the speaker 15. Note that the support request from the user does not have to be input from the mobile terminal 10, but may be input from another terminal. Furthermore, the method for inputting the support request from the user is not limited, and for example, voice input, character input, gesture input, etc. may be used. Furthermore, the result of the support processing by the support processing unit 334 does not have to be output to the mobile terminal 10, but may be output to the user via another terminal.

[0127] Next, an abnormality warning process of the medical support system 1 according to this embodiment will be described with reference to Fig. 11. Fig. 11 is a flowchart showing an example of an abnormality warning process of the medical support system 1 according to this embodiment. Note that, although a case where a warning is issued via the mobile terminal 10 will be described here, the warning does not have to be output from the display unit 13 or speaker 15 of the mobile terminal 10, but may be issued in the form of a display, sound, or the like from another device installed in the medical site (treatment room, operating room, etc.).

[0128] 11 , the recording processing unit 333 of the medical support system 1 acquires medical treatment information (step S211). The recording processing unit 333 acquires, for example, medical treatment information (e.g., details of the medical treatment) from the record information stored in the record information storage unit 327.

[0129] Next, the recording processor 333 detects an abnormality in the medical treatment information (step S212). For example, the recording processor 333 references the medical treatment information and detects the abnormality according to a preset rule for detecting an abnormality (for example, whether or not the contents of text data indicating a medication order by voice from a medical professional extracted from the sound data match those of text data indicating the type, amount, and flow rate of medication actually used extracted from the screen data of the medication administration device). Alternatively, for example, the recording processor 333 may detect an abnormality in the medical treatment information using an abnormality detection model that is the learning result stored in the learning result storage unit 323.

[0130] Next, the recording processing unit 333 determines whether or not an abnormality has been detected (step S213). If an abnormality has been detected (step S213: YES), the recording processing unit 333 proceeds to step S214. If an abnormality has not been detected (step S213: NO), the recording processing unit 333 returns to step S211.

[0131] In step S214, the recording processing unit 333 causes the mobile terminal 10 to output a warning based on the abnormality via the output control unit 192. After the process of step S214, the recording processing unit 333 returns the process to step S211.

[0132] Next, the editing and processing process of the record information in the medical support system 1 according to this embodiment will be described with reference to Fig. 12. Fig. 12 is a diagram showing an example of the editing and processing process of the record information in the medical support system 1 according to this embodiment. Note that, although the case where an editing request is input to the management terminal 40 and the result of the editing process is output will be described here, the terminal to which the editing request is input and the terminal to which the result of the editing process is output are not limited to the management terminal 40 and may be other terminals.

[0133] 12, first, the terminal control unit 45 of the management terminal 40 transmits a request to view the recorded information to the information collecting server 30 (step S301). In response to an operation of the input unit 42 by the user (administrator), the terminal control unit 45 transmits the request to view the recorded information to the information collecting server 30 via the NW communication unit 41.

[0134] Next, the editing processing unit 335 of the information collecting server 30 generates a list of record information (step S302). The editing processing unit 335 generates a list of record information, for example, as shown in display screen G2 in Fig. 13. In the example shown in Fig. 13, information indicating an overview of the record information (date, patient name, attending physician, and symptoms) is summarized on one line, and the list of record information is configured by displaying the information indicating the overview of each piece of record information side by side.

[0135] Next, the edit processing unit 335 transmits the list of record information to the management terminal 40 (step S303). The edit processing unit 335 transmits the generated list of record information to the management terminal 40 via the NW communication unit 31.

[0136] Next, the display control unit 451 of the management terminal 40 displays the list of recorded information on the display unit 43 (step S304). The display control unit 451 displays the list of recorded information on the display unit 43, for example, as shown on a display screen G2 in FIG.

[0137] Next, in response to the user (administrator) operating the input unit 42, the terminal control unit 45 transmits a request to view details of the recorded information to the information collecting server 30 (step S305). For example, when the user selects information (any "line") showing an overview of the recorded information on the display screen G2 shown in Fig. 13 by using the cursor, touch operation, or the like, a request to view details of the selected recorded information is transmitted to the information collecting server 30.

[0138] Next, the editing processing unit 335 transmits detailed information of the recorded information corresponding to the request to view the details of the recorded information to the management terminal 40 (step S306).

[0139] Next, the display control unit 451 causes the display unit 43 to display detailed information of the corresponding recording information (step S307). The detailed information of the recording information may be displayed as a timeline display of the recording information, such as a screen G1 shown in FIG.

[0140] Next, the terminal control unit 45 transmits an editing / processing request to the information collection server 30 in response to the user (administrator) operating the input unit 42 (step S308). The user can input instructions such as correcting keywords displayed on the timeline while referring to the screen of the display unit 43.

[0141] Next, the editing processing unit 335 executes processing in accordance with the editing / processing request (step S309). The editing processing unit 335 stores the recorded information after executing the processing in accordance with the editing / processing request in the recorded information storage unit 327, thereby updating the recorded information.

[0142] Next, the editing processing unit 335 transmits the results of the editing and processing, that is, the updated record information, to the management terminal 40 (step S310).

[0143] Next, the display control unit 451 causes the display unit 43 to display the results of the editing and processing, that is, the updated record information (step S311).

[0144] 14 is a diagram showing an example of a summary text screen of the recorded information of the medical support system 1 according to this embodiment. The editing processing unit 335 may edit the recorded information automatically or in response to an instruction input from the user to generate summary text such as the display screen G3 shown in FIG. 14 instead of the timeline display, and display the summary text on the display unit 43 via the display control unit 451. The editing processing unit 335 may also edit the recorded information automatically or in response to an instruction input from the user to generate a list (archive information) including, for example, the date, the doctor in charge, the nurse in charge, and symptoms, and display the list on the display unit 43 of the management terminal 40 via the display control unit 451.

[0145] Next, the medical document generation process of the medical support system 1 according to this embodiment will be described with reference to Fig. 15. Fig. 15 is a flowchart showing an example of the medical document generation process of the medical support system 1 according to this embodiment.

[0146] As shown in FIG. 15 , the medical document generation unit 336 of the information collection server 30 first determines whether a request for creating a medical document has been received (step S321). The request for creating a medical document is input by a user, for example, via the management terminal 40. However, the input means for the request is not limited to this, and any terminal may be used to input the request. If the medical document generation unit 336 has received a request for creating a medical document (step S321: YES), the medical document generation unit 336 proceeds to step S322. If the medical document generation unit 336 has not received a request for creating a medical document (step S321: NO), the medical document generation unit 336 returns to step S321.

[0147] In step S322, the medical document generation unit 336 generates a medical document from the record information in response to the creation request. The medical document generation unit 336 generates a medical document from the record information stored in the record information storage unit 327 based on the medical document format stored in the format storage unit 328.

[0148] Next, the medical document generation unit 336 stores the generated medical document in the medical document storage unit 329 (step S323).

[0149] Next, the medical document generation unit 336 transmits the medical document to the hospital management server 50 (step S324). The hospital management server 50 manages medical documents such as electronic medical records, medical receipts (medical fee statements), and prescriptions, and outputs the medical receipts (medical fee statements) and prescriptions to the patient. After processing step S324, the medical document generation unit 336 ends the processing.

[0150] As described above, the medical support system 1 (support system) according to this embodiment includes a collection processing unit 191, a recording processing unit 333, and an output control unit 192. The collection processing unit 191 collects sound data (sound information) and image data (image information) associated with a medical procedure performed by a user (e.g., a medical professional). The recording processing unit 333 extracts a first keyword from the sound data collected by the collection processing unit 191 and stores, as record information, information associating the first keyword with date and time information on when the first keyword was uttered and information associating the image data collected by the collection processing unit 191 with date and time information on when the image data was acquired in the record information storage unit 327. The output control unit 192 displays, on the display unit 13, a timeline display in which the keywords and image data are arranged in chronological order, based on the record information stored in the record information storage unit 327.

[0151] As a result, the medical support system 1 according to the present embodiment associates keywords and image data with date and time information and stores them as record information in the record information storage unit 327, thereby automatically collecting and preserving more accurate records of medical procedures. Therefore, the medical support system 1 according to the present embodiment further reduces the burden on medical personnel and improves work efficiency and accuracy. Furthermore, the medical support system 1 according to the present embodiment can improve convenience. In particular, in emergency medical settings, treatment is performed in situations where every second counts, making it difficult to manually take quick and accurate notes on the details of medical procedures. Therefore, technology that can automatically record medical procedures has been desired. The medical support system 1 according to the present embodiment automatically and accurately records the details of medical procedures based on the user's voice, the screen of the medical device, and the like. This can improve user convenience, particularly when applied to emergency medical settings.

[0152] The aforementioned Patent Document 1 discloses a technology for recognizing medical procedures performed during surgery as events and creating a display screen that lists the recognized events in chronological order. However, the display screen of Patent Document 1 displays text data indicating the events and links to related image data. For example, if a user wants to view an image, the user must select the link and view the image on a separate screen. In contrast, the present embodiment associates keywords and image data related to the medical procedure with date and time information and stores them as recorded information. A timeline display is also possible, displaying the recorded information in chronological order. In other words, the image data itself is displayed chronologically along with the text data related to the medical procedure, allowing the user to confirm and understand the flow of the medical procedure at a glance. Thus, the medical support system 1 according to the present embodiment can display the content of the medical procedure in a more easily understandable format, thereby further improving user convenience.

[0153] In this embodiment, the collection processing unit 191 starts collecting sound data or image data when a start operation for starting the collection of sound data or image data is detected as a hands-free operation performed by the user without contact. Also, the collection processing unit 191 ends the collection of sound data or image data when a stop operation for ending the collection of sound data or image data is detected as a hands-free operation.

[0154] As a result, the medical support system 1 according to this embodiment allows the user to operate the collection of audio data or image data through non-contact operations, enabling the user to perform medical procedures and record the medical procedures while maintaining hygienic conditions. Therefore, the medical support system 1 according to this embodiment can further reduce the burden on medical staff and improve convenience.

[0155] The medical support system 1 according to this embodiment also includes a microphone 22 (sound collection unit) and an imaging unit 23. The microphone 22 (sound collection unit) collects sounds associated with the medical procedure and outputs the sound data. The imaging unit 23 captures images associated with the medical procedure and outputs the image data. Hands-free operations include at least one of operations based on the user's voice detected based on the sound data output by the microphone 22, operations based on the user's motion detected based on the image data output by the imaging unit 23, and operations based on the user's gaze detected by a headset (headset device 20) equipped with the imaging unit 23.

[0156] As a result, the medical support system 1 according to this embodiment can record medical procedures, etc., through voice operation, motion operation, and gaze operation, thereby further reducing the burden on medical personnel and improving convenience.

[0157] In addition, in this embodiment, the output control unit 192 adds and displays text based on the user's voice at a specified position in the image data displayed on the display unit 13 .

[0158] As a result, the medical support system 1 according to this embodiment can add text to image data using the user's voice, making it possible to record information more clearly and accurately.

[0159] Furthermore, in this embodiment, the recording processing unit 333 extracts a second keyword, which is a keyword included in the image data collected by the collection processing unit 191, and stores information associating the image data, date and time information (e.g., time information) when the image information was acquired, and the second keyword as recorded information in the recorded information storage unit 327. Based on the recorded information, the output control unit 192 displays the image information, the date and time information when the image information was acquired, and the second keyword in association with each other on the timeline display on the display unit 13.

[0160] As a result, the medical support system 1 according to this embodiment associates the second keyword extracted from the image data with the image data and displays it on the display unit 13, thereby enabling records of medical procedures to be kept more clearly and accurately.

[0161] In this embodiment, the recording processing unit 333 stores the recording information created for each period in which sound data and image data are collected in the recording information storage unit 327. The output control unit 192 displays a timeline display on the display unit 13 for each period.

[0162] As a result, in the medical support system 1 according to this embodiment, the record information is recorded for each period, so that the record information can be recorded efficiently. Furthermore, in the medical support system 1 according to this embodiment, the record information can be checked for each period by displaying a timeline, so that the work efficiency can be improved.

[0163] In this embodiment, the first and second keywords include at least one of terms related to the medical procedure, terms related to the tools used in the medical procedure, terms related to medication, and terms related to the patient's biological information. This allows the medical support system 1 according to this embodiment to accurately record medical procedure record information.

[0164] In this embodiment, the users include multiple medical professionals. The record processing unit 333 identifies the speaker who uttered the first keyword from among the multiple medical professionals based on the sound data, and stores, as record information, information in which identification information (e.g., a user ID) identifying the speaker, the first keyword, and date and time information are associated with each other in the record information storage unit 327.

[0165] As a result, the medical support system 1 according to this embodiment can easily and accurately record medical procedure information for multiple medical personnel, thereby further reducing the burden on medical personnel and improving work efficiency and accuracy.

[0166] Furthermore, in this embodiment, the record processing unit 333 outputs a warning when an abnormality is detected in the medical procedure based on the record information stored in the record information storage unit 327. This allows the medical support system 1 according to this embodiment to further improve the efficiency and accuracy of medical procedures.

[0167] The medical support system 1 according to this embodiment also includes a support processing unit 334. The support processing unit 334 executes support processing in response to a support request for medical treatment from a user, and outputs the support processing results to the user.

[0168] As a result, in the medical support system 1 according to this embodiment, the user can receive appropriate support during medical treatment, thereby further improving the efficiency and accuracy of medical treatment.

[0169] Furthermore, in this embodiment, the support processing unit 334 generates a proposal regarding a medical procedure based on past record information, and outputs the generated proposal regarding a medical procedure to the user as a support processing result.

[0170] As a result, the medical support system 1 according to this embodiment allows users to receive appropriate suggestions based on past performance while performing medical procedures, thereby further improving the efficiency and accuracy of medical procedures.

[0171] The medical support system 1 according to this embodiment also includes an edit processing unit 335. The edit processing unit 335 edits the record information stored in the record information storage unit 327, and outputs the edited record information to the user.

[0172] As a result, the medical support system 1 according to this embodiment can check the recorded information at a later date and, for example, appropriately correct any erroneous records, thereby further improving work efficiency and work accuracy.

[0173] The medical support system 1 according to this embodiment also includes a medical document generation unit 336. The medical document generation unit 336 generates medical documents in a preset format (e.g., electronic medical records, medical receipts (medical fee statements), prescriptions, etc.) based on the record information stored in the record information storage unit 327.

[0174] As a result, the medical support system 1 according to this embodiment automatically generates medical documents (e.g., electronic medical records, medical receipts (medical fee statements), prescriptions, etc.) from recorded information, thereby further reducing the burden on medical personnel and further improving work efficiency and accuracy.

[0175] In this embodiment, the display unit that displays the keywords and image data in chronological order is the display unit 13 of the mobile terminal 10 carried by the user. As a result, in the medical support system 1 according to this embodiment, the user can appropriately check the recorded information on the mobile terminal 10 while performing medical procedures.

[0176] The support method according to this embodiment also includes a collection processing step, a recording processing step, and an output control step. In the collection processing step, the collection processing unit 191 collects sound data and image data associated with a medical procedure performed by a user. In the recording processing step, the recording processing unit 333 extracts a first keyword, which is a keyword included in the sound data collected by the collection processing unit 191, and stores, as record information, information associating the first keyword with date and time information when the first keyword was uttered, and information associating the image data collected by the collection processing unit 191 with date and time information when the image information was acquired, in the record information storage unit 327. In the output control step, the output control unit 192 causes the display unit 13 to display a timeline display in which the keywords and image data are arranged in chronological order, based on the record information stored in the record information storage unit 327.

[0177] As a result, the support method according to this embodiment has the same effects as the medical support system 1 described above, and can further reduce the burden on medical staff and improve work efficiency and work accuracy.

[0178] Furthermore, in the above-described embodiment, an example has been described in which the mobile terminal 10 includes the collection processing unit 191 and the output control unit 192. However, the present invention is not limited to this, and the information collecting server 30 (an example of a server device) may also include the collection processing unit 191 and the output control unit 192. That is, the server device (information collecting server 30) according to the present embodiment may include the above-described collection processing unit 191, the recording processing unit 333, and the output control unit 192. As a result, the server device (information collecting server 30) according to the present embodiment can achieve the same effects as the above-described medical support system 1, further reducing the burden on medical personnel and improving work efficiency and work accuracy.

[0179] Second Embodiment Next, a medical support system 1a according to a second embodiment will be described with reference to the drawings.

[0180] In this embodiment, a modified example will be described in which a timeline based on recorded information is displayed on an external display device 55 installed in a medical setting such as an operating room in addition to the display unit 13 of the mobile terminal 10.

[0181] 16 is a functional block diagram showing an example of a medical support system 1a according to the second embodiment. As shown in FIG. 16, the medical support system 1a includes a mobile terminal 10, a headset device 20, an external image capture device 25, an information collection server 30a, a management terminal 40, a hospital management server 50, and an external display device 55.

[0182] 16, the same components as those in the first embodiment shown in FIG. 1 are denoted by the same reference numerals, and the description thereof will be omitted. The medical support system 1a is an example of a support system.

[0183] The external display device 55 is, for example, a large-screen display device that is fixedly installed in an operating room or the like. The external display device 55 is, for example, a liquid crystal display, and is an example of a display unit. Here, the display unit includes the external display device 55 (an example of a display device) installed in a place where a medical procedure is performed (for example, an operating room, etc.). The external display device 55 also includes a display control unit 551.

[0184] The display control unit 551 displays various information on the external display device 55 in response to instructions from the recording processing unit 333a of the information collection server 30a (described later). The display control unit 551, like the display unit 13 of the mobile terminal 10, displays a timeline based on the recorded information. In this embodiment, the timeline display is performed in real time while a medical procedure is being performed, and the timeline display is updated each time a procedure is performed (specifically, each time sound data and image data related to the procedure are collected and the recorded information is updated). This means that new information related to the procedure is added to the timeline as needed. In other words, the recording processing unit 333a, via the display control unit 551, updates the timeline display to the latest information, i.e., displays various information related to the procedure (e.g., patient biometric information, information related to symptoms, or information related to medication) on the external display device 55. The display control unit 551 also displays associated information associated with the recorded information on the same screen as the timeline display. The associated information may be, for example, basic information about the patient, information related to the patient's symptoms, or information related to medication. The display on the external display device 55 will be described in detail later.

[0185] The information collection server 30a is a server device that performs support processing to reduce the burden on medical personnel, such as recording various information (recorded information) associated with medical procedures performed by medical personnel and displaying display information based on the recorded information on the display unit 13 of the mobile terminal 10 and the external display device 55. The information collection server 30a is capable of communicating with the mobile terminal 10, the management terminal 40, the hospital management server 50, and the external display device 55 via the network NW1.

[0186] The information collecting server 30a includes a network communication unit 31, a server storage unit 32, and a server control unit 33a. The server control unit 33a is a functional unit realized by, for example, causing a processor (not shown) including a CPU to execute a program stored in the server storage unit 32. The server control unit 33a comprehensively controls the information collecting server 30a and executes various processes. The server control unit 33a includes a voice recognition processing unit 331, an image recognition processing unit 332, a recording processing unit 333a, a support processing unit 334, an editing processing unit 335, a medical document generation unit 336, and a learning processing unit 337.

[0187] The recording processing unit 333a has the same functions as the recording processing unit 333 of the first embodiment described above. Furthermore, the recording processing unit 333a causes the external display device 55 to display a timeline display, in which keywords and image data are arranged in chronological order, via the display control unit 551 of the external display device 55. At this time, the recording processing unit 333a causes the external display device 55 to display the timeline display, updating it to the latest information based on the recorded information that is updated as needed, via the display control unit 551. Furthermore, the recording processing unit 333a causes the display control unit 551 to display, along with the timeline display of the recorded information, accompanying information of the recorded information on the same screen. Examples of the accompanying information include basic information about the patient (such as the patient's name, date of birth, age, height, weight, blood type, symptoms, and biological information (vital information)), information about the patient's symptoms, and information about medication. The recording processing unit 333a also updates and displays the accompanying information via the display control unit 551 in accordance with updates to the timeline display.

[0188] Here, a display example of the external display device 55 in this embodiment will be described with reference to Figures 17 and 18. Figure 17 is a diagram showing an example of the operation of the medical support system 1a according to this embodiment.

[0189] 17 , in this embodiment, the display control unit 551 displays a timeline of recorded information in an area SG1 (first area) on the screen of the external display device 55, and also displays basic information of the patient in an area SG2 (second area). Here, the basic information is an example of accompanying information, such as the patient's name, height, weight, blood type, symptoms, and biological information (vital information). The biological information (vital information) includes, for example, heart rate, blood pressure, body temperature, respiratory rate, etc.

[0190] In addition, when the timeline display is updated in area SG1 (first area) and additional basic information about the patient is displayed, the recording processing unit 333a causes the display control unit 551 to display the newly displayed basic information so that the value of the basic information is transcribed as the latest basic information in area SG2 (second area).

[0191] That is, in this embodiment, the recording processing unit 333a causes the external display device 55 to display a timeline display in which keywords and image data are arranged in chronological order, via the display control unit 551 of the external display device 55, and also causes the external display device 55 to display the patient's latest biometric information (heart rate, blood pressure, body temperature, respiratory rate, etc.) based on the recorded information.

[0192] 18 is a diagram showing another example of the operation of the medical support system 1a according to this embodiment. Here, a modified example of the information displayed on the external display device 55 will be described.

[0193] 18 , in this modification, the display control unit 551 displays a timeline of the recorded information in an area SG1 (first area) on the screen of the external display device 55, and also displays information about the patient's symptoms in an area SG3 (third area). Here, the information about the patient's symptoms is an example of accompanying information, and includes, for example, the name of the symptom and an image PH3 of the affected area.

[0194] Furthermore, the external display device 55 displays information about medication extracted from the recorded information in an area SG4 (fourth area). Here, the information about medication is an example of accompanying information, and includes, for example, the name and dosage of the administered medication, and an image PH4 of the label of the administered medication pack.

[0195] In this manner, in this modified example, the recording processing unit 333a causes the external display device 55 to display a timeline display in which keywords and image data are arranged in chronological order via the display control unit 551 of the external display device 55, and also causes the external display device 55 to display information regarding the patient's symptoms and information regarding medication based on the recorded information.

[0196] At that time, when the timeline display is updated in area SG1 (first area) and information about the patient's symptoms is additionally displayed, the recording processing unit 333a causes the display control unit 551 to display the newly displayed information so that it is transcribed into area SG3 (third area) as information about the latest patient's symptoms. Also, when the timeline display is updated in area SG1 (first area) and information about medication is additionally displayed, the recording processing unit 333a causes the display control unit 551 to display the newly displayed information so that it is transcribed into area SG4 (fourth area) as information about the latest medication.

[0197] The type and content of the associated information to be displayed on the same screen as the timeline display may be arbitrarily set by the user. For example, by setting items that the user wants to confirm during a medical procedure as associated information, the user can perform the procedure while checking the record information (timeline display) and the associated information, which are updated as needed. Furthermore, if the user wants to extract and separately save some information from the record information, setting the desired information as associated information automatically extracts the desired information, thereby eliminating the need for the user to manually extract the desired information from the record information. For example, information contained in a medical document may be set as the associated information. In this case, when the associated information is displayed on the same screen as the timeline display, the display of the associated information may be controlled so that the medical document format specified by the user is displayed, and the corresponding information from the timeline display is automatically transcribed into blank fields on the medical document. This allows the user to check the content of the medical document that will ultimately be created by the medical document generation unit 336 based on the record information in advance on-site.

[0198] As described above, in this embodiment, the display unit to which the record information is output is the external display device 55 (display device) installed at the location where the medical procedure is performed. The record processing unit 333a causes the external display device 55 to display, via the display control unit 551 (an example of an output control unit), a timeline display in which keywords and image information are arranged in chronological order, while updating the timeline display as needed (updating to the latest information). Furthermore, the record processing unit 333a causes, via the display control unit 551, associated information (e.g., the patient's latest basic information (e.g., biological information), latest symptom information, or medication information) to be displayed on the same screen as the timeline display of the record information, while updating the associated information as needed, based on the updated record information.

[0199] As a result, the medical support system 1a according to this embodiment allows a user to perform treatment while checking a timeline display of record information that is updated as needed and the accompanying information, thereby improving user convenience. Furthermore, by automatically transcribing a portion of the record information to display the accompanying information, the user can avoid having to manually transcribe the information. Thus, this embodiment further reduces the burden on medical professionals and improves work efficiency and accuracy.

[0200] The display according to this embodiment does not necessarily have to be displayed on the external display device 55, but may be displayed on another display device. For example, displays such as those shown in Figures 17 and 18 may be displayed on the display unit 13 of the mobile terminal 10.

[0201] Furthermore, the information displayed as the associated information may be editable by the editing processing unit 335 of the information collecting server 30a, similar to the record information in the first embodiment. This allows for accurate correction when an error is found in the information extracted from the record information as the associated information. Note that when the edit processing unit 335 edits the record information, the corresponding description of the associated information may be automatically corrected. In other words, editing (changing) the record information may be automatically reflected in the associated information. This allows for correction of both errors in the record information and the associated information extracted from the record information with a single editing process, thereby improving operational efficiency.

[0202] In the first and second embodiments described above, the timeline display as shown in FIG. 7 may be shared in real time over any network inside or outside the hospital. In this case, an external specialist (e.g., an external doctor from another department) may be able to add comments to the shared record information (particularly image data). Also, comments on AI diagnosis may be added.

[0203] The timeline display and the added comments may be displayed on a single screen. In this case, the display control unit 551 (output control unit 192) may change the font and color of the comments from the specialist and the AI ​​diagnosis to make them easier to understand.

[0204] As a result, the medical support system 1a according to this embodiment can make more effective use of recorded information by utilizing external specialists and AI diagnosis, and can improve operational accuracy (accuracy of diagnosis and treatment).

[0205] 7, the display control unit 551 (192) may change the background color of time periods in which there are sudden fluctuations in the data. The recording processing unit 333a (333) may automatically record the start and end times of sudden fluctuations. Here, a "sudden fluctuation" may be, for example, a sudden fluctuation in biological data (heart rate, blood pressure).

[0206] As a result, the medical support system 1a(1) according to this embodiment can clearly display the recorded portions that require attention and in which abnormalities may have occurred, so that when checking the recorded information at a later date, the desired recorded information can be quickly located.

[0207] Third Embodiment Next, a medical support system 1b according to a third embodiment will be described with reference to the drawings.

[0208] This embodiment relates to a method for more effectively utilizing the record information recorded in the first and second embodiments. The purpose of this embodiment is to effectively utilize medical treatment information (e.g., date and time information, information about the patient's symptoms, information about the treatment content, information about the time required for the treatment, and progress information showing the progress after the treatment) collected by a medical institution. As an example, the following describes a case where the medical treatment information is the record information recorded in the first and second embodiments. However, the medical treatment information handled in this embodiment is not limited to this example and may be any information showing the content of the medical treatment. Figure 19 is a functional block diagram showing an example of a medical support system 1b according to the third embodiment.

[0209] 19 , medical support system 1b includes a mobile terminal 10, a headset device 20, an external image capture device 25, an information collection server 30, a management terminal 40, a hospital management server 50, and an information support server 60. In this embodiment, mobile terminal 10, external image capture device 25, information collection server 30, management terminal 40, hospital management server 50, and information support server 60 are connected to network NW1 and can communicate with each other. Note that medical support system 1b is an example of a support system.

[0210] 19, the same components as those in the first embodiment shown in Fig. 1 are denoted by the same reference numerals, and the description thereof will be omitted. For the sake of explanation, Fig. 19 shows only the recorded information storage unit 327 of the functions of the information collecting server 30 shown in Fig. 1.

[0211] The information support server 60 is a server device that executes various processes for more effectively utilizing the recorded information stored in the recorded information storage unit 327 of the information collecting server 30. The information support server 60 includes a network communication unit 61, a server storage unit 62, and a server control unit 63.

[0212] The NW communication unit 61 is a communication device that can be connected to the network NW1 via, for example, a wireless LAN or a wired LAN. The NW communication unit 61 performs data communication between, for example, the mobile terminal 10, the management terminal 40, and the information collection server 30 via the network NW1. The NW communication unit 61 receives, for example, record information from the information collection server 30. The NW communication unit 61 also receives various requests from the management terminal 40 to make effective use of the record information.

[0213] The server storage unit 62 is a storage unit realized by a memory (not shown) provided in the information support server 60 , and stores various information used by the information support server 60 .

[0214] The server memory unit 62 includes an aggregate information memory unit 621, a learning data memory unit 622, a learning result memory unit 623, a regional information memory unit 624, a prediction result memory unit 625, a personnel allocation information memory unit 626, a labor management information memory unit 627, and an externally provided information memory unit 628.

[0215] The aggregated information storage unit 621 stores the aggregation results obtained by the aggregation processing unit 631 (described later) aggregating the record information stored in the record information storage unit 327 of the information collection server 30. The aggregated information storage unit 621 stores, as aggregation results, statistical data on the number of patients and symptoms for each period (e.g., month and day), statistical data on patient information (age, gender) and the time required for treatment, statistical data on symptoms and the time required for treatment, statistical data on the content of medical procedures and the time required for treatment for each medical professional, and statistical data showing the relationship between the treatment performed, the type and amount of medication administered, the tools used, and post-treatment progress information for each symptom. When storing the time required for treatment, the aggregated information storage unit 621 may store the time required for treatment in emergency medical care separately from the time required for treatment in other regular medical care. The aggregated information storage unit 621 may also store statistical data on the time required from the time an emergency transport inquiry is made until the patient is transported.

[0216] That is, the tabulation results include statistical data on the number of patients and symptoms for each period, statistical data on symptoms and the time required for medical treatment, and statistical data on patient information and the time required for medical treatment.

[0217] The learning data storage unit 622 stores various types of learning data for executing machine learning processing. The learning data storage unit 622 stores, for example, learning data processed to generate a target learning model based on recorded information and aggregation results.

[0218] The learning result storage unit 623 stores the learning results obtained by the learning processing unit 632 (described later) executing machine learning processing using the learning data stored in the learning data storage unit 622. The learning result storage unit 623 stores various learning models, such as a prediction model that predicts busy periods at medical institutions, a prediction model that predicts the time required for treatment for each symptom, and an estimation model that estimates treatment candidates, medication candidates, tool candidates, or treatment timing (including treatment procedures) for each symptom.

[0219] The area information storage unit 624 stores area information including at least one of information about the weather in the area, information about traffic volume in the area, information about events held in the area, information about the population of the area, information about schools in the area, information about industries in the area, and information about other medical institutions located in the area. The area information stored in the area information storage unit 624 is obtained from an external server via the network NW1. The weather information includes weather, temperature, humidity, wind speed, etc.

[0220] The prediction result storage unit 625 stores the prediction results predicted by the prediction processing unit 633 (described later). The personnel allocation information storage unit 626 stores the personnel allocation information generated by the personnel allocation generation unit 634 (described later).

[0221] The labor management information storage unit 627 stores various information for managing the work of medical professionals who are users. For example, the labor management information storage unit 627 stores labor evaluation information and working hour management information for each medical professional.

[0222] The externally provided information storage unit 628 stores externally provided information generated by the externally provided processing unit 637 (described later) based on the recorded information. The externally provided information is information provided from the medical support system 1b to external institutions. The externally provided information stored in the externally provided information storage unit 628 includes, for example, information that can be provided to medical associations or other hospitals and used as data for academic conference presentations on cases, information that can be provided to government-related organizations such as the Ministry of Health, Labor and Welfare and used to calculate medical fees determined based on the difficulty of treatment, and information that can be provided to pharmaceutical companies or medical device manufacturers and used as research and development data. Information that can be used as research and development data includes, for example, information such as the type and amount of medication used in treatment and information on the progress after treatment.

[0223] The server control unit 63 is a functional unit realized by, for example, causing a processor (not shown) including a CPU to execute a program stored in the server storage unit 62. The server control unit 63 comprehensively controls the information support server 60 and executes various processes. The server control unit 63 includes a tallying unit 631, a learning unit 632, a prediction unit 633, a personnel allocation generation unit 634, a labor management unit 635, a proposal unit 636, and an external provision unit 637.

[0224] The tallying processor 631 tally the record information recorded based on the information collected during the medical procedure. Here, the record information is information stored in the record information storage unit 327 of the information collection server 30, and is information including, for example, patient information indicating the patient, medical worker information indicating the medical worker who performed the medical procedure, and medical procedure information indicating the content of the medical procedure associated with date and time information.

[0225] For example, based on record information (information indicating the content of medical procedures at medical institutions), the aggregation processing unit 631 aggregates statistical data on the number of patients and symptoms for each period (month and day), statistical data on patient information (age, gender) and the time required for treatment, statistical data on symptoms and the time required for treatment, and statistical data indicating the relationship between the treatment performed, the type and amount of medication administered, the tools used, and post-treatment progress information for each symptom. When aggregating the time required for treatment, the aggregation information storage unit 621 may aggregate the time required for treatment in emergency medical care separately from the time required for treatment in other ordinary medical care. This is because emergency medical care often targets patients in urgent need, and therefore information on the time required for treatment may be particularly important. The aggregation information storage unit 621 may also aggregate statistical data on the time required from the time an emergency transport inquiry is received until the patient is transported.

[0226] The tallying processor 631 also tally the details of medical procedures and the time required for the medical procedures for each medical worker based on the recorded information, for example, to generate statistical data regarding the work of the medical workers.

[0227] The tallying processor 631 may analyze the man-hours of each medical worker based on the collection time of the record information, for example. The tallying processor 631 may also tally the procedure and timing of treatment for each of a plurality of hospitals and a plurality of doctors, for example.

[0228] The calculation processing unit 631 may also calculate the difficulty of medical treatment based on the diagnosis result (disease name, etc.) and the collection time of the recorded information. The calculation processing unit 631 may also perform calculations by associating, for example, the medical device (tool) with the collection time of the recorded information (required time) or the collected information (treatment effect: for example, improvement in blood data, changes in vital signs (blood pressure, pulse rate, consciousness, respiration, body temperature, oxygen saturation, etc.)).

[0229] The counting processing unit 631 may also determine a correlation between the area information and a specific symptom by counting the recorded information in association with the area information stored in the area information storage unit 624. Possible correlations between the area information and a specific symptom include, for example, "when the temperature is below AA degrees and the humidity is below BB%, the number of influenza patients increases by cc% compared to normal," "when the temperature is above DD degrees and the humidity is above EE%, the number of heatstroke patients increases by FF% compared to normal," and "when traffic volume is above GG% compared to normal, the number of patients due to traffic accidents increases by hh%." The counting processing unit 631 stores the counting results in the counting information storage unit 621.

[0230] The learning processing unit 632 generates learning data corresponding to the target of machine learning based on the recorded information or the aggregation results, and stores the data in the learning data storage unit 622. The learning processing unit 632 also generates various learning models using the learning data stored in the learning data storage unit 622. Through machine learning processing, the learning processing unit 632 generates, for example, a prediction model that predicts busy periods at medical institutions, a prediction model that predicts the time required for treatment for each symptom, and an estimation model that estimates treatment candidates, medication candidates, tool candidates, or treatment timing (including treatment procedures) for each symptom.

[0231] When generating these models, the learning processing unit 632 may perform machine learning that further includes patient information (age, gender, etc.). Including patient information can improve the prediction (estimation) accuracy of the model. The learning processing unit 632 stores the learning results of the machine learning process, such as a prediction model, in the learning result storage unit 623.

[0232] The prediction processing unit 633 predicts the busy season of the medical institution based on the tallying results obtained by the tallying processing unit 631. For example, the tallying processing unit 631 tally the number of patients for each symptom based on the recorded information, and the prediction processing unit 633 predicts the busy season for each medical department based on the tallying number of patients for each symptom.

[0233] Furthermore, the prediction processing unit 633 predicts the busy season of the medical institution, for example, based on the tallying results stored in the tallying information storage unit 621 and the regional information stored in the regional information storage unit 624. Specifically, the prediction processing unit 633 predicts an increase or decrease in the number of patients with a specific symptom according to the current regional information, based on the correlation between the regional information obtained by the tallying processing unit 631 and the specific symptom.

[0234] Furthermore, for example, the prediction processing unit 633 predicts busy periods based on information about events held in a region included in the region information, in accordance with the number of people in the region during the event period (for example, the prediction processing unit 633 predicts that there is a possibility of an increase in heatstroke patients if a crowded event is held in the summer). Furthermore, for example, the prediction processing unit 633 predicts busy periods based on information about the region's industries included in the region information (for example, the prediction processing unit 633 predicts that in a region with many factories, there will be an increase in patients visiting medical institutions due to injuries caused by accidents in the factories).

[0235] Furthermore, for example, the prediction processing unit 633 predicts busy periods based on information about the population of the region included in the region information (for example, the prediction processing unit 633 predicts the number of patients with symptoms according to gender, age, etc., taking into account the male-to-female ratio and age structure of the population of the region (there is a possibility of an increase in patients with diseases specific to men / women, an increase in patients with diseases specific to children / elderly people, etc.)). Furthermore, for example, the prediction processing unit 633 predicts busy periods based on information about schools in the region included in the region information, for example, according to the number of elementary schools, junior high schools, high schools, and universities (for example, the prediction processing unit 633 predicts that there is a possibility of an increase in patients with diseases specific to children in a region with a large number of elementary schools, etc.).

[0236] Furthermore, for example, the prediction processing unit 633 predicts busy periods based on information about other medical institutions located in the area included in the area information (for example, the prediction processing unit 633 predicts the number of patients with a specific symptom who will visit its own medical institution, taking into consideration the medical departments and scales of its own medical institution and the other medical institutions). The prediction processing unit 633 may also predict busy periods for the medical institution based on a prediction model stored in the learning result storage unit 623. The prediction model here is a prediction model that predicts busy periods for the medical institution.

[0237] Furthermore, the prediction processing unit 633 may predict the required time for treatment for each symptom based on statistical data on the symptoms and the time taken for treatment compiled by the compilation processing unit 631. For this prediction, a prediction model for predicting the required time for treatment for each symptom may be used, which is the result of machine learning performed by the learning processing unit 632 using the symptoms or diagnosis results (such as disease name) included in the record information and the required time for treatment (based on the collection time of the record information) as training data.

[0238] The prediction processing unit 633 may use the information about the treatment time for each symptom predicted in this way to predict the time when an inquiry is made about whether or not emergency transport is possible, and predict the time when emergency transport can be accepted and the number of patients that can be accepted. Specifically, information about the status of medical personnel at the medical institution at the time of the emergency transport (information indicating the working status, such as how many medical personnel from which departments are working, the current work status of each person (whether they are treating a patient or free), and when they will be free if they are treating a patient) and information about emergency transport (basic information of the patient involved in emergency transport (such as the patient's name, height, weight, blood type, symptoms, and biological information (vital information))) are input to the prediction processing unit 633 via the NW communication unit 61. The prediction processing unit 633 calculates the time when emergency transport can be accepted and the number of patients that can be accepted, for example, when an inquiry is made about whether emergency transport can be accepted and the number of patients that can be accepted. Specifically, information about the status of medical personnel at the medical institution at the time of the emergency transport (information indicating the working status, such as how many medical personnel from which departments are working, the current work status of each person (whether they are treating a patient or free), and when they will be free if they are treating a patient) and information about emergency transport (basic information of the patient involved in emergency transport (such as the patient's name, height, weight, blood type, symptoms, and biological information (vital information))) are input. Based on the statistical data (strictly speaking, the time required for treatment for each symptom predicted therefrom), information on the status of medical personnel, and information on emergency transport (particularly, information on the symptoms of patients involved in emergency transport), the time available for admission to the emergency transport and the number of patients that can be admitted may be predicted, taking into consideration the status of medical personnel who can respond to the symptoms of patients involved in the emergency transport and the time required (predicted value) for treating the symptoms. In this case, the prediction process is performed using statistical data on the time required for treatment in emergency medical care compiled by the compilation processing unit 631, which makes it possible to improve the prediction accuracy.

[0239] The prediction processing unit 633 transmits the predicted results of the time when emergency transport can be accepted and the number of patients that can be accepted via the NW communication unit 61 to any terminal that the user can access, such as the mobile terminal 10, and outputs the results from that terminal to a medical professional. The medical professional can check the output information and communicate a response to the inquiry about whether emergency transport can be accepted. Note that at this time, the final response about whether or not to accept the patient may be made taking into account the time it takes for the patient to be transported, using statistical data compiled by the compilation processing unit 631 on the time it takes from the inquiry about emergency transport to the patient being transported.

[0240] Furthermore, the prediction processing unit 633 may predict the optimal treatment for each symptom (optimal treatment, optimal amount of drug administered, optimal tool used, optimal treatment timing (including treatment procedure), etc.) based on statistical data indicating the relationship between the treatment performed, the type and amount of drug administered, the tool used, and post-treatment progress information for each symptom, compiled by the compilation processing unit 631. For this prediction, an estimation model may be used that estimates candidate treatments, candidate medications, candidate tools used, or candidate treatment timing for each symptom, which is the result of machine learning performed by the learning processing unit 632 using the symptoms or diagnosis results (such as disease name) included in the record information, the treatment performed, the type and amount of drug administered, the tool used, and post-treatment progress information as training data.

[0241] In addition, in the prediction processing using the results of the machine learning described above, the prediction processing unit 633 may use the learning results obtained by the learning processing unit 632 through machine learning, including patient information (age, gender, etc.).

[0242] The prediction processing unit 633 stores the various prediction results described above (busy periods at medical institutions, time required for treatment for each symptom, acceptable times and number of patients for emergency transport, optimal treatment for each symptom (optimal treatment, optimal amount of medication to be used, optimal tools to be used, optimal timing of treatment, etc.) in the prediction result storage unit 625.

[0243] The staffing generation unit 634 generates a staffing plan for medical personnel capable of handling the busy periods of the medical institution based on the busy periods of the medical institution predicted by the prediction processing unit 633. For example, the staffing generation unit 634 generates a staffing plan for medical personnel in a department corresponding to the symptom based on the increase or decrease in the number of patients for each symptom predicted by the prediction processing unit 633 so as to handle the number of patients. For example, a user can send a request for generating a staffing plan (staffing request) from the input unit 42 of the management terminal 40 to the staffing generation unit 634 of the information support server 60. Note that the staffing request by the user may be input from any terminal. Upon receiving the staffing request, the staffing generation unit 634 executes the staffing generation process described above.

[0244] The staffing generation unit 634 stores the generated staffing results (staffing plan) in the staffing information storage unit 626. The staffing generation unit 634 also outputs the generated staffing results to the requestor (the terminal from which the staffing request was input). For example, the staffing generation unit 634 transmits the staffing results to the requestor, the management terminal 40, via the NW communication unit 61.

[0245] The labor management processing unit 635 executes labor management processing for medical personnel. For example, labor management includes evaluation of the work content of medical personnel and management of working hours. The labor management processing unit 635 executes labor management processing based on, for example, statistical data on the work of medical personnel compiled by the compilation processing unit 631. Specifically, the evaluation of work content takes into account information contained in the record information, such as the number of patients treated, the symptoms of the patients treated, the difficulty of the treatment, the time required for the treatment, and the progress after the treatment. Furthermore, the management of working hours takes into account information contained in the record information, such as the time required for the treatment.

[0246] The labor management processing unit 635 stores the results of labor management in the labor management information storage unit 627. The information related to labor management stored in the labor management information storage unit 627 may be output from any terminal (e.g., mobile terminal 10) connected to the network NW1 via the NW communication unit 61. However, from the perspective of information confidentiality, it is desirable that only authorized personnel can view the information related to labor management. For example, access rights are set so that only personnel in charge of labor management and superiors of personnel subject to labor management among users can view the information related to labor management stored in the labor management information storage unit 627.

[0247] The proposal processing unit 636 proposes treatments that can be performed on a new patient based on the results of predictions made by the prediction processing unit 633 based on past record information. Specifically, for example, the proposal processing unit 636 proposes treatment options, medication options, tool options, or treatment timing options for a new patient based on the optimal treatments for each symptom (optimal treatment, optimal drug dosage, optimal tool options, and optimal treatment timing (including treatment procedures)) predicted by the prediction processing unit 633. This enables the proposal processing unit 636 to propose appropriate treatment options when a patient with symptoms similar to those included in past record information is brought in by ambulance.

[0248] Furthermore, the proposal processing unit 636 may propose treatment details for similar past cases in real time while a medical professional is treating a patient. For example, a user can send a proposal request (proposal request) to the proposal processing unit 636 of the information support server 60 from the input unit 42 of the management terminal 40. The proposal request by the user may be input from any terminal. Upon receiving the proposal request, the proposal processing unit 636 executes the proposal process.

[0249] Furthermore, when proposing a treatment that can be performed on a new patient, the proposal processing unit 636 may also propose candidates for medical professionals who can perform the treatment. For example, the proposal processing unit 636 may search for a doctor who can perform the treatment and propose to call the doctor.

[0250] The proposal processing unit 636 may make these proposals using the learning results stored in the learning result storage unit 623. The proposal processing unit 636 may make various proposals based on, for example, the results of various prediction processes using the learning results by the prediction processing unit 633. Specifically, the proposal processing unit 636 may propose, for example, optimal treatments for each symptom (optimal treatments, optimal drug dosages, optimal tools to be used, optimal treatment timing (including treatment procedures), etc.) predicted by the prediction processing unit 633 based on an estimation model that estimates candidate treatments, candidate medications, candidate tools to be used, or treatment timing (including treatment procedures) for each symptom.

[0251] Furthermore, the proposal processing unit 636 transmits these proposals via the NW communication unit 61 to any terminal that the user can check, such as the mobile terminal 10, and causes the terminal to output the proposals to the medical professional. Furthermore, the proposal processing unit 636 may refer to the recorded information and cause the mobile terminal 10 to output a warning if the content of the actual treatment currently being performed (e.g., the content of the treatment, timing, type of tool, medication content, etc.) deviates from the content predicted by the prediction processing unit 633 (i.e., the content proposed by the proposal processing unit 636).

[0252] The presence or absence of a discrepancy may be determined according to a predetermined rule. The rule can be arbitrarily set by the user. Examples of the rule may include when the proposed treatment differs from the actual treatment, when the type of drug proposed differs from the type of drug actually being used, when the proposed drug dosage differs from the actual drug dosage by more than a predetermined threshold, and when the timing (including the procedure) of the proposed treatment / medication differs from the actual timing (including the procedure). The warning may be in any form, such as a display or audio, as long as it is recognizable to the user. The warning may also be output from any device other than the mobile terminal 10 that the user can view.

[0253] The external provision processing unit 637 generates information to be provided to an external organization by executing a preset processing process on the recorded information or the aggregated results of the recorded information stored in the aggregate information storage unit 621. Here, the provided information to be provided to an external organization (external provided information) is, for example, information that can be provided to a medical association or another hospital, etc., and used as data for academic conference presentations on cases, information that can be provided to a government-related organization such as the Ministry of Health, Labor and Welfare, and used to calculate medical fees determined according to the difficulty of treatment, information that can be provided to a pharmaceutical company or a medical device manufacturer, and used as research and development data, etc.

[0254] In generating the externally provided information, from the perspective of protecting patient privacy, processing may be performed as appropriate to prevent identification of individuals in accordance with relevant laws, regulations, and guidelines, such as anonymizing or pseudonymizing patient names and masking faces in image data. For example, a user can send a request to generate externally provided information (a request for externally provided information) from the input unit 42 of the management terminal 40 to the externally provided information processing unit 637 of the information support server 60. The user's request for externally provided information may be input from any terminal. The request for externally provided information includes information about the recipient of the externally provided information. Upon receiving the request for externally provided information, the externally provided information processing unit 637 performs processing in a format appropriate to the recipient of the externally provided information, thereby generating the externally provided information.

[0255] The external provision processing unit 637 stores the generated provision information (external provision information) to be provided to an external organization in the external provision information storage unit 628. The information stored in the external provision information storage unit 628 is transmitted to an external organization (not shown) designated by the user via the NW communication unit 61, for example.

[0256] Next, the operation of the medical support system 1b according to this embodiment will be described with reference to the drawings. Fig. 20 is a flowchart showing an example of the learning process of the medical support system 1b according to this embodiment.

[0257] 20 , the tallying unit 631 of the information support server 60 acquires the recorded information and executes tallying processing (step S401). The tallying unit 631 acquires the recorded information from the information collecting server 30 via the NW communication unit 61 and executes various tallying processes according to the purpose. The tallying unit 631 stores the tallying results in the tally information storage unit 621.

[0258] Next, the learning processing unit 632 of the information assistance server 60 generates learning data (step S402). The learning processing unit 632 generates learning data using the aggregation results by the aggregation processing unit 631, and stores the learning data in the learning data storage unit 622.

[0259] Next, the learning processing unit 632 executes a learning process based on the learning data (step S403). The learning processing unit 632 executes a machine learning process using the learning data stored in the learning data storage unit 622.

[0260] Next, the learning processing unit 632 stores the learning result in the learning result storage unit 623 (step S404). After the process of step S404, the learning processing unit 632 ends the learning process.

[0261] Next, the personnel allocation process of the medical support system 1b according to this embodiment will be described with reference to Fig. 21. Fig. 21 is a flowchart showing an example of the personnel allocation process of the medical support system 1b according to this embodiment.

[0262] 21 , the server control unit 63 of the information support server 60 determines whether a staffing request has been received (step S501). The server control unit 63 determines whether a staffing request has been received from, for example, the management terminal 40 via the NW communication unit 61. If the server control unit 63 has received a staffing request (step S501: YES), the server control unit 63 proceeds to step S502. If the server control unit 63 has not received a staffing request (step S501: NO), the server control unit 63 returns the process to step S501.

[0263] In step S502, the prediction processing unit 633 of the server control unit 63 predicts the busy season of the medical institution. For example, the prediction processing unit 633 predicts the busy season of the medical institution using statistical data of past record information compiled by the compilation processing unit 631. Alternatively, the prediction processing unit 633 may predict the busy season of the medical institution using the learning results (prediction model) stored in the learning result storage unit 623. The prediction processing unit 633 stores the prediction results in the prediction result storage unit 625.

[0264] Next, the staffing generation unit 634 of the server control unit 63 generates a staffing plan based on the prediction result (step S503). The staffing generation unit 634 generates a staffing plan so as to, for example, increase the number of medical personnel during the predicted busy period. The staffing generation unit 634 stores the generated staffing plan in the staffing information storage unit 626.

[0265] Next, the personnel assignment generation unit 634 outputs the personnel assignment plan to the request source (step S504). The personnel assignment generation unit 634 transmits the personnel assignment plan to, for example, the request source, the management terminal 40, via the NW communication unit 61. After processing step S504, the personnel assignment generation unit 634 ends the personnel assignment process.

[0266] Next, the labor management process of the medical support system 1b according to this embodiment will be described with reference to Fig. 22. Fig. 22 is a flowchart showing an example of the labor management process of the medical support system 1b according to this embodiment.

[0267] 22 , the tallying processor 631 of the information support server 60 acquires the record information and performs tallying processing for each medical worker (step S511). The tallying processor 631 acquires the record information from the information collection server 30 via the NW communication unit 61 and performs tallying processing for each medical worker. Specifically, in the tallying processing for each medical worker, the content of the medical procedure and the period required for the medical procedure are tallied based on the record information, and statistical data on the work of the medical worker is generated. The tallying processor 631 stores the tallying results in the tally information storage unit 621.

[0268] Next, the labor management processing unit 635 of the information support server 60 executes labor management processing for each medical worker based on the aggregation result (step S512). For example, the labor management processing includes evaluation of the work content of the medical worker and management of working hours.

[0269] Next, the labor management processing unit 635 stores information about the results of labor management in the labor management information storage unit 627 (step S513). After the processing of step S513, the labor management processing unit 635 ends the labor management processing.

[0270] Next, the proposal processing of the medical support system 1b according to this embodiment will be described with reference to FIG. 23 . FIG. 23 is a flowchart showing an example of the proposal processing of the medical support system 1b according to this embodiment. Although not shown, the medical support system 1b may execute the series of processes shown in FIG. 23 by receiving a proposal request (proposal request) from a user. For example, the user can transmit a proposal request to the information support server 60 from the input unit 42 of the management terminal 40. The staffing request by the user may be input from any terminal.

[0271] 23, the server control unit 63 of the information support server 60 acquires the record information from the information collecting server 30 via the NW communication unit 61 (step S521).

[0272] Next, the prediction processing unit 633 of the server control unit 63 predicts the optimal treatment for the new patient based on the past record information (step S522). Specifically, the prediction processing unit 633 predicts the optimal treatment for each symptom (optimal treatment, optimal drug dosage, optimal tool, optimal treatment timing (including treatment procedure), etc.) based on statistical data indicating the relationship between the treatment performed for each symptom, the type and amount of medication administered, the tool used, and post-treatment progress information, which are compiled by the compilation processing unit 631. The prediction processing unit 633 may predict the optimal treatment for the new patient using learning results (an estimation model that estimates treatment candidates, medication candidates, tool candidates, or treatment timing candidates for each symptom) that have been learned in advance and stored in the learning result storage unit 623. The prediction processing unit 633 stores the prediction results in the prediction result storage unit 625.

[0273] Next, the proposal processor 636 proposes an optimal treatment based on the prediction result (step S523). The proposal processor 636 transmits the content of the optimal treatment, which is the proposal content, to the mobile terminal 10 or any terminal that the user can check via the NW communication unit 61.

[0274] Next, the proposal processing unit 636 determines whether there is a discrepancy between the predicted / proposed treatment and the actual treatment (step S524). In this determination, the presence or absence of a discrepancy between the two can be determined according to a predetermined rule. If there is a discrepancy between the two (step S524: YES), the proposal processing unit 636 proceeds to step S525. On the other hand, if there is no discrepancy between the two (step S524: NO), the proposal processing unit 636 ends the process.

[0275] In step S525, the proposal processing unit 636 outputs a warning. That is, the proposal processing unit 636 transmits a warning to the mobile terminal 10 via the NW communication unit 61, and outputs the warning to the display unit 13 or the speaker 15. Note that the warning may be output from any terminal other than the mobile terminal 10 that the user can confirm. After the processing of step S525, the proposal processing unit 636 ends the processing.

[0276] Next, the external provision process of the medical support system 1b according to this embodiment will be described with reference to Fig. 24. Fig. 24 is a flowchart showing an example of the external provision process of the medical support system 1b according to this embodiment.

[0277] 24, the server control unit 63 of the information support server 60 determines whether a request for externally provided information has been received (step S531). The server control unit 63 determines whether a request for externally provided information has been received, for example, from the management terminal 40, via the NW communication unit 61. If the server control unit 63 has received a request for externally provided information (step S531: YES), the process proceeds to step S532. If the server control unit 63 has not received a request for externally provided information (step S531: NO), the process returns to step S531.

[0278] In step S532, the external provision processing unit 637 of the server control unit 63 generates external provision information in response to the request from the recorded information or the result of tallying the recorded information.

[0279] Next, the external provision processing unit 637 stores the generated external provision information in the external provision information storage unit 628 (step S533). After the process of step S533, the external provision processing unit 637 ends the external provision process.

[0280] As described above, the information support server 60 according to this embodiment includes the tallying processor 631 and the prediction processor 633. The tallying processor 631 tallys up the time required for treatment of patients for each symptom based on medical treatment information indicating the content of medical treatment at the medical institution. When an inquiry is made about the acceptability of emergency transport, the prediction processor 633 predicts the time and number of patients that can be accepted for emergency transport at the medical institution based on the tallying results compiled by the tallying processor 631, the status of medical personnel at the medical institution, and the symptoms of patients involved in emergency transport.

[0281] As a result, the information support server 60 of this embodiment compiles the time required to treat patients for each symptom, and predicts the time and number of patients that can be accepted for emergency transport at a medical institution based on the compilation results, the status of medical personnel at the medical institution, and the symptoms of patients involved in emergency transport, thereby making it possible to more effectively utilize data obtained during medical procedures.

[0282] The information support server 60 of the medical support system 1b according to this embodiment also includes a compilation processing unit 631, a prediction processing unit 633, and a staffing generation unit 634. The compilation processing unit 631 compiles medical treatment information indicating the details of medical treatments at medical institutions. The prediction processing unit 633 predicts the busy season of the medical institution based on the compilation results compiled by the compilation processing unit 631. The staffing generation unit 634 generates a staffing plan of medical personnel capable of handling the busy season of the medical institution predicted by the prediction processing unit 633.

[0283] As a result, the information support server 60 according to this embodiment can use medical treatment information obtained during medical treatment for staff allocation, and can effectively utilize recorded information. Therefore, the medical support system 1b according to this embodiment can reduce the burden on medical staff and improve work efficiency and work accuracy.

[0284] Furthermore, in this embodiment, the counting processing unit 631 counts the number of patients for each symptom based on the medical treatment information. The prediction processing unit 633 predicts the busy season for each medical department based on the number of patients for each symptom counted by the counting processing unit 631. The staffing generation unit 634 generates a staffing plan for medical professionals in each medical department that can handle the busy season based on the busy season for each medical department predicted by the prediction processing unit 633. This allows the information support server 60 according to this embodiment to generate an appropriate staffing plan for medical professionals in each medical department.

[0285] In this embodiment, the prediction processing unit 633 predicts the busy season of the medical institution based on the tabulation results and regional information about the region in which the medical institution is located. The regional information includes at least one of information about the local weather, information about traffic volume in the region, information about events held in the region, information about the local population, information about the local industry, and information about other medical institutions located in the region. This allows the information support server 60 of this embodiment to more accurately predict the busy season of the medical institution based on the regional information about the region in which the medical procedure will be performed.

[0286] In addition, in this embodiment, the aggregation results include statistical data on the number of patients and symptoms for each period, statistical data on symptoms and the time required for medical treatment, and statistical data on patient information and the time required for medical treatment.

[0287] As a result, the information support server 60 according to this embodiment can more accurately predict busy periods at medical institutions by using statistical data for each period and statistical data that takes into account the time required for medical procedures as the aggregation results.

[0288] The information support server 60 according to this embodiment also includes a labor management processing unit 635. The labor management processing unit 635 performs labor evaluation processing and work hour management processing for medical personnel. The tallying processing unit 631 tallyes the content of medical procedures and the time required for the medical procedures for each medical personnel based on the recorded information, and generates statistical data related to the work of the medical personnel. The labor management processing unit 635 then performs labor evaluation processing and labor management processing based on the statistical data related to the work of the medical personnel.

[0289] As a result, the information support server 60 according to this embodiment can use the recorded information obtained during medical procedures to perform labor evaluation processing and labor management processing for medical personnel, thereby enabling even more effective use of the recorded information.

[0290] The information support server 60 according to this embodiment also includes a proposal processor 636. The proposal processor 636 proposes possible treatment options or medication options for a new patient based on the past performance of recorded information. This allows the medical support system 1b according to this embodiment to make more effective use of recorded information.

[0291] In this embodiment, the recorded information also includes progress information indicating the progress after the medical procedure. The proposal processing unit 636 proposes treatments that can be performed on a new patient based on the results of past recorded information including progress information, based on the prediction results from the prediction processing unit 633. The treatments proposed for a new patient include candidate procedures, candidate medications, candidate jigs, or candidate treatment timings.

[0292] As a result, the information support server 60 according to this embodiment can propose possible treatments or medications that can be performed, thereby making it possible to make more effective use of the recorded information.

[0293] In addition, in this embodiment, when proposing a treatment for a new patient, the proposal processing unit 636 proposes candidate medical professionals who can handle the content of the treatment, thereby enabling the information support server 60 according to this embodiment to further improve work efficiency and work accuracy.

[0294] The information support server 6 according to this embodiment also includes an external provision processing unit 637. The external provision processing unit 637 performs a preset processing operation on medical procedure information indicating the details of medical procedures performed at a medical institution to generate information to be provided to an external organization. Here, the external organization may be another medical institution or a medical association, and the information to be provided to the external organization may be information that can be used as data for academic conference presentations related to cases. Alternatively, the external organization may be a government-related organization, and the information to be provided to the external organization may be information that can be used to calculate or revise medical fees, which are determined based on the difficulty of treatment. Alternatively, the external organization may be a pharmaceutical company or a medical device manufacturer, and the information to be provided to the external organization may be information that can be used as research and development data. This allows the information support server 6 according to this embodiment to more effectively utilize medical procedure information.

[0295] The information assistance server 60 according to this embodiment also includes a learning processing unit 632. The learning processing unit 632 performs machine learning on learning data based on the aggregation results aggregated by the aggregation processing unit 631, and generates, as the learning result, a prediction model for predicting busy periods at medical institutions. The prediction processing unit 633 predicts busy periods at medical institutions based on the prediction model.

[0296] As a result, the information support server 60 according to this embodiment can use machine learning to accurately and efficiently predict busy periods at medical institutions.

[0297] The information assistance server 60 according to this embodiment also includes a collection processing unit 191 and a recording processing unit 333. The collection processing unit 191 collects sound information and image information associated with medical procedures. The recording processing unit 333 extracts keywords contained in the collected sound information associated with medical procedures, and stores, as record information, information associating the keywords with date and time information when the keywords were uttered, and information associating the collected image information associated with medical procedures with date and time information when the image information was acquired, in the record information storage unit 327. The record information is used as the medical procedure information. The compilation processing unit 631 compiles the record information stored in the record information storage unit 327.

[0298] As a result, the information support server 60 of this embodiment, like the first embodiment described above, can automatically collect and keep more accurate records of medical procedures, further reducing the burden on medical professionals and improving work efficiency and accuracy.

[0299] The information support method according to this embodiment also includes a compilation step, a prediction step, and a staffing allocation generation step. In the compilation step, the compilation processing unit 631 compiles record information recorded based on information collected during medical procedures, the record information including patient information indicating the patient, medical professional information indicating the medical professional who performed the medical procedure, and medical procedure information indicating the content of the medical procedure associated with date and time information. In the prediction processing step, the prediction processing unit 633 predicts the busy season of the medical institution based on the compilation results compiled by the compilation processing unit 631. In the staffing allocation generation step, the staffing allocation generation unit 634 generates a staffing allocation of medical professionals capable of handling the busy season of the medical institution based on the busy season of the medical institution predicted by the prediction processing unit 633.

[0300] As a result, the information support method according to this embodiment has the same effect as the information support server 60 described above, and can reduce the burden on medical staff and improve work efficiency and work accuracy.

[0301] The information support method according to this embodiment also includes a counting process step and a prediction process step. In the counting process step, the counting processor 631 counts the time required for treatment of each patient symptom based on medical treatment information indicating the content of medical treatment at the medical institution. In the prediction process step, when an inquiry is made about the acceptability of emergency transport, the prediction processor 633 predicts the time and number of patients that can be accepted for emergency transport at the medical institution based on the counting results compiled by the counting processor 631, the status of medical personnel at the medical institution, and the symptoms of the patient involved in the emergency transport.

[0302] As a result, the information support method according to this embodiment has the same effect as the information support server 60 described above, and allows data obtained during medical treatment to be used more effectively.

[0303] In the labor management processing by the labor management processing unit 635, the time required for treatment (required time) may be compared with a reference value to estimate the time and mental burden (for example, reflecting that a treatment that normally takes this long was completed in such a short time). Furthermore, in the labor management processing, the labor management processing unit 635 may perform labor management using a points system, taking into account the estimated time and mental burden in addition to indicators such as the number of emergency patients treated and the treatment time.

[0304] Furthermore, the labor management processing unit 635 may add to the remuneration of doctors and nurses when points exceed a standard in the labor management processing. Furthermore, the external provision processing unit 637 may provide statistical data on the required time for each symptom / disease content compiled by the compilation processing unit 631 to government-related organizations such as the Ministry of Health, Labor and Welfare, so that the data can be used to revise medical fees.

[0305] 25 and 26 are diagrams illustrating the hardware configuration of each device in the medical support system 1 (1a, 1b) according to the embodiment. Fig. 25 shows the hardware configuration of each server device, that is, the information collection server 30 (30a), the hospital management server 50, and the information support server 60.

[0306] As shown in FIG. 25, each of the information collection server 30 (30a), the hospital management server 50, and the information support server 60 includes a communication device H31, a memory H32, and a processor H33.

[0307] The communication device H31 is a communication device such as a LAN card that can be connected to the network NW1. The memory H32 is a storage device such as a RAM, a flash memory, or an HDD, and stores various information and programs used by the information collection server 30 (30a), the hospital management server 50, and the information support server 60.

[0308] The processor H33 is a processing circuit including, for example, a CPU, etc. The processor H33 executes various processes of the information collection server 30 (30a), the hospital management server 50, and the information support server 60 by executing programs stored in the memory H32.

[0309] FIG. 26 shows the hardware configuration of each terminal device, the mobile terminal 10 and the management terminal 40.

[0310] As shown in Figure 26, each terminal device, the mobile terminal 10 and the management terminal 40, includes a communication device H11, an input device H12, a display H13, a wireless communication device H14, a microphone H15, a speaker H16, a camera H17, a memory H18, and a processor H19.

[0311] The communication device H11 is a communication device connectable to the network NW1, such as a LAN card, a wireless LAN card, a mobile communication device, etc. The input device H12 is an input device such as a switch, a button, a touch sensor, or a non-contact sensor.

[0312] The display H13 is, for example, a display device such as a liquid crystal display, an organic electroluminescence (EL) display, etc. The wireless communication device H14 is, for example, a Bluetooth (registered trademark) communication device.

[0313] The microphone H15 is a device for collecting sound. The speaker H16 is a device for emitting sound. The camera H17 has, for example, a CCD image sensor and realizes the imaging unit 16 described above.

[0314] The memory H18 is a storage device such as a RAM, flash memory, or HDD, and stores various types of information and programs used by the mobile terminal 10 and the management terminal 40.

[0315] The processor H19 is, for example, a processing circuit including a CPU, etc. The processor H26 executes various processes of the mobile terminal 10 and the management terminal 40 by executing programs stored in the memory H25.

[0316] The present disclosure is not limited to the above-described embodiments and may be modified within the scope of the present disclosure. For example, in the above-described embodiments, an example has been described in which the information collection server 30 (30a) includes a voice recognition processing unit 331, an image recognition processing unit 332, a recording processing unit 333 (333a), a support processing unit 334, an editing processing unit 335, and a medical document generation unit 336. However, the present disclosure is not limited to such an example. It is sufficient for the medical support system 1 (1a) as a whole to realize the above-described functions. The device configuration may be arbitrary, and each function may be installed in any device. For example, some or all of the functions of the information collection server 30 (30a) may be provided in the mobile terminal 10. For example, in the first embodiment, if the mobile terminal 10 has all of the functions of the information collection server 30, even if a medical procedure is performed during a power outage or in a location where communication with the information collection server 30 is poor, the mobile terminal 10 can execute the functions of the medical support system 1 described above, such as displaying the timeline of recorded information, thereby realizing smooth recording of the medical procedure. In this case, after communication with the information collection server 30 is restored, various information, such as recorded information, generated and stored by the mobile terminal 10 may be transmitted to and stored in the information collection server 30. Similarly, for the medical support system 1b, the device configuration and the functions installed in each device may be arbitrarily modified so that each of the functions described above can be realized. Furthermore, some or all of the matters described in each of the above embodiments may be combined with each other to the extent possible.

[0317] Furthermore, in each of the above embodiments, an example of a support system has been described in which the support system is applied to a medical support system 1 (1a, 1b) in the medical field, but the present invention is not limited to this, and the support system may also be applied to, for example, the manufacturing industry or the construction industry (work in a factory or on a site).

[0318] When applied to the manufacturing and construction industries, the support system of the present disclosure is useful in situations where it is not possible to operate a terminal by hand, such as when wearing thick gloves, when hands get dirty, at high altitudes, underwater, in the dark, etc. The support system of the present disclosure can describe and record the situation using voice, and take photographs using a wearable device such as smart glasses using voice control to record the situation, thereby reducing the burden on workers and improving work efficiency and accuracy.

[0319] In each of the above embodiments, the collected image data may be image data of a medicine pack. In this case, the image recognition processing unit 332 may recognize the type and remaining amount of medicine (i.e., the dosage) from the image data showing the medicine pack, and the recording processing unit 333 (333a) may record the recognition result as record information.

[0320] That is, in the embodiment described above, the image recognition processing unit 332 extracts text data contained in image data, but the present disclosure is not limited to such an example, and the image recognition processing unit 332 may recognize various situations in medical settings using various known image recognition technologies. For example, as described above, the image recognition processing unit 332 recognizes the type and remaining amount of medicine (i.e., the dosage) from image data showing a medicine pack, and the recording processing unit 333 (333a) records the recognition result as record information, making it possible to monitor whether the correct medicine is being administered in the appropriate amount.

[0321] As a result, the recording processing unit 333 (333a) can output a warning when the medication order instructed by voice differs from the type, amount, and flow rate of the medication actually used. In addition to the above example, for example, the image recognition processing unit 332 may identify the medical professional engaged in the medical procedure using facial recognition technology from image data showing the face of the medical professional, or identify symptoms by comparing image data showing the affected area of ​​the patient with a database or a learning model generated by machine learning.

[0322] In the embodiment described above, keywords to be extracted (i.e., keywords to be included in the record information) are stored in advance in the keyword information storage unit 326, and the recording processing unit 333 (333a) extracts keywords to be included in the record information by comparing text data converted from sound data by the voice recognition processing unit 331 and text data detected from image data by the image recognition processing unit 332 with the keywords stored in the keyword information storage unit 326. However, the present disclosure is not limited to such an example. The recording processing unit 333 (333a) may extract keywords using a learning model that machine-learns keywords to be included in the record information. In this case, each time keywords are extracted and record information is created in the medical support system 1 (1a), the learning model may be updated using the record information as learning data. This may improve the accuracy of the learning model each time record information is created using the medical support system 1 (1a).

[0323] Furthermore, the timeline display of recorded information is not limited to the above-described examples. The timeline display may be appropriately devised to improve the visibility of medical professionals viewing the timeline display. For example, when a keyword is displayed in the timeline display, the source of the keyword (i.e., information indicating the speaker if the keyword is extracted from audio data, or the name of the device shown in the image if the keyword is extracted from image data) may be displayed alongside the keyword. Furthermore, for keywords extracted from audio data, information that can identify the speaker (such as the speaker's name or nickname, or an icon or ID indicating the speaker) may be displayed alongside the keyword. This makes it possible to easily confirm the source of the keyword (e.g., who made the statement related to the keyword) on the timeline display.

[0324] Furthermore, for example, keywords may be categorized, and each keyword may be displayed in a different color for each category in the timeline display. For example, keywords may be categorized into categories such as those related to vital signs (blood pressure, pulse rate, etc.), medication names, and medical materials used in treatment, and each category may be displayed in a different color. Furthermore, when the editing processing unit 335 searches for and outputs desired information from the record information in response to a user's instruction, the search may be performed for each category. In this case, the user may be able to specify a period for the record information to be searched. For example, the user may specify a period for each category, such as a list of medications used today or a list of medical devices used over the past year, to search for and display desired information in a list. This increases the user's accessibility to desired information and further improves convenience. Categorization may be performed automatically by the recording processing unit 333 (333a). For example, the recording processing unit 333 (333a) may perform categorization by comparing the information with a predetermined dictionary or using machine learning.

[0325] Each component of the medical support system 1 (1a, 1b) described above has an internal computer system. A program for implementing the functions of each component of the medical support system 1 (1a, 1b) described above may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed to perform processing in each component of the medical support system 1 (1a, 1b) described above. Here, "reading a program recorded on a recording medium into a computer system and executing it" includes installing the program into a computer system. The term "computer system" used here includes hardware such as an OS and peripheral devices.

[0326] Furthermore, a "computer system" may include multiple computer devices connected via a network, including communication lines such as the Internet, WAN, LAN, and dedicated lines. Furthermore, a "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into a computer system. Thus, the recording medium storing the program may be a non-transitory recording medium such as a CD-ROM.

[0327] The recording medium also includes internal or external recording media accessible from a distribution server for distributing the program. The program may be divided into multiple parts, downloaded at different times, and then combined into each component of the medical support system 1 (1a, 1b), or each divided program may be distributed by a different distribution server. Furthermore, the term "computer-readable recording medium" also includes a storage medium that stores a program for a certain period of time, such as volatile memory (RAM) within a computer system that serves as a server or client when a program is transmitted over a network. The program may also be a storage medium for implementing part of the above-described functions. Furthermore, the program may be a so-called differential file (differential program) that can realize the above-described functions in combination with a program already stored in the computer system.

[0328] DESCRIPTION OF SYMBOLS 1, 1a, 1b... Medical support system, 10... Mobile terminal, 11, 31, 41, 61... Network communication unit, 12, 42... Input unit, 13, 43... Display unit, 14, 22... Microphone, 15... Speaker, 16, 23... Imaging unit, 17, 21... Wireless communication unit, 18, 44... Terminal storage unit, 19, 45... Terminal control unit, 20... Headset device, 25... External imaging device, 30, 30a... Information collection server, 32, 62...server storage unit, 33, 33a, 63...server control unit, 40...management terminal, 50...hospital management server, 55...external display device, 60...information support server, 181...setting information storage unit, 191...collection processing unit, 192...output control unit, 193...setting processing unit, 321...terminal information storage unit, 322...user information storage unit, 323...learning result storage unit, 324...collected sound information storage unit, 325...collected image information information storage unit, 326...keyword information storage unit, 327...recorded information storage unit, 328...format storage unit, 329...medical document storage unit, 331...voice recognition processing unit, 332...image recognition processing unit, 333, 333a...recording processing unit, 334...support processing unit, 335...editing processing unit, 336...medical document generation unit, 337, 632...learning processing unit, 451, 551...display control unit, 621...aggregated information storage unit, 622...learning data storage unit, 623...learning result storage unit, 624...regional information storage unit, 625...prediction result storage unit, 626...personnel allocation information storage unit, 627...labor management information storage unit, 628...external provision information storage unit, 631...aggregation processing unit, 633...prediction processing unit, 634...personnel allocation generation unit, 635...labor management processing unit, 636...proposal processing unit, 637...external provision processing unit, NW1...network

Claims

1. A collection and processing unit that collects sound and image information from users, A recording processing unit extracts a first keyword which is a keyword contained in the sound information collected by the collection processing unit, and stores information relating the first keyword to the date and time information in which the first keyword was spoken, and information relating the image information collected by the collection processing unit to the date and time information in which the image information was acquired, as recording information in a recording information storage unit. An output control unit causes a display unit to display a timeline display in which the first keyword and the image information are arranged in chronological order based on the recorded information stored in the recorded information storage unit. An input unit that receives information entered by the user by touching the display unit, Equipped with, When the display unit is displaying the image information, the output control unit adds and displays text based on the user's voice on top of the trajectory traced by the user's finger. Support system.

2. The aforementioned collection processing unit, When a hands-free operation performed by the user without physical contact is detected, and an initiation operation to start collecting the sound information or image information is initiated, the collection of the sound information or image information is initiated. The support system according to claim 1.

3. The aforementioned collection processing unit, In the hands-free operation performed by the user without contact, if an termination operation is detected that ends the collection of the sound information or the image information, the collection of the sound information or the image information will be terminated. The support system according to claim 1.

4. A sound collection unit that collects sounds associated with medical procedures and outputs the sound information, An imaging unit that captures images associated with the aforementioned medical procedure and outputs the image information. Equipped with, The hands-free operation includes at least one of the following: voice operation by the user detected based on the sound information output by the sound pickup unit; motion operation by the user detected based on the image information output by the imaging unit; and gaze operation by the user detected by the headset equipped with the imaging unit. The support system according to claim 2.

5. The recording processing unit extracts a second keyword which is a keyword included in the image information collected by the collection processing unit, and stores information relating the image information, the date and time information on which the image information was acquired, and the second keyword in the recording information storage unit as recording information. The output control unit, based on the recorded information, displays the image information, the date and time information on which the image information was acquired, and the second keyword in association with each other on the timeline display of the display unit. The support system according to any one of claims 1 to 4.

6. The recording processing unit causes the recording information created for each period during which the sound information and the image information were collected to be stored in the recording information storage unit. The output control unit causes the timeline display to be displayed on the display unit for each of the specified periods. The support system according to any one of claims 1 to 4.

7. The first and second keywords mentioned above include at least one of the following: terms related to treatment, terms related to jigs used in the treatment, terms related to drugs, and terms related to the patient's biological information. The support system according to claim 5.

8. The aforementioned users include multiple healthcare professionals, The recording processing unit identifies the speaker among the multiple medical professionals who uttered the first keyword based on the sound information, and stores the information relating the identification information for the speaker, the first keyword, and the date and time information as the recording information in the recording information storage unit. The support system according to any one of claims 1 to 4.

9. The recording processing unit outputs a warning when it detects an abnormality in a medical procedure based on the recording information stored in the recording information storage unit. The support system according to any one of claims 1 to 4.

10. The system includes a support processing unit that performs support processing in response to a user's request for support regarding medical procedures and outputs the support processing results to the user. The support system according to any one of claims 1 to 4.

11. The support processing unit generates a proposal regarding medical procedures based on the past recorded information, and outputs the generated proposal regarding medical procedures to the user as the support processing result. The support system according to claim 10.

12. The recording information storage unit includes an editing processing unit that edits the recorded information stored in it and outputs the edited recorded information to the user. The support system according to any one of claims 1 to 4.

13. The system includes a medical document generation unit that generates medical documents in a predetermined format based on the recorded information stored in the recorded information storage unit. The support system according to any one of claims 1 to 4.

14. The aforementioned display unit is a display unit of a mobile terminal carried by the user. The support system according to any one of claims 1 to 4.

15. The aforementioned display unit is a display device installed in a place where medical procedures are performed. The recording processing unit, via the output control unit, causes the timeline display to be updated with the latest information and displayed on the display device. The support system according to any one of claims 1 to 4.

16. A collection and processing unit that collects sound and image information from users, A recording processing unit extracts a first keyword which is a keyword contained in the sound information collected by the collection processing unit, and stores information relating the first keyword to the date and time information in which the first keyword was spoken, and information relating the image information collected by the collection processing unit to the date and time information in which the image information was acquired, as recording information in a recording information storage unit. Equipped with, The recording processing unit causes the display device to display a timeline display in which the first keyword and the image information are arranged in chronological order, based on the recording information stored in the recording information storage unit. The recording processing unit, when the display device is displaying the image information, receives information input by the user through a touch operation on the display device, and adds and displays text based on the user's voice on top of the trajectory traced by the user's finger. Server device.

17. The collection processing unit collects sound and image information from the user. The recording processing unit extracts a first keyword which is a keyword contained in the sound information collected by the collection processing unit, and stores information relating the first keyword to the date and time information in which the first keyword was spoken, and information relating the image information collected by the collection processing unit to the date and time information in which the image information was acquired, as recorded information in the recording information storage unit. The output control unit causes the display unit to display a timeline display in which the first keyword and the image information are arranged in chronological order, based on the recorded information stored in the recorded information storage unit. The input unit receives information entered by the user by touching the display unit. When the display unit is displaying the image information, the output control unit adds and displays text based on the user's voice on top of the trajectory traced by the user's finger. How to help.