Support system, server device, and support method

Abstract

An assistance system according to the present invention comprises: a collection processing unit that collects sound information and image information associated with a medical care action by a user; a recording processing unit that extracts a first keyword, which is a keyword included in the sound information collected by the collection processing unit, and causes a recording information storage unit to store, as recording information, information in which the first keyword, and date and time information about the date and time at which the first keyword was uttered are associated with each other, and information in which the image information collected by the collection processing unit, and the date and time information about the date and time at which the image information was acquired are associated with each other; and an output control unit that causes a display unit to display a timeline display in which the first keyword and the image information are displayed side by side in chronological order on the basis of the recording information stored in the recording information storage unit.

Description

Technical Field

[0001] The present disclosure relates to a support system, a server device, and a support method.

Background Art

[0002] There is known a technique for reducing the burden on medical staff such as doctors by enabling easy confirmation of data obtained during medical treatment (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The above-described conventional technology facilitates confirmation of the progress of medical treatment by medical staff by displaying character output extracted from keywords in the instructions given by the attending physician together with the time. On the other hand, in order to further reduce the burden on medical staff, there is a need for a technology that records data obtained during medical treatment in more detail.

[0005] The present disclosure has been made to solve the above problems, and an object thereof is to provide a support system, a server device, and a support method that can further reduce the burden on medical staff and improve work efficiency and work accuracy.

Means for Solving the Problems

[0006] In order to solve the above problems, one aspect of the present disclosure is by a user soundA collection processing unit that collects information and image information; a recording processing unit that extracts a first keyword which is a keyword contained in the sound information collected by the collection processing unit, and stores information relating the first keyword to the date and time information in which the first keyword was spoken, and information relating the image information collected by the collection processing unit to the date and time information in which the image information was acquired, as recording information in a recording information storage unit; and an output control unit that displays a timeline display in which the first keyword and the image information are arranged in chronological order based on the recording information stored in the recording information storage unit on a display unit. , an input unit that receives information entered by the user by touching the display unit, Equipped with The output control unit, when the display unit is displaying the image information, adds and displays text based on the user's voice on top of the trajectory traced by the user's finger. It is a support system.

[0007] Furthermore, one aspect of this disclosure is by the user sound The system comprises: an information collection processing unit that collects information and image information; a recording processing unit that extracts a first keyword which is a keyword contained in the sound information collected by the information collection processing unit, and stores as recorded information in a recording information storage unit information relating the first keyword to the date and time information in which the first keyword was spoken, and information relating the image information collected by the information collection processing unit to the date and time information in which the image information was acquired; and the recording processing unit displays on a display device a timeline display in which the first keyword and the image information are arranged in chronological order based on the recorded information stored in the recording information storage unit. The recording processing unit, when the display device is displaying the image information, receives information input by the user through a touch operation on the display device, and adds and displays text based on the user's voice on top of the trajectory traced by the user's finger. This is a server device.

[0008] Furthermore, in one aspect of this disclosure, the data collection processing unit is performed by the user. sound The system collects information and image information, and the recording processing unit extracts a first keyword which is a keyword contained in the sound information collected by the collection processing unit. The system stores information relating the first keyword to the date and time information in which the first keyword was spoken, and information relating the image information collected by the collection processing unit to the date and time information in which the image information was acquired, as recorded information in the recording information storage unit. The output control unit displays a timeline display on the display unit, which arranges the first keyword and the image information in chronological order based on the recorded information stored in the recording information storage unit. The input unit receives information entered by the user through touch operation on the display unit, and the output control unit, while the display unit is displaying the image information, adds and displays text based on the user's voice on the path traced by the user's finger. This is a method of support. [Effects of the Invention]

[0009] According to this disclosure, it is possible to further reduce the burden on healthcare professionals and improve work efficiency and accuracy. [Brief explanation of the drawing]

[0010] [Figure 1] This is a functional block diagram showing an example of a medical support system according to the first embodiment. [Figure 2] This figure shows an example of data in the terminal information storage unit in the first embodiment. [Figure 3] This figure shows an example of data in the user information storage unit in the first embodiment. [Figure 4] This figure shows an example of data from the sound information storage unit in the first embodiment. [Figure 5] This figure shows an example of data from the collected image information storage unit in the first embodiment. [Figure 6] This figure shows an example of data in the recording information storage unit in the first embodiment. [Figure 7] This figure shows an example of how recorded information on a mobile terminal is displayed in the first embodiment. [Figure 8] This figure shows an example of setting the display position and type of biological data in the first embodiment. [Figure 9] This figure shows an example of the collection and recording process of record information in a medical support system according to the first embodiment. [Figure 10] This figure shows an example of support processing for a medical support system according to the first embodiment. [Figure 11] This flowchart shows an example of an abnormality warning process for a medical support system according to the first embodiment. [Figure 12] This figure shows an example of editing and processing of recorded information in a medical support system according to the first embodiment. [Figure 13]It is a diagram showing an example of an archive screen of recording information of a medical support system according to the first embodiment. [Figure 14] It is a diagram showing an example of a summary text screen of recording information of a medical support system according to the first embodiment. [Figure 15] It is a flowchart showing an example of a medical document generation process of a medical support system according to the first embodiment. [Figure 16] It is a functional block diagram showing an example of a medical support system according to the second embodiment. [Figure 17] It is a diagram showing an example of the operation of a medical support system according to the second embodiment. [Figure 18] It is a diagram showing another example of the operation of a medical support system according to the second embodiment. [Figure 19] It is a functional block diagram showing an example of a medical support system according to the third embodiment. [Figure 20] It is a flowchart showing an example of a learning process of a medical support system according to the third embodiment. [Figure 21] It is a flowchart showing an example of a personnel allocation process of a medical support system according to the third embodiment. [Figure 22] It is a flowchart showing an example of a labor management process of a medical support system according to the third embodiment. [Figure 23] It is a flowchart showing an example of a proposal process of a medical support system according to the third embodiment. [Figure 24] It is a flowchart showing an example of an external provision process of a medical support system according to the third embodiment. [Figure 25] It is a diagram for explaining an example of the hardware configuration of each server device of a medical support system according to an embodiment. [Figure 26] It is a diagram for explaining an example of the hardware configuration of each terminal device of a medical support system according to an embodiment.

Modes for Carrying Out the Invention

[0011] Hereinafter, a support system, server device, and support method according to one embodiment of the present disclosure will be described with reference to the drawings.

[0012] (First Embodiment) Figure 1 is a functional block diagram showing an example of the medical support system 1 according to this embodiment. The medical support system 1 is a system that supports medical procedures performed by medical professionals in medical settings such as treatment rooms, operating rooms, and ambulances. Medical professionals who use the medical support system 1 include doctors, nurses, and paramedics.

[0013] As shown in Figure 1, the medical support system 1 comprises a mobile terminal 10, a headset device 20, an external imaging device 25, an information collection server 30, a management terminal 40, and a hospital management server 50. In this embodiment, the mobile terminal 10, the external imaging device 25, the information collection server 30, the management terminal 40, and the hospital management server 50 are connected to a network NW1 and can communicate with each other.

[0014] Medical support system 1 is just one example of a support system. Furthermore, the medical support system 1 may include multiple mobile terminals 10 and headset devices 20, as well as multiple external imaging devices 25.

[0015] The mobile device 10 is a device carried by the user, a healthcare professional, and is, for example, a smartphone, a tablet, etc.

[0016] The mobile terminal 10 comprises a network communication unit 11, an input unit 12, a display unit 13, a microphone 14, a speaker 15, an imaging unit 16, a wireless communication unit 17, a terminal storage unit 18, and a terminal control unit 19.

[0017] The NW communication unit 11 is a communication device that can connect to the network NW1, for example, via a wireless LAN (Local Area Network). The NW communication unit 11 communicates data with the information collection server 30 and the external imaging device 25 via the network NW1. For example, the NW communication unit 11 receives image data captured by the external imaging device 25. Also, for example, the NW communication unit 11 transmits collected information collected by the collection processing unit 191 of the mobile terminal 10 (described later) to the information collection server 30.

[0018] The input unit 12 is an input device such as a keyboard or a touch sensor on a touchscreen, and accepts various information entered by the user. The user can, for example, input various setting information for using the medical support system 1, which is stored in the setting information storage unit 181 (described later), via the input unit 12.

[0019] The display unit 13 is, for example, a liquid crystal display and displays (outputs) various information used by the mobile terminal 10. The display unit 13 displays, for example, sound data (sound information) and recorded information based on image data collected in connection with medical procedures.

[0020] The microphone 14 picks up sounds from the surrounding area of ​​the mobile terminal 10 and outputs sound data (sound information) to the terminal control unit 19. The microphone 14 is a built-in microphone of the mobile terminal 10 and is an example of a sound pickup unit.

[0021] Speaker 15 is a built-in speaker of the mobile terminal 10 and outputs various sound data. For example, when a user requests to check the corresponding sound data for the timeline display of recorded information (described later), speaker 15 outputs (breathes out) the sound data included in the recorded information (sound data corresponding to the original data). Speaker 15 also outputs (breathes out) warning sounds from the information collection server 30, sound results of various information searches, and sound suggestions, etc.

[0022] The imaging unit 16 is, for example, a camera device that captures still images and video images, and is the built-in camera of the mobile terminal 10. The imaging unit 16 captures various images and outputs image data (image information) to the terminal control unit 19. The image information (image data) includes data for still images and video data.

[0023] The wireless communication unit 17 is, for example, a Bluetooth® communication device and performs data communication with the headset device 20. The wireless communication unit 17 receives, for example, sound data and image data acquired by the headset device 20 and outputs them to the terminal control unit 19.

[0024] The terminal storage unit 18 is a storage unit realized by the memory (not shown) provided by the mobile terminal 10, and stores various information used by the mobile terminal 10. The terminal storage unit 18 includes a configuration information storage unit 181.

[0025] The setting information storage unit 181 stores various setting information for the mobile terminal 10 used to operate the medical support system 1. For example, the setting information storage unit 181 stores setting information for hands-free operations that perform various operations without contact in the medical support system 1. The setting information related to hands-free operations includes information about actions corresponding to various operations, such as a start operation to begin collecting sound data and image data, a stop operation to end the collection of sound data and image data, an information retrieval request operation, and an assistance request operation.

[0026] Here, hands-free operation includes, for example, operation by the user's voice, operation by the user's gaze, and motion operation using the hand or fingers. Furthermore, the setting information storage unit 181 stores, for example, information about voice keywords corresponding to various operations as setting information related to voice operation.

[0027] The terminal control unit 19 is a functional unit that is realized, for example, by causing a processor (not shown) including a CPU (Central Processing Unit) to execute a program stored in the terminal storage unit 18. The terminal control unit 19 comprehensively controls the mobile terminal 10 and executes various processes.

[0028] The terminal control unit 19 comprises a data collection processing unit 191, an output control unit 192, and a setting processing unit 193. The data collection processing unit 191, the output control unit 192, and the setting processing unit 193 are implemented by having a processor (not shown) execute, for example, an application program for using the medical support system 1.

[0029] The data collection processing unit 191 collects sound data (sound information) and image data (image information) associated with medical procedures performed by the user. Here, the sound data includes, for example, voice data spoken by the user (medical professional), and the image data includes still image data or video data captured by the user using the headset device 20 described later, and still image data or video data captured by the external imaging device 25.

[0030] The data collection processing unit 191 starts collecting sound data or image data when it detects, for example, a start operation to begin collecting sound data or image data through hands-free operation by a medical professional. Alternatively, the data collection processing unit 191 starts collecting sound data or image data when, for example, sound data acquired from the microphone 14 or headset device 20 is converted to text by the speech recognition processing unit 331 of the information collection server 30 (described later), and keywords are extracted by the recording processing unit 333, and the extracted keywords match keywords corresponding to the start operation stored in the setting information storage unit 181.

[0031] The data collection processing unit 191 transmits the collected sound data or image data to the information collection server 30 via the network communication unit 11. The sound data transmitted to the information collection server 30 is stored in the collected sound information storage unit 324, described later, in association with date and time information. Similarly, the image data transmitted to the information collection server 30 is stored in the collected image information storage unit 325, described later, in association with date and time information.

[0032] Furthermore, the data collection processing unit 191 terminates the collection of sound data or image data when it detects a termination operation to end the collection of sound data or image data via hands-free operation. The data collection processing unit 191 terminates the collection of sound data or image data when, for example, sound data acquired from the microphone 14 or headset device 20 is converted to text by the speech recognition processing unit 331 of the information collection server 30 (described later), and keywords are extracted, and the extracted keywords match keywords corresponding to termination operations stored in the setting information storage unit 181. Alternatively, the data collection processing unit 191 terminates the collection of sound data or image data when, for example, image data acquired from the imaging unit 16, headset device 20, or external imaging device 25 is analyzed by the image recognition processing unit 332 of the information collection server 30 (described later), and a predetermined user action is extracted, and the action matches an action corresponding to termination operations stored in the setting information storage unit 181.

[0033] Furthermore, if the data collection processing unit 191 detects, via hands-free operation, other operations such as a request for information retrieval or a request for support, it transmits a processing request corresponding to the detected request to the information collection server 30 via the network communication unit 11.

[0034] The output control unit 192 controls the operation of the display unit 13 and causes it to display various types of information. Specifically, the output control unit 192 displays a timeline on the display unit 13, which arranges keywords and image information in chronological order based on the recorded information stored in the recorded information storage unit 327 of the information collection server 30, which will be described later. For example, the output control unit 192 displays on the display unit 13 a link between date and time information (e.g., time information), keywords, and image data, based on the recorded information. Here, the keywords are keywords extracted from sound data or image data. Details of the keywords and the details of the display content of the output control unit 192 will be described later. The output control unit 192 also controls the operation of the speaker 15 and causes it to output (emit) various types of information. Specifically, the output control unit 192 outputs (emits) sound data included in the recorded information to the speaker 15. Furthermore, the output control unit 192 outputs (breathes out) warning sounds from the information collection server 30, various information search result sounds, and various suggestion sounds to the speaker 15.

[0035] The setting processing unit 194 performs various setting processes and various setting change processes for using the medical support system 1, for example, in response to user input via the input unit 12. The setting processing unit 194 acquires various setting information via the input unit 12 and stores it in the setting information storage unit 181. The setting information is, for example, setting information for hands-free operation, which allows various operations to be performed without contact in the medical support system 1.

[0036] The headset device 20 is, for example, a device worn on the head by a medical professional, and primarily transmits the medical professional's voice data and image data of objects the medical professional is viewing to the mobile terminal 10 via wireless communication. The headset device 20 comprises a wireless communication unit 21, a microphone 22, and an imaging unit 23.

[0037] The wireless communication unit 21 is, for example, a Bluetooth® communication device, and performs data communication wirelessly with the wireless communication unit 17 of the mobile terminal 10 described above. The wireless communication unit 21 transmits, for example, voice data and image data to the mobile terminal 10 wirelessly.

[0038] Microphone 22 primarily captures sounds emitted by medical personnel during medical procedures and generates sound data. Microphone 22 is an example of a sound-collecting unit, and the sound data generated by microphone 22 is transmitted to the mobile terminal 10 via the wireless communication unit 21.

[0039] The imaging unit 23 is, for example, a camera device that captures still images and moving images. For example, during a medical procedure, it captures images including the object that a medical professional is looking at and generates image data. The image data generated by the imaging unit 23 is transmitted to the mobile terminal 10 via the wireless communication unit 21.

[0040] The external imaging device 25 is installed in a medical setting (treatment room, operating room, ambulance, etc.) to capture images of the medical environment. For example, the external imaging device 25 is placed in front of medical equipment such as measuring devices and dispensing devices, and during medical procedures, it captures images of the measuring device's screen, images showing the status of drug dispensing, etc., and generates (collects) various image data. Alternatively, the external imaging device 25 is a fixed-point camera that is placed in a location that allows for an overhead view of the medical setting (especially treatment tables, operating tables, and ambulances) such as the ceiling, and captures images of the medical personnel and patients (including the affected area) during medical procedures. The external imaging device 25 is an example of an imaging unit. The image data generated (collected) by the external imaging device 25 is transmitted to the mobile terminal 10 via the network NW1.

[0041] The management terminal 40 is a terminal device for managing the medical support system 1, and is, for example, a personal computer, smartphone, tablet, etc. The management terminal 40 verifies, edits, and processes recorded information during medical procedures, and controls the creation of medical documents such as electronic medical records, claims (medical fee statements), and prescriptions. The management terminal 40 comprises a network communication unit 41, an input unit 42, a display unit 43, a terminal storage unit 44, and a terminal control unit 45.

[0042] The NW communication unit 41 is a communication device that can connect to the network NW1, for example, via a wireless LAN or a wired LAN. The NW communication unit 41 communicates data with, for example, the information collection server 30 via the network NW1. The NW communication unit 41 transmits various control information (various processing requests) for the management of the medical support system 1 to the information collection server 30. The NW communication unit 41 receives, for example, various display information based on recorded information from the information collection server 30.

[0043] The input unit 42 is an input device such as a keyboard, a touch sensor on a touchscreen, or a mouse, and receives various types of information entered by the user or administrator of the medical support system 1. For example, the input unit 42 receives requests from users to edit recorded information and requests from users to create medical documents.

[0044] The display unit 43 is, for example, a liquid crystal display and displays (outputs) various information used by the management terminal 40. The display unit 43 displays, for example, recording information, search information for recording information, editing information, archive information, etc.

[0045] The terminal storage unit 44 is a storage unit implemented by the memory (not shown) of the management terminal 40, and stores various information used by the management terminal 40. The terminal control unit 45 is a functional unit that is realized, for example, by causing a processor (not shown) including a CPU to execute a program stored in the terminal storage unit 44. The terminal control unit 45 comprehensively controls the management terminal 40 and executes various processes. The terminal control unit 45 includes a display control unit 451.

[0046] The display control unit 451 displays various information received from the information collection server 30 via the network communication unit 41 on the display unit 43. The display control unit 451 displays screens on the display unit 43 for viewing, searching, modifying, processing, etc., of recorded information recorded during medical procedures. The display control unit 451 also displays on the display unit 43, for example, an archive screen or summary screen which is a list of medical procedures performed on a patient.

[0047] The hospital management server 50 is a server device that performs tasks such as registering and managing patient information, managing electronic medical records, outputting prescriptions and medical fee statements, and is an existing hospital management system. The hospital management server 50 is connected to network NW1 and can communicate data with, for example, the information collection server 30 and the management terminal 40.

[0048] The information collection server 30 is a server device that records various information (recorded information) related to medical procedures performed by medical professionals and performs support processing to reduce the burden on medical professionals, such as displaying information based on the recorded information on the display unit 13 of the mobile terminal 10. The information collection server 30 can communicate with the mobile terminal 10, the management terminal 40, and the hospital management server 50 via the network NW1. The information gathering server 30 comprises a network communication unit 31, a server storage unit 32, and a server control unit 33.

[0049] The NW communication unit 31 is a communication device that can connect to the network NW1, for example, via a wireless LAN or a wired LAN. The NW communication unit 31 performs data communication with, for example, the mobile terminal 10 and the management terminal 40 via the network NW1. The NW communication unit 31 receives collected information, such as sound data and image data, from the mobile terminal 10. The NW communication unit 31 also transmits display information to the mobile terminal 10 or the management terminal 40.

[0050] The server storage unit 32 is a storage unit realized by the memory (not shown) provided by the information collection server 30, and stores various types of information used by the information collection server 30.

[0051] The server storage unit 32 includes a terminal information storage unit 321, a user information storage unit 322, a learning result storage unit 323, a collected sound information storage unit 324, a collected image information storage unit 325, a keyword information storage unit 326, a recording information storage unit 327, a format storage unit 328, and a medical document storage unit 329.

[0052] The terminal information storage unit 321 stores information about the mobile terminal 10 and external imaging device 25 that can be used in the medical support system 1. Here, an example of data in the terminal information storage unit 321 will be explained with reference to Figure 2.

[0053] Figure 2 shows an example of data in the terminal information storage unit 321 in this embodiment. As shown in Figure 2, the terminal information storage unit 321 stores terminal information that associates the terminal ID, MAC address, user ID, device type, and collected information.

[0054] Here, the terminal ID is identification information that identifies the mobile terminal 10 or the external imaging device 25. The MAC address is address information that identifies the device on the network NW1. The user ID is user identification information that identifies the user (healthcare professional) using the mobile terminal 10 or the external imaging device 25. The device type indicates the type of device (either the mobile terminal 10 or the external imaging device 25), and the collected information indicates the type of data that can be collected.

[0055] In the example shown in Figure 2, the device with terminal ID "MN001" has a MAC address of "XX:XX:XX:XX:XX:XX" and a user ID of "U0001". It also indicates that the device type is a "mobile terminal" and the collected information is "sound and images".

[0056] Returning to the explanation of Figure 1, the user information storage unit 322 stores information about users (healthcare professionals) who use the medical support system 1. Now, referring to Figure 3, we will explain an example of data in the user information storage unit 322.

[0057] Figure 3 shows an example of data in the user information storage unit 322 in this embodiment. As shown in Figure 3, the user information storage unit 322 stores the user ID, name, occupation information, and status in association with each other. Here, the occupation information indicates the user's occupation, such as nurse, doctor, radiographer, or paramedic. The status indicates the user's current work status. The status is used when requesting assistance from users on standby, or when seeking opinions on medical procedures or patient symptoms.

[0058] For example, the example shown in Figure 3 indicates that user ID "U0001" is a user (healthcare worker) whose name is "△△ Ichiro" and whose occupation is "nurse". It also indicates that this user's status is "receiving medication / treatment".

[0059] Furthermore, it indicates that user ID "U0002" is a healthcare professional whose name is "Taro XX" and whose occupation is "Doctor". It also indicates that this user's status is "Waiting".

[0060] Returning to the explanation of Figure 1, the learning result storage unit 323 stores the learning results of various machine learning processes performed by the learning processing unit 337, which will be described later. The learning result storage unit 323 stores, for example, speech recognition models, image recognition models, abnormality detection models for medical professionals' procedures and medication administration, and proposed models related to medical procedures as learning results.

[0061] The sound information storage unit 324 stores the collected sound data (raw data). Here, with reference to Figure 4, an example of the data stored in the sound information storage unit 324 will be explained. Figure 4 shows an example of data from the sound information storage unit 324 in this embodiment.

[0062] As shown in Figure 4, the collected sound information storage unit 324 stores the data ID, terminal ID, date and time information, and sound information in association with each other. Here, the data ID is data identification information assigned by the collection processing unit 191 described above to a series of sound data collected during the period from the detection of the start operation to the detection of the end operation of sound data collection, and the terminal ID is identification information that identifies the mobile terminal 10 that collected the data. The date and time information indicates the date and time when the sound data was collected (for example, the date and time when collection started). The sound information is information that indicates the sound data, for example, the file name of the sound data.

[0063] In the example shown in Figure 4, it is indicated that the sound data with data ID "DT00001" was acquired by mobile terminal 10 with terminal ID "MT001" at the date and time "20XX / 11 / 01 10:15" (November 1, 20XX, 10:15 AM). It is also indicated that the sound data is stored as sound information "SXXX1".

[0064] Returning to the explanation of Figure 1, the collected image information storage unit 325 stores the collected image data (raw data). Now, referring to Figure 5, an example of the data stored in the collected image information storage unit 325 will be explained. Figure 5 shows an example of data from the collected image information storage unit 325 in this embodiment.

[0065] As shown in Figure 5, the collected image information storage unit 325 stores the data ID, terminal ID, date and time information, and image information in association with each other. Here, the data ID is data identification information assigned by the collection processing unit 191 described above to a series of image data collected during the period from the detection of the start operation to the detection of the end operation of image data collection, and the terminal ID is identification information that identifies the mobile terminal 10 (or external imaging device 25) that collected the data. The date and time information indicates the date and time when the image data was collected (for example, the date and time when collection started). The image information is information that indicates the image data, for example, the file name of the image data (for example, video data).

[0066] In the example shown in Figure 5, it is indicated that image data with data ID "DT10001" was acquired by mobile terminal 10 with terminal ID "MT001" at the date and time "20XX / 11 / 01 10:15" (November 1, 20XX, 10:15 AM). It is also indicated that the image data is stored as image information "MVXXX1".

[0067] Returning to the explanation of Figure 1, the keyword information storage unit 326 stores information indicating keywords to be extracted from sound data or image data. Here, the keywords include at least one of the following: terms related to the procedure (e.g., "hemostasis", "intravenous drip", etc.), terms related to the tools used in the procedure (e.g., "scalpel", "forceps", etc.), terms related to drugs, and terms related to the patient's biological information (e.g., "blood pressure", "pulse", "respiratory rate", etc.).

[0068] The recording information storage unit 327 stores recording information based on the collected sound data and image data. The recording information is stored in the recording information storage unit 327 by the recording processing unit 333, which will be described later. Now, with reference to Figure 6, an example of the data in the recording information storage unit 327 will be explained.

[0069] Figure 6 shows an example of data in the recording information storage unit 327 in this embodiment. As shown in Figure 6, the recording information storage unit 327 stores recording information that associates patient name, date and time information, user ID, medical procedure information, attached image, and source data. Here, the patient name indicates the name of the patient who received the medical procedure, and the user ID is the identification information of the medical professional. Note that the patient name is an example of identification information that identifies a patient. The medical procedure information indicates the content of the medical procedure, for example, by showing keywords extracted from sound data or image data, and the attached image indicates image data that is displayed together with the keywords. The source data indicates the sound data (audio data) and image data (video data, etc.) that were the basis for the recording information.

[0070] For example, in the example shown in Figure 6, the recorded information for patient name "Hanako ○○" indicates that on November 1, 20XX, at 9:25 PM, a user with user ID "U0001" performed a medical procedure, uttering the phrase "Consciousness level: Clear," and an image "G0001" was recorded as an attached image. It also indicates that the source data for this recorded information is "DT00001, DT10001." The source data consists of the data IDs of the collected sound information storage unit 324 and the collected image information storage unit 325.

[0071] Thus, the recording information storage unit 327 stores as recorded information information that associates a keyword with the date and time information in which the keyword was spoken, and as recorded information information that associates the collected image information with the date and time information in which the image information was acquired. In addition, the recording information storage unit 327 may also store as recorded information information that associates image data with date and time information and keywords detected from the image data.

[0072] The recording information storage unit 327 may also store information such as patient information, information about the medical professionals (doctors, nurses) who provided care, the content of the medical procedure (time-series data linked to time information), the time taken for the procedure (procedure duration), and the progress after the procedure (treatment effect).

[0073] Returning to the explanation of Figure 1, the format storage unit 328 stores format information (format information) for creating medical documents (e.g., electronic medical records, prescriptions, etc.). The format storage unit 328 stores, for example, identification information indicating a medical document and the format information in association.

[0074] The medical document storage unit 329 stores medical documents generated from recorded information by the medical document generation unit 336, which will be described later. These medical documents include, for example, electronic medical records, prescriptions, and medical fee statements.

[0075] The server control unit 33 is a functional unit that is realized, for example, by causing a processor (not shown) including a CPU to execute a program stored in the server storage unit 32. The server control unit 33 comprehensively controls the information collection server 30 and executes various processes. The server control unit 33 includes a speech recognition processing unit 331, an image recognition processing unit 332, a recording processing unit 333, a support processing unit 334, an editing processing unit 335, a medical document generation unit 336, and a learning processing unit 337.

[0076] The speech recognition processing unit 331 converts the sound data (speech data) collected by the collection processing unit 191 into text data (character data). Various known methods may be used to convert sound data into text data. For example, the speech recognition processing unit 331 may convert sound data into text data based on the learning results (speech recognition model) stored in the learning result storage unit 323.

[0077] The image recognition processing unit 332 detects text data (character data) contained in image data (video data, etc.) collected by the data collection processing unit 191. For example, the image recognition processing unit 332 extracts text data indicating the patient's biological data (heart rate, blood pressure, etc.) and the type or amount / flow rate of medication used from images of the screens of medical devices such as measuring devices and drug dispensing devices captured by the external imaging device 25, using image recognition. The image recognition processing unit 332 also detects pre-set motions (hands-free operation) from image data collected by the data collection processing unit 191. Various known methods may be used to detect text data and pre-set motions from image data. For example, the image recognition processing unit 332 may detect text data and pre-set motions from image data based on the learning results (image recognition model) stored in the learning result storage unit 323.

[0078] The recording processing unit 333 extracts keywords contained in the sound data collected by the collection processing unit 191, and stores in the recording information storage unit 327 information relating the keywords to the date and time information (e.g., time information) in which the keywords were spoken, and information relating the image data collected by the collection processing unit 191 to the date and time information (e.g., time information) in which the image data was acquired, as recording information. At this time, the recording processing unit 333 creates recording information including, for example, the patient's name and the user ID (healthcare worker's ID), and stores the recording information in the recording information storage unit 327 in the format shown in Figure 6 above. Specifically, the recording processing unit 333 extracts keywords contained in the sound data by comparing the text data converted from the sound data by the speech recognition processing unit 331 with pre-set keywords stored in the keyword information storage unit 326 (for example, the terms related to the procedure, the terms related to the tools used in the procedure, the terms related to the drugs, and the terms related to the patient's biological information, etc., as described above). In this specification, keywords extracted from sound data will also be referred to as the first keyword.

[0079] Furthermore, the recording processing unit 333 may extract keywords contained in the image data collected by the collection processing unit 191 and store information relating the image data, the date and time information (e.g., time information) on which the image data was acquired, and the keywords detected from the image data as recording information in the recording information storage unit 327. In this case, the recording processing unit 333 extracts keywords contained in the image data by comparing the text data (character data) detected from the image data by the image recognition processing unit 332 with the pre-set keywords stored in the keyword information storage unit 326 (e.g., terms related to the procedure, terms related to the tools used in the procedure, terms related to the drugs, and terms related to the patient's biological information, etc., as described above). In this specification, keywords extracted from image data are also referred to as second keywords.

[0080] Furthermore, the recording processing unit 333 creates recording information for each period during which sound data and image data are collected, and stores the created recording information in the recording information storage unit 327. In addition, the recording processing unit 333 may identify the speaker who uttered the keyword among multiple medical professionals based on the sound data (voice data), and store information as recording information in the recording information storage unit 327, as identification information that identifies the speaker, the keyword, and date and time information. That is, the recording processing unit 333 stores the recording information storage unit 327 with the recording information as shown in Figure 6.

[0081] Furthermore, when the data collection processing unit 191 aggregates collected data from multiple mobile terminals 10, multiple headset devices 20, and multiple external imaging devices 25 located in the same medical setting (treatment room, operating room, etc.) to create recorded information, the recording processing unit 333 may create the recorded information including the identification information of each of the multiple mobile terminals 10, multiple headset devices 20, and multiple external imaging devices 25 (that is, the recording information is created in such a way that it can be identified which mobile terminal 10, headset device 20, and external imaging device 25 the sound data and image data that form the basis of the recorded information were collected from), and store the recorded information in the recorded information storage unit 327.

[0082] Furthermore, the recording processing unit 333, based on the recording information stored in the recording information storage unit 327, causes the mobile terminal 10 to display a timeline display, such as the one shown in Figure 7, via the output control unit 192. Specifically, the recording processing unit 333 transmits the recording information stored in the recording information storage unit 327 to the mobile terminal 10 via the NW communication unit 31, and the output control unit 192 of the mobile terminal 10 displays a timeline display, such as the screen G1 shown in Figure 7, on the display unit 13 based on the recording information. The recording processing unit 333 may also display the timeline display of the recording information on a display device other than the mobile terminal 10.

[0083] Figure 7 shows an example of how the recorded information of the mobile terminal 10 in this embodiment is displayed. As shown in Figure 7, the mobile terminal 10 displays keywords and image data in chronological order on the display unit 13.

[0084] In Figure 7, image data PH1 is an image of the burned area of ​​the patient captured by the imaging unit 23 of the headset device 20. The recording processing unit 333 and the output control unit 192 may superimpose, for example, lines surrounding the burned area or text (e.g., "Depth I", "Depth III", etc.) onto the image. In other words, the recording processing unit 333 and the output control unit 192 may add and display any object based on user instructions (e.g., lines surrounding the affected area, text entered by the user's voice or any input device) at a specified position in the image data displayed by the display unit 13.

[0085] In this case, the location where an object is to be added can be specified by any method. For example, the user may specify the location by moving a cursor on the screen via the pointing device included in the input unit 12. For example, the user can draw a line at a location corresponding to the trajectory of the cursor's movement by performing a so-called drag operation, or specify the area where text will be displayed. Alternatively, if the display unit 13 is a touchscreen (i.e., the input unit 12 is a touch sensor included in the touchscreen, and input is possible by touch operation on the screen of the display unit 13), the user may draw a line by tracing on the screen or specify the location where text will be displayed.

[0086] Furthermore, when text is displayed, a so-called "speech-to-draw" technique may be used, in which text based on the user's voice is superimposed on the path traced by the cursor or finger. For "speech-to-draw," for example, the techniques described in Japanese Patent Publication No. 2018-22528 and Japanese Patent Publication No. 2019-71104 can be utilized.

[0087] Furthermore, the recording processing unit 333 and the output control unit 192 may be configured to automatically suggest, for example, the extent of the burn, the depth of each burn, the burn to be annotated, etc., using the machine learning results (i.e., the image recognition model stored in the learning result storage unit 323) as determined by the image recognition processing unit 332. Furthermore, the recording processing unit 333 and the output control unit 192 may, from a privacy standpoint, mask the patient's face in the output image.

[0088] Furthermore, in Figure 7, when the recording processing unit 333 detects that the button BT1 has been pressed by the user, it outputs the sound data (voice data) corresponding to the original data from the sound data stored in the collected sound information storage unit 324 to the speaker 15 of the mobile terminal 10 via the output control unit 192.

[0089] Furthermore, in Figure 7, the image data PH2 is an image of the display screen of a medical device captured by an external imaging device 25 (an example of an imaging unit). As described above, the image recognition processing unit 332 can extract text data indicating the patient's biological data from the image data PH2 of such a medical device's display screen. In doing so, the image recognition processing unit 332 associates the display position and type of biological data (pulse, blood pressure, etc.) on the screen for each medical device, and based on this information, extracts text data (keywords) such as pulse and blood pressure from the image data PH2 (details will be described later with reference to Figure 8). The extracted text data is recorded as recording information by the recording processing unit 333 in association with the original image data PH2, and is displayed together with the original image data PH2 in the timeline display as shown in Figure 7.

[0090] Furthermore, if the recording processing unit 333 detects an abnormality from the recorded information, it outputs a warning from the display unit 13 or speaker 15 of the mobile terminal 10 via the output control unit 192. For example, rules for detecting abnormalities may be set in advance based on expected abnormality patterns, and the recording processing unit 333 detects abnormalities according to these rules. Specifically, for example, the recording processing unit 333 refers to the recorded information and determines whether the content of the audio data indicating the medical professional's voice-based drug order extracted from the sound data matches the content of the audio data indicating the type or amount / flow rate of the drug actually used extracted from the screen data of the drug dispensing device. If the audio-based drug order does not match the type or amount / flow rate of the drug actually used, the recording processing unit 333 outputs a warning from the display unit 13 or speaker 15 of the mobile terminal 10 via the output control unit 192. Furthermore, the recording processing unit 333 may refer to the recorded information and, based on the text data indicating the remaining amount of medication such as intravenous drips extracted from the image data, determine whether the remaining amount is appropriate to the instructions given by the medical professional. If it is determined to be inappropriate, it may output a warning from the display unit 13 or speaker 15 of the mobile terminal 10 via the output control unit 192. Note that the warning does not have to be output from the display unit 13 or speaker 15 of the mobile terminal 10; it may be given in the form of a display or sound from another device installed in the medical setting (treatment room, operating room, etc.). In addition, the recording processing unit 333 may detect an anomaly and issue a warning using the results of machine learning, i.e., the anomaly detection model of the medical professional's procedures and medication administration stored in the learning result storage unit 323.

[0091] Furthermore, the recording processing unit 333 may be configured to automatically instruct the data collection processing unit 191 to start audio and image recording if there is a sudden change in the patient's biological data such as heart rate and blood pressure. Specifically, the recording processing unit 333 determines, based on text data indicating the patient's biological data extracted from the screen data of the measuring device, whether there is a sudden change in the patient's biological data such as heart rate and blood pressure that exceeds predetermined reference values, and based on the result, can instruct the data collection processing unit 191 to start audio and image recording.

[0092] Figure 8 shows an example of setting the display position and type of biological data in this embodiment. In Figure 8, it is assumed that the second numerical data from the left on the top line, DT1, is pre-set as "pulse rate," and the numerical data on the middle line, DT2, is pre-set as "blood pressure." With these settings in place, the image recognition processing unit 332 extracts text data indicating heart rate and blood pressure values ​​from the image data collected by the data acquisition processing unit 191.

[0093] Returning to the explanation of Figure 1, the support processing unit 334 executes various support processes. When the support processing unit 334 detects a support request from a user regarding medical procedures, it executes support processing corresponding to the support request and outputs the support processing results to the user. For example, support requests may include searching for information on medical procedures, making suggestions regarding medical procedures, or requesting assistance from other users.

[0094] For example, when the support processing unit 334 receives a request from the mobile terminal 10 for a user to search for information related to medical procedures, it searches for information such as terminology, procedures, medication, treatment, etc., and transmits the search results to the mobile terminal 10. The support processing results are then output from the display unit 13 or speaker 15 via the output control unit 192.

[0095] Furthermore, when the support processing unit 334 receives a request from the mobile terminal 10 for a suggestion regarding medical procedures from the user, it generates a suggestion regarding medical procedures based on past recorded information, transmits the generated suggestion regarding medical procedures to the mobile terminal 10, and outputs it as a support processing result from the display unit 13 or speaker 15 via the output control unit 192. For example, the suggestion regarding medical procedures is a suggestion regarding the optimal treatment (procedure or medication, etc.) for the patient. For example, the support processing unit 334 searches past recorded information, extracts recorded information with a high degree of similarity to the current recorded information, and proposes the treatment method, the type and amount of medication to be administered, etc., based on the recorded information with a high degree of similarity. The support processing unit 334 may also make suggestions for procedures, medication, etc., using, for example, the machine learning results from the learning processing unit 337 (i.e., the suggestion model regarding medical procedures stored in the learning result storage unit 323).

[0096] Furthermore, the support processing unit 334 may, for example, when it receives a request for assistance from a user to another user from a mobile terminal 10, check the status of the user stored in the user information storage unit 322, extract a user who can respond to the request for assistance, and arrange for assistance (specifically, calling the user to the treatment room or requesting remote advice) from the available user (such as a doctor). When arranging assistance, the support processing unit 334 may also transmit information related to the timeline display of recorded information to the available user (such as a doctor) to share information about the patient and the details of the medical procedures performed up to that point.

[0097] Furthermore, user support requests do not necessarily have to be entered from the mobile terminal 10; they may be entered from other terminals. Also, the method of inputting user support requests is not limited; for example, voice input, text input, gesture input, etc., may be used. In addition, the results of the support processing by the support processing unit 334 do not necessarily have to be output to the mobile terminal 10; they may be output to the user via other terminals.

[0098] Returning to the explanation of Figure 1, the editing processing unit 335 performs editing processes such as viewing, searching, editing, and processing the recorded information stored in the recorded information storage unit 327. Specifically, the editing processing unit 335 edits the recorded information stored in the recorded information storage unit 327 and outputs the edited recorded information to the user. For example, the editing processing unit 335 displays the recorded information in a timeline display on the display unit 43 of the management terminal 40 (terminal device) via the display control unit 451. The user can perform editing operations, such as correcting keywords, while referring to the display screen. The editing processing unit 335 receives an editing request from the management terminal 40 in response to the editing operation, edits the recorded information stored in the recorded information storage unit 327 in accordance with the editing request, and outputs the edited recorded information to the management terminal 40. This makes it possible to easily correct errors even if there are errors in keyword extraction by the speech recognition processing unit 331 and the image recognition processing unit 332. For convenience during the editing process, the editing processing unit 335 can also display only the recorded information containing the specified keyword on the timeline (search and filtering process).

[0099] Furthermore, when recording information is created using information aggregated from multiple mobile terminals 10, multiple headset devices 20, and multiple external imaging devices 25, the editing processing unit 335 may allow the user (or administrator) to select keywords to be used (or not used) in the medical document via the management terminal 40, and reflect these selections in the medical document created by the medical document generation unit 336, which will be described later. The editing processing unit 335 may, for example, prioritize the information collected from the doctor's mobile terminal 10 over the information collected from the nurse's mobile terminal 10, and integrate the information accordingly.

[0100] Furthermore, the editing processing unit 335 may generate a summary of medical procedures using the recorded information stored in the recorded information storage unit 327 and display it, for example, on the display unit 43 of the management terminal 40 via the display control unit 451. The editing processing unit 335 may also generate a list (archive information) from the recorded information stored in the recorded information storage unit 327, including, for example, the date, the attending physician, the attending nurse, and the symptoms, and display it on the display unit 43 of the management terminal 40 via the display control unit 451. The generation of summaries and archive information may be performed automatically by the editing processing unit 335 or by user input. When the generation of summaries and archive information is performed automatically, various generation AI tools may be utilized. When the generation of summaries and archive information is performed by user input, the user may specify the content to be reflected in the summary and archive information from the recorded information, as well as its format. That is, based on user input, the editing processing unit 335 may extract only the necessary information from the recorded information and output it in an arbitrary format. In this case, the editing processing unit 335 may provide a search function that allows the user to search for desired information from the recorded information, and the user may use this search function to search for and specify the desired information to be output from the recorded information. In this case, the user may also specify the period of the recorded information to be searched (e.g., today, the past year). Note that the terminal to which the editing request is entered and the terminal to which the results of the editing process are output are not limited to the management terminal 40, but may be other terminals.

[0101] The medical document generation unit 336 generates medical documents in a pre-set format based on the recorded information stored in the recorded information storage unit 327, either at the request of the user or according to pre-set instructions. The method of inputting a request for medical document generation by the user is not limited; for example, the request may be entered via the mobile terminal 10 or from another terminal. Also, when medical documents are generated automatically, the user or the administrator of the medical support system 1 may appropriately set the medical documents to be automatically generated. The medical document generation unit 336 generates medical documents from the recorded information based on the format of the medical documents stored in the format storage unit 328. Specifically, the medical document generation unit 336 generates medical documents by referring to the format of the medical documents and writing the corresponding content contained in the recorded information into each blank item. For example, a numerical value indicating blood pressure in the recorded information will be automatically transferred to the location in the medical document where the numerical value of blood pressure is entered. Here, medical documents include, for example, electronic medical records, medical claim forms (medical fee statements), prescriptions, and emergency medical service activity reports. Furthermore, various AI generation tools may be used to generate medical documents.

[0102] Furthermore, the medical document generation unit 336 stores the generated medical documents in the medical document storage unit 329 and transmits them to the hospital management server 50 via the NW communication unit 31. The hospital management server 50 manages electronic medical records, claims (medical fee statements), and prescriptions, and outputs claims (medical fee statements) and prescriptions to patients.

[0103] The learning processing unit 337 performs various machine learning processes. Specifically, the learning processing unit 337 generates speech recognition models, image recognition models, models for detecting anomalies in medical practitioners' procedures and medication administration, and proposed models related to medical procedures by performing machine learning using past recorded information as training data. The learning processing unit 337 stores the various models, which are the results of the learning process, in the learning result storage unit 323. Note that, in addition to recorded information, various types of information related to past medical procedures may be used as training data for machine learning.

[0104] Next, the operation of the medical support system 1 according to this embodiment will be described with reference to the drawings. Figure 9 shows an example of the collection and recording process of record information in the medical support system 1 according to this embodiment.

[0105] As shown in Figure 9, the mobile terminal 10 of the medical support system 1 first detects sound and images (step S101). The data collection processing unit 191 of the mobile terminal 10 collects sound data and image data from the mobile terminal 10, the headset device 20, and the external imaging device 25.

[0106] Next, the data collection processing unit 191 transmits the sound data and image data to the information collection server 30 (step S102). The data collection processing unit 191 transmits the sound data and image data via the NW communication unit 11 and requests speech recognition processing and image recognition processing.

[0107] Next, the information collection server 30 performs speech recognition processing and image recognition processing (step S103). The speech recognition processing unit 331 of the information collection server 30 converts the received sound data (voice data) into text data. The image recognition processing unit 332 of the information collection server 30 extracts hands-free operations contained in the received image data.

[0108] Next, the information collection server 30 transmits the recognition information, which is the processing result of the speech recognition processing and image recognition processing, to the mobile terminal 10 (step S104). The server control unit 33 of the information collection server 30 transmits the recognition information (text data, hands-free operation, etc.) to the mobile terminal 10 via the NW communication unit 31.

[0109] Next, the data collection processing unit 191 determines whether or not to start recording (step S105). Based on the setting information in the setting information storage unit 181, the data collection processing unit 191 determines whether or not the received recognition information (text data, hands-free operation, etc.) is a start operation. If it is a start operation (step S105: YES), the process proceeds to step S106. If it is not a start operation (step S105: NO), the data collection processing unit 191 returns to step S101.

[0110] In step S106, the data collection processing unit 191 starts the data collection process and transmits the collected information (raw data) and date and time information acquired from, for example, the mobile terminal 10, the headset device 20, and the external imaging device 25 to the information collection server 30. Here, the collected information is, for example, sound data and image data.

[0111] Next, the recording processing unit 333 of the information collection server 30 stores the collected information (step S107). The recording processing unit 333 associates the sound data with the date and time information and stores it in the collected sound information storage unit 324 (see, for example, Figure 4), and also associates the image data with the date and time information and stores it in the collected image information storage unit 325 (see, for example, Figure 5).

[0112] Next, the recording processing unit 333 extracts keywords from the collected information (step S108). The recording processing unit 333 extracts keywords contained in the sound data by, for example, comparing the text data converted from the sound data by the speech recognition processing unit 331 with whether the keyword information storage unit 326 contains pre-set keywords.

[0113] Next, the recording processing unit 333 determines whether or not a keyword exists (whether or not it has been extracted) (step S109). If a keyword exists (step S109: YES), the recording processing unit 333 proceeds to step S110. If a keyword does not exist (step S109: NO), the recording processing unit 333 proceeds to step S113.

[0114] In step S110, the recording processing unit 333 records medical procedure information based on keywords as recording information. Specifically, the recording processing unit 333 uses the information stored in the collected sound information storage unit 324 and the collected image information storage unit 325 in step S107 to store in the recording information storage unit 327 information relating the keywords extracted in step S109 to the date and time information of when the keywords were spoken, and information relating the image data collected by the collection processing unit 191 to the date and time information of when the image data was acquired, as recording information. The recording processing unit 333 stores the recording information in the recording information storage unit 327 in a format such as that shown in Figure 6.

[0115] Next, the recording processing unit 333 transmits the recording information to the mobile terminal 10 (step S111). The recording processing unit 333 transmits the recording information stored in the recording information storage unit 327 to a terminal that outputs recording information, such as the mobile terminal 10. The recording processing unit 333 may also generate timeline display information based on the recording information and transmit the timeline display information to the mobile terminal 10 or other terminal.

[0116] The output control unit 192 of the mobile terminal 10 displays the recorded information as a timeline (step S112). For example, as shown in Figure 7, the output control unit 192 causes the display unit 13 of the mobile terminal 10 to display a timeline based on the recorded information.

[0117] Next, in step S113, the recording processing unit 333 determines whether or not recording has ended. The recording processing unit 333 uses the speech recognition processing unit 331 and the image recognition processing unit 332 to determine whether or not an end operation has been detected by hands-free operation. If an end operation is detected and recording has ended (step S113: YES), the recording processing unit 333 proceeds to step S114. If an end operation is not detected and recording has not ended (step S113: NO), the recording processing unit 333 returns to step S107.

[0118] In step S114, the recording processing unit 333 sends a completion notification to the mobile terminal 10.

[0119] Next, the data collection processing unit 191 determines whether or not recording has ended (step S115). The data collection processing unit 191 determines whether or not recording has ended (data collection has ended) by, for example, receiving a completion notification. If recording has ended (step S115: YES), the data collection processing unit 191 returns to step S101. If recording has not ended (step S115: NO), the data collection processing unit 191 returns to step S106. In step S110, the recording processing unit 333 may further include text data extracted from the image data by the image recognition processing unit 332 to create the recording information.

[0120] Next, with reference to Figure 10, the support processing of the medical support system 1 according to this embodiment will be described. Figure 10 shows an example of the support processing of the medical support system 1 according to this embodiment. Here, we will explain an example in which a user makes a support request by voice.

[0121] As shown in Figure 10, first, the mobile terminal 10 transmits sound data to the information collection server 30 (step S201). The terminal control unit 19 of the mobile terminal 10 acquires sound data detected by, for example, the microphone 14 of the mobile terminal 10 and the microphone 22 of the headset device 20, and transmits it to the information collection server 30 via the NW communication unit 11.

[0122] Next, the speech recognition processing unit 331 of the information collection server 30 performs speech recognition processing (step S202). The speech recognition processing unit 331 converts the sound data into text data.

[0123] Next, the support processing unit 334 determines whether or not there is a support request (step S203). The support processing unit 334 determines whether or not there is a support request, such as information retrieval, for the text data converted from the sound data. If there is a support request (step S203: YES), the support processing unit 334 proceeds to step S204. If there is no support request (step S203: NO), the support processing unit 334 returns to step S202.

[0124] In step S204, the support processing unit 334 executes processing in response to the support request. Processing in response to the support request here includes, for example, searching for information on medical procedures, making suggestions regarding medical procedures, and requesting assistance from other users.

[0125] Next, the support processing unit 334 transmits the support processing result to the mobile terminal 10 (step S205).

[0126] Next, the output control unit 192 of the mobile terminal 10 displays the support processing result on the display unit 13 or outputs it as sound (step S206). When outputting the support processing result as sound, the output control unit 192 synthesizes the support processing result into speech and outputs it as sound from the speaker 15. Note that the user's support request does not have to be input from the mobile terminal 10, but may be input from another terminal. Also, the method of inputting the user's support request is not limited, and for example, voice input, text input, gesture input, etc. may be used. Furthermore, the result of the support processing by the support processing unit 334 does not have to be output to the mobile terminal 10, but may be output to the user via another terminal.

[0127] Next, with reference to Figure 11, the abnormality warning process of the medical support system 1 according to this embodiment will be described. Figure 11 is a flowchart illustrating an example of the abnormal warning processing of the medical support system 1 according to this embodiment. Here, we describe the case where a warning is issued via the mobile terminal 10, but the warning does not necessarily have to be output from the display unit 13 or speaker 15 of the mobile terminal 10. It may also be issued in the form of a display or sound from other devices installed in the medical setting (treatment room, operating room, etc.).

[0128] As shown in Figure 11, the recording processing unit 333 of the medical support system 1 acquires medical procedure information (step S211). The recording processing unit 333 acquires, for example, medical procedure information (for example, the content of the medical procedure) from the recording information stored in the recording information storage unit 327.

[0129] Next, the recording processing unit 333 detects anomalies in the medical procedure information (step S212). For example, the recording processing unit 333 refers to the medical procedure information and detects anomalies according to pre-set rules for detecting anomalies (for example, whether the content of text data indicating a drug order in the voice of a medical professional extracted from sound data matches the content of text data indicating the type, amount, and flow rate of the drug actually used extracted from the screen data of the drug dispensing device). Alternatively, for example, the recording processing unit 333 may detect anomalies in the medical procedure information using an anomaly detection model, which is a learning result stored in the learning result storage unit 323.

[0130] Next, the recording processing unit 333 determines whether or not an abnormality has been detected (step S213). If an abnormality has been detected (step S213: YES), the recording processing unit 333 proceeds to step S214. If no abnormality has been detected (step S213: NO), the recording processing unit 333 returns to step S211.

[0131] In step S214, the recording processing unit 333 causes the mobile terminal 10 to output a warning based on an abnormality via the output control unit 192. After processing in step S214, the recording processing unit 333 returns to processing in step S211.

[0132] Next, with reference to Figure 12, the editing and processing of recorded information in the medical support system 1 according to this embodiment will be described. Figure 12 shows an example of the editing and processing of recorded information in the medical support system 1 according to this embodiment. Here, we will describe the case in which an editing request is input to the management terminal 40 and the results of the editing process are output to the management terminal 40. However, the terminal to which the editing request is input and the terminal to which the results of the editing process are output are not limited to the management terminal 40, but may be other terminals.

[0133] As shown in Figure 12, first, the terminal control unit 45 of the management terminal 40 sends a request to view the recorded information to the information collection server 30 (step S301). The terminal control unit 45 sends a request to view the recorded information to the information collection server 30 via the NW communication unit 41 in response to the operation of the input unit 42 by the user (administrator).

[0134] Next, the editing processing unit 335 of the information collection server 30 generates a list of recorded information (step S302). The editing processing unit 335 generates a list of recorded information, for example, like the display screen G2 shown in Figure 13. In the example shown in Figure 13, information indicating the summary of the recorded information (date, patient name, attending physician, and symptoms) is summarized on one line, and the list of recorded information is constructed by displaying the information indicating the summary of each piece of recorded information side by side.

[0135] Next, the editing processing unit 335 sends a list of recorded information to the management terminal 40 (step S303). The editing processing unit 335 sends the generated list of recorded information to the management terminal 40 via the NW communication unit 31.

[0136] Next, the display control unit 451 of the management terminal 40 displays a list of recorded information on the display unit 43 (step S304). The display control unit 451 displays a list of recorded information on the display unit 43, for example, as shown in display screen G2 in Figure 13.

[0137] Next, the terminal control unit 45 sends a request to the information collection server 30 to view the details of the recorded information in response to the user's (administrator's) operation of the input unit 42 (step S305). For example, when a user selects information showing an overview of the recorded information (any "row") on the display screen G2 shown in Figure 13 using a cursor, touch operation, etc., a request to view the details of the selected recorded information is sent to the information collection server 30.

[0138] Next, the editing processing unit 335 transmits the detailed record information to the management terminal 40 in response to the request to view the details of the record information (step S306).

[0139] Next, the display control unit 451 displays detailed information of the corresponding recording information on the display unit 43 (step S307). The detailed information of the recording information may be displayed as a timeline display of the recording information, for example, as shown in screen G1 in Figure 7.

[0140] Next, the terminal control unit 45 sends an editing / processing request to the information collection server 30 in response to the user's (administrator's) operation of the input unit 42 (step S308). The user can then input instructions, such as modifying keywords on the timeline display, while referring to the screen of the display unit 43.

[0141] Next, the editing processing unit 335 executes processing in response to the editing and processing request (step S309). The editing processing unit 335 stores the recorded information after executing processing in response to the editing and processing request in the recorded information storage unit 327 and updates the recorded information.

[0142] Next, the editing processing unit 335 transmits the editing and processing results, i.e., the updated record information, to the management terminal 40 (step S310).

[0143] Next, the display control unit 451 displays the editing and processing results, i.e., the updated recorded information, on the display unit 43 (step S311).

[0144] Figure 14 shows an example of a summary text screen of the recorded information of the medical support system 1 according to this embodiment. The editing processing unit 335 may automatically edit the recorded information or based on user input to generate a summary text, such as the display screen G3 shown in Figure 14, instead of a timeline display, and display it on the display unit 43 via the display control unit 451. Alternatively, the editing processing unit 335 may automatically edit the recorded information or based on user input to generate a list (archive information) including, for example, the date, attending physician, attending nurse, and symptoms, and display it on the display unit 43 of the management terminal 40 via the display control unit 451.

[0145] Next, with reference to Figure 15, the medical document generation process of the medical support system 1 according to this embodiment will be described. Figure 15 is a flowchart showing an example of the medical document generation process of the medical support system 1 according to this embodiment.

[0146] As shown in Figure 15, the medical document generation unit 336 of the information collection server 30 first determines whether or not it has received a request to create a medical document (step S321). The request to create a medical document is entered by a user, for example, via the management terminal 40. However, the means of inputting the creation request is not limited to this, and any terminal may be used to input the creation request. If the medical document generation unit 336 has received a request to create a medical document (step S321: YES), it proceeds to step S322. If the medical document generation unit 336 has not received a request to create a medical document (step S321: NO), it returns to step S321.

[0147] In step S322, the medical document generation unit 336 generates a medical document from the recorded information in response to a creation request. The medical document generation unit 336 generates a medical document from the recorded information stored in the recorded information storage unit 327 based on the format of the medical document stored in the format storage unit 328.

[0148] Next, the medical document generation unit 336 stores the generated medical documents in the medical document storage unit 329 (step S323).

[0149] Next, the medical document generation unit 336 sends the medical documents to the hospital management server 50 (step S324). The hospital management server 50 manages medical documents such as electronic medical records, claims (medical fee statements), and prescriptions, and also outputs claims (medical fee statements) and prescriptions to patients. After processing in step S324, the medical document generation unit 336 terminates its processing.

[0150] As described above, the medical support system 1 (support system) according to this embodiment comprises a data collection processing unit 191, a recording processing unit 333, and an output control unit 192. The data collection processing unit 191 collects sound data (sound information) and image data (image information) associated with medical procedures performed by a user (for example, a medical professional). The recording processing unit 333 extracts a first keyword contained in the sound data collected by the data collection processing unit 191, and stores information relating the first keyword to the date and time information in which the first keyword was spoken, and information relating the image data collected by the data collection processing unit 191 to the date and time information in which the image data was acquired, as recorded information in the recorded information storage unit 327. The output control unit 192 displays a timeline display on the display unit 13, which displays the keyword and image data in chronological order based on the recorded information stored in the recorded information storage unit 327.

[0151] As a result, the medical support system 1 according to this embodiment associates keywords, image data, and date and time information and stores them as recorded information in the recorded information storage unit 327, enabling the automatic collection and retention of more accurate records of medical procedures. Therefore, the medical support system 1 according to this embodiment can further reduce the burden on medical personnel and improve work efficiency and accuracy. In addition, the medical support system 1 according to this embodiment can improve convenience. In particular, in emergency medical settings, treatment is performed in a time-sensitive situation, making it difficult to quickly and accurately take notes of the details of medical procedures manually, and there has been a need for technology that can automatically record medical procedures. According to the medical support system 1 of this embodiment, the details of medical procedures are automatically and accurately recorded based on the user's voice, the screen of medical equipment, etc., so it can improve user convenience, especially when applied to emergency medical settings.

[0152] In the aforementioned Patent Document 1, a technology is disclosed for recognizing medical procedures performed during surgery as events and creating a display screen that arranges the recognized events in chronological order. However, the display screen in Patent Document 1 shows text data indicating the events and links to related image data. For example, if a user wants to view an image, they must select the link and view the image on a separate screen. In contrast, this embodiment stores keywords related to medical procedures, image data, and date / time information as recorded information, and displays this recorded information in chronological order as a timeline. In other words, since the image data itself is displayed chronologically along with the text data related to the medical procedures, users can quickly see and understand the flow of medical procedures by viewing the display. Thus, the medical support system 1 according to this embodiment can display the content of medical procedures in a more easily understandable format, further improving user convenience.

[0153] In this embodiment, the collection processing unit 191 starts collecting sound data or image data when it detects a start operation, which is a hands-free operation performed by the user to initiate the collection of sound data or image data. The collection processing unit 191 also stops collecting sound data or image data when it detects an end operation, which is a hands-free operation to terminate the collection of sound data or image data.

[0154] As a result, the medical support system 1 according to this embodiment allows users to collect sound data or image data through contactless operation, enabling users to perform medical procedures and record them while maintaining hygiene. Therefore, the medical support system 1 according to this embodiment can further reduce the burden on medical personnel and improve convenience.

[0155] Furthermore, the medical support system 1 according to this embodiment includes a microphone 22 (sound pickup unit) and an imaging unit 23. The microphone 22 (sound pickup unit) picks up sounds associated with medical procedures and outputs sound data. The imaging unit 23 captures images associated with medical procedures and outputs image data. Hands-free operation includes at least one of the following: operation by the user's voice detected based on the sound data output by the microphone 22; operation by the user's motion detected based on the image data output by the imaging unit 23; and operation by the user's gaze detected by a headset (headset device 20) equipped with the imaging unit 23.

[0156] As a result, the medical support system 1 according to this embodiment can record medical procedures, etc., through voice control, motion control, and eye-tracking control, thereby further reducing the burden on medical professionals and improving convenience.

[0157] In this embodiment, the output control unit 192 adds and displays text based on the user's voice at a specified position in the image data displayed by the display unit 13.

[0158] As a result, the medical support system 1 according to this embodiment can add text to image data based on the user's voice, allowing for clearer and more accurate recording of information.

[0159] In this embodiment, the recording processing unit 333 extracts a second keyword, which is a keyword included in the image data collected by the collection processing unit 191, and stores the information relating the image data, the date and time information (e.g., time information) on which the image data was acquired, and the second keyword, as recording information in the recording information storage unit 327. Based on the recording information, the output control unit 192 displays the image data, the date and time information on which the image data was acquired, and the second keyword in association with each other on the timeline display in the display unit 13.

[0160] As a result, the medical support system 1 according to this embodiment displays the second keyword extracted from the image data in association with the image data on the display unit 13, making it possible to record medical procedures in a clearer and more accurate manner.

[0161] In this embodiment, the recording processing unit 333 stores the recording information created for each period during which sound data and image data were collected in the recording information storage unit 327. The output control unit 192 displays a timeline on the display unit 13 for each period.

[0162] As a result, in the medical support system 1 according to this embodiment, record information is recorded for each period, allowing for efficient recording of record information. Furthermore, in the medical support system 1 according to this embodiment, the record information can be reviewed for each period using a timeline display, thereby improving work efficiency.

[0163] Furthermore, in this embodiment, the first keyword and the second keyword include at least one of the following: terms related to treatment, terms related to jigs used in treatment, terms related to drugs, and terms related to the patient's biological information. As a result, the medical support system 1 according to this embodiment can accurately record information about medical procedures.

[0164] Furthermore, in this embodiment, the users include multiple medical professionals. Based on the sound data, the recording processing unit 333 identifies the speaker who uttered the first keyword among the multiple medical professionals, and stores the information as recording information in the recording information storage unit 327, which associates identification information (e.g., user ID) that identifies the speaker, the first keyword, and date and time information.

[0165] As a result, the medical support system 1 according to this embodiment can accurately and easily record information on medical procedures performed by multiple medical professionals, thereby further reducing the burden on medical professionals and improving work efficiency and accuracy.

[0166] In this embodiment, the recording processing unit 333 outputs a warning when it detects an abnormality in a medical procedure based on the recording information stored in the recording information storage unit 327. As a result, the medical support system 1 according to this embodiment can further improve the efficiency and accuracy of medical procedures.

[0167] Furthermore, the medical support system 1 according to this embodiment includes a support processing unit 334. The support processing unit 334 executes support processing in response to a user's request for support regarding medical procedures and outputs the support processing results to the user.

[0168] As a result, in the medical support system 1 according to this embodiment, users can receive appropriate support during medical procedures, thereby further improving the efficiency and accuracy of medical procedures.

[0169] Furthermore, in this embodiment, the support processing unit 334 generates suggestions regarding medical procedures based on past recorded information and outputs the generated suggestions regarding medical procedures to the user as a support processing result.

[0170] As a result, the medical support system 1 according to this embodiment can further improve the efficiency and accuracy of medical procedures because users can receive appropriate suggestions based on past performance during medical procedures.

[0171] Furthermore, the medical support system 1 according to this embodiment includes an editing processing unit 335. The editing processing unit 335 edits the recorded information stored in the recorded information storage unit 327 and outputs the edited recorded information to the user.

[0172] As a result, the medical support system 1 according to this embodiment can later review the recorded information and, for example, appropriately correct any erroneous records, thereby further improving operational efficiency and accuracy.

[0173] Furthermore, the medical support system 1 according to this embodiment includes a medical document generation unit 336. The medical document generation unit 336 generates medical documents (for example, electronic medical records, medical claim forms (medical fee statements), prescriptions, etc.) in a preset format based on the recorded information stored in the recorded information storage unit 327.

[0174] As a result, the medical support system 1 according to this embodiment automatically generates medical documents (e.g., electronic medical records, claims (medical fee statements), prescriptions, etc.) from recorded information, further reducing the burden on medical professionals and further improving work efficiency and accuracy.

[0175] Furthermore, in this embodiment, the display unit that arranges and displays keywords and image data in chronological order is the display unit 13 of the mobile terminal 10 carried by the user. As a result, in the medical support system 1 according to this embodiment, the user can appropriately check the recorded information on the mobile terminal 10 while performing medical procedures.

[0176] Furthermore, the support method according to this embodiment includes a data collection step, a recording step, and an output control step. In the data collection step, the data collection processing unit 191 collects sound data and image data associated with medical procedures performed by the user. In the recording step, the recording processing unit 333 extracts a first keyword, which is a keyword contained in the sound data collected by the data collection processing unit 191, and stores in the recording information storage unit 327 information relating the first keyword to the date and time information in which the first keyword was spoken, and information relating the image data collected by the data collection processing unit 191 to the date and time information in which the image data was acquired, as recording information. In the output control step, the output control unit 192 displays a timeline display on the display unit 13, which displays the keyword and image data arranged in chronological order based on the recording information stored in the recording information storage unit 327.

[0177] As a result, the support method according to this embodiment has the same effects as the medical support system 1 described above, further reducing the burden on medical professionals and improving work efficiency and accuracy.

[0178] Furthermore, although the above-described embodiment describes an example in which the mobile terminal 10 comprises a data collection processing unit 191 and an output control unit 192, it is not limited to this, and the information collection server 30 (an example of a server device) may also comprise a data collection processing unit 191 and an output control unit 192. In other words, the server device (information collection server 30) according to this embodiment may comprise the above-described data collection processing unit 191 and a recording processing unit 333 and an output control unit 192. As a result, the server device (information collection server 30) according to this embodiment has the same effects as the medical support system 1 described above, further reducing the burden on medical personnel and improving work efficiency and accuracy.

[0179] (Second Embodiment) Next, with reference to the drawings, a medical support system 1a according to a second embodiment will be described.

[0180] In this embodiment, in addition to the display unit 13 of the mobile terminal 10, a modified example will be described in which a timeline display based on recorded information is performed on an external display device 55 installed in a medical setting such as an operating room.

[0181] Figure 16 is a functional block diagram showing an example of a medical support system 1a according to the second embodiment. As shown in Figure 16, the medical support system 1a comprises a mobile terminal 10, a headset device 20, an external imaging device 25, an information collection server 30a, a management terminal 40, a hospital management server 50, and an external display device 55.

[0182] In Figure 16, components identical to those in the first embodiment shown in Figure 1 are given the same reference numerals, and their descriptions are omitted. Furthermore, medical support system 1a is an example of a support system.

[0183] The external display device 55 is, for example, a large-screen display device that is permanently installed in an operating room or the like. The external display device 55 is, for example, a liquid crystal display, and is an example of a display unit. Here, the display unit includes the external display device 55 (an example of a display unit) installed in a place where medical procedures are performed (for example, an operating room). Furthermore, the external display device 55 includes a display control unit 551.

[0184] The display control unit 551 displays various information on the external display device 55 in response to instructions from the recording processing unit 333a of the information collection server 30a, which will be described later. The display control unit 551 displays a timeline based on the recorded information, similar to the display unit 13 of the mobile terminal 10. In this embodiment, the timeline display is performed in real time while the medical procedure is being carried out, and the timeline display is updated each time a procedure is performed (more specifically, each time sound data and image data related to the procedure are collected and the recorded information is updated) (i.e., new information related to the procedure is added to the timeline as it occurs). In other words, the recording processing unit 333a updates the timeline display to the latest information via the display control unit 551, that is, updates various information related to the procedure (for example, the patient's biological information, information related to symptoms, or information related to medication) to the latest information and displays it on the external display device 55. The display control unit 551 also displays supplementary information associated with the recorded information on the same screen as the timeline display. Supplementary information includes, for example, the patient's basic information, information related to the patient's symptoms, or information related to medication. Details of the display on the external display device 55 will be described later.

[0185] The information collection server 30a is a server device that records various information (recorded information) related to medical procedures performed by medical professionals and performs support processing to reduce the burden on medical professionals, such as displaying information based on the recorded information on the display unit 13 of the mobile terminal 10 and the external display device 55. The information collection server 30a can communicate with the mobile terminal 10, the management terminal 40, the hospital management server 50, and the external display device 55 via the network NW1.

[0186] The information gathering server 30a comprises a network communication unit 31, a server storage unit 32, and a server control unit 33a. The server control unit 33a is a functional unit that is realized, for example, by causing a processor (not shown) including a CPU to execute a program stored in the server storage unit 32. The server control unit 33a comprehensively controls the information collection server 30a and executes various processes. The server control unit 33a includes a speech recognition processing unit 331, an image recognition processing unit 332, a recording processing unit 333a, a support processing unit 334, an editing processing unit 335, a medical document generation unit 336, and a learning processing unit 337.

[0187] The recording processing unit 333a has the same functions as the recording processing unit 333 in the first embodiment described above. Furthermore, the recording processing unit 333a, via the display control unit 551 of the external display device 55, causes the external display device 55 to display a timeline display in which keywords and image data are arranged in chronological order. At that time, the recording processing unit 333a, via the display control unit 551, updates the timeline display to the latest information based on the recording information that is updated as it is received, and displays it on the external display device 55. In addition, the recording processing unit 333a, via the display control unit 551, displays supplementary information of the recording information on the same screen as the timeline display of the recording information. Supplementary information may include, for example, the patient's basic information (patient's name, date of birth, age, height, weight, blood type, symptoms, and biological information (vital information)), information about the patient's symptoms, and information about medication. The recording processing unit 333a, via the display control unit 551, updates the supplementary information in accordance with the update of the timeline display.

[0188] Now, with reference to Figures 17 and 18, an example of the display of the external display device 55 in this embodiment will be described. Figure 17 shows an example of the operation of the medical support system 1a according to this embodiment.

[0189] As shown in Figure 17, in this embodiment, the display control unit 551 displays a timeline of recorded information in area SG1 (first area) on the screen of the external display device 55, and displays the patient's basic information in area SG2 (second area). Here, basic information is an example of supplementary information, such as the patient's name, height, weight, blood type, symptoms, and vital signs. In addition, vital signs include, for example, heart rate, blood pressure, body temperature, respiratory rate, etc.

[0190] Furthermore, when the timeline display in area SG1 (first area) is updated and additional basic patient information is displayed, the recording processing unit 333a instructs the display control unit 551 to display the newly displayed basic information values ​​so that they can be transferred to area SG2 (second area) as the latest basic information.

[0191] In other words, in this embodiment, the recording processing unit 333a, via the display control unit 551 of the external display device 55, causes the external display device 55 to display a timeline display in which keywords and image data are arranged in chronological order, and also causes the external display device 55 to display the patient's latest biological information (heart rate, blood pressure, body temperature, respiratory rate, etc.) based on the recorded information.

[0192] Figure 18 shows another example of the operation of the medical support system 1a according to this embodiment. Here, we will describe a modified version of the information displayed on the external display device 55.

[0193] As shown in Figure 18, in this modified example, the display control unit 551 displays a timeline of recorded information in area SG1 (first area) on the screen of the external display device 55, and displays information about the patient's symptoms in area SG3 (third area). Here, the information about the patient's symptoms is an example of supplementary information and includes, for example, the name of the symptom and an image PH3 of the affected area.

[0194] Furthermore, the external display device 55 displays information about medication administration extracted from the recorded information in area SG4 (fourth area). Here, the information about medication administration is an example of supplementary information and includes, for example, the name and amount of the administered drug, and an image PH4 of the label of the administered drug pack.

[0195] In this modified example, the recording processing unit 333a, via the display control unit 551 of the external display device 55, causes the external display device 55 to display a timeline display in which keywords and image data are arranged in chronological order, and also causes the external display device 55 to display information about the patient's symptoms and information about medication based on the recorded information.

[0196] In this case, the recording processing unit 333a instructs the display control unit 551 to transfer the newly displayed information to area SG3 (third area) as the latest information regarding the patient's symptoms when the timeline display in area SG1 (first area) is updated and additional information regarding medication is displayed. Furthermore, the recording processing unit 333a instructs the display control unit 551 to transfer the newly displayed information to area SG4 (fourth area) as the latest information regarding medication when the timeline display in area SG1 (first area) is updated and additional information regarding medication is displayed.

[0197] Furthermore, the type and content of supplementary information displayed on the same screen as the timeline display may be arbitrarily set by the user. For example, by setting items that the user wants to check during a medical procedure as supplementary information, the user can perform the procedure while checking the record information (timeline display) and supplementary information that are updated in real time. Also, if the user wants to extract some information from the record information and save it separately, by setting the information to be extracted as supplementary information, the desired information will be automatically extracted, saving the user the trouble of manually extracting the desired information from the record information. For example, information contained in any medical document may be set as supplementary information. In this case, when the supplementary information is displayed on the same screen as the timeline display, the format of the medical document specified by the user may be displayed, and the display of the supplementary information may be controlled so that the corresponding information from the timeline display is automatically transferred to the blank items in that medical document. This makes it possible for the user to check the content of the medical document that is ultimately scheduled to be created by the medical document generation unit 336 based on the record information in advance at the site.

[0198] As described above, in this embodiment, the display unit that outputs the recorded information is an external display device 55 (display device) installed at the location where medical procedures are performed. The recording processing unit 333a, via the display control unit 551 (an example of an output control unit), displays a timeline display in which keywords and image information are arranged in chronological order, updating it as needed (updating it to the latest information). The recording processing unit 333a also, via the display control unit 551, displays supplementary information (for example, the latest basic information of the patient (e.g., biological information), information on the latest symptoms, or information on medication) on the same screen as the timeline display of the recorded information, updating it as needed.

[0199] As a result, the medical support system 1a according to this embodiment allows for the display of a timeline of continuously updated record information and enables treatment while checking accompanying information, thereby improving user convenience. Furthermore, by configuring the system so that accompanying information is displayed by automatically transcribing a portion of the record information, the effort required for users to manually transcribe the information is eliminated. Thus, this embodiment further reduces the burden on medical professionals and improves work efficiency and accuracy.

[0200] The display in this embodiment does not necessarily have to be displayed on the external display device 55, but may be displayed on other display devices. For example, the displays shown in Figures 17 and 18 may be displayed on the display unit 13 of the mobile terminal 10.

[0201] Furthermore, the information displayed as supplementary information may also be editable by the editing processing unit 335 of the information collection server 30a, similar to the recorded information in the first embodiment. This makes it possible to accurately correct any errors in the information extracted from the recorded information as supplementary information. In addition, the system may be configured so that when the recorded information is edited by the editing processing unit 335, the corresponding description in the supplementary information is automatically corrected. That is, editing (changing) the recorded information may be automatically reflected in the supplementary information. This makes it possible to correct errors in both the recorded information and the supplementary information created by extracting from the recorded information in a single editing process, thereby improving work efficiency.

[0202] Furthermore, in the first and second embodiments described above, the timeline display shown in Figure 7 may be shared in real time to any network inside or outside the hospital. In this case, external specialists (e.g., doctors from other departments or outside the hospital) may add comments to the shared recorded information (especially image data). Comments from the AI ​​diagnosis may also be added.

[0203] Furthermore, the timeline display and the added comments may be displayed on a single screen. In this case, the display control unit 551 (output control unit 192) may change the font and color of the specialist and AI diagnosis comments to make them easier to understand.

[0204] As a result, the medical support system 1a according to this embodiment can make more effective use of recorded information by utilizing external specialists and AI diagnosis, and can also improve the accuracy of operations (accuracy of diagnosis and treatment).

[0205] Furthermore, in the timeline display shown in Figure 7, the display control unit 551(192) may change the background color of time periods when there are rapid fluctuations in the data. Also, the recording processing unit 333a(333) may automatically record the start and end times of rapid fluctuations. Here, "rapid fluctuations" could refer to, for example, rapid fluctuations in biological data (heart rate, blood pressure).

[0206] As a result, the medical support system 1a(1) according to this embodiment can clearly display the recording portion where a noteworthy abnormality may have occurred, allowing users to quickly access the desired recording information when reviewing the recording information at a later date.

[0207] (Third embodiment) Next, with reference to the drawings, a medical support system 1b according to a third embodiment will be described.

[0208] This embodiment relates to a method for more effectively utilizing the recorded information recorded in the first and second embodiments. This embodiment aims to effectively utilize medical procedure information (for example, date and time information, information about the patient's symptoms, information about the content of the treatment, information about the duration of the treatment, and progress information showing the progress after the treatment) collected at a medical institution. As an example, the case in which the medical procedure information is the recorded information recorded in the first and second embodiments will be described below, but the medical procedure information handled in this embodiment is not limited to this example and may be any information that indicates the content of a medical procedure. Figure 19 is a functional block diagram showing an example of a medical support system 1b according to the third embodiment.

[0209] As shown in Figure 19, the medical support system 1b comprises a mobile terminal 10, a headset device 20, an external imaging device 25, an information collection server 30, a management terminal 40, a hospital management server 50, and an information support server 60. In this embodiment, the mobile terminal 10, the external imaging device 25, the information collection server 30, the management terminal 40, the hospital management server 50, and the information support server 60 are connected to a network NW1 and can communicate with each other. Note that the medical support system 1b is just one example of a support system.

[0210] In Figure 19, components identical to those in the first embodiment shown in Figure 1 are given the same reference numerals, and their descriptions are omitted. Furthermore, for explanatory purposes, Figure 19 only shows the recording information storage unit 327 among the functions of the information collection server 30 shown in Figure 1.

[0211] The information support server 60 is a server device that performs various processes to make more effective use of the recorded information stored in the recorded information storage unit 327 of the information collection server 30. The information support server 60 comprises an NW communication unit 61, a server storage unit 62, and a server control unit 63.

[0212] The NW communication unit 61 is a communication device that can connect to the network NW1, for example, via a wireless LAN or a wired LAN. The NW communication unit 61 performs data communication with, for example, a mobile terminal 10, a management terminal 40, and an information collection server 30 via the network NW1. The NW communication unit 61 receives, for example, recorded information from the information collection server 30. The NW communication unit 61 also receives various requests from the management terminal 40 for the effective use of the recorded information.

[0213] The server storage unit 62 is a storage unit realized by the memory (not shown) provided by the information support server 60, and stores various types of information used by the information support server 60.

[0214] The server storage unit 62 includes an aggregation information storage unit 621, a learning data storage unit 622, a learning result storage unit 623, a regional information storage unit 624, a prediction result storage unit 625, a personnel allocation information storage unit 626, a labor management information storage unit 627, and an external provision information storage unit 628.

[0215] The aggregated information storage unit 621 stores the aggregated results obtained by the aggregation processing unit 631, described later, from the record information storage unit 327 of the information collection server 30. The aggregated information storage unit 621 stores, as aggregated results, statistical data such as the number of patients and symptoms for each period (e.g., month, day), statistical data between patient information (age, gender) and the time spent on treatment, statistical data between symptoms and the time spent on treatment, statistical data for each medical professional, such as the content of medical procedures and the time spent on treatment, and statistical data showing the relationship between the procedures performed, the type and amount of drugs administered, the tools used, and post-treatment progress information for each symptom. The aggregated information storage unit 621 may also store the time spent on treatment separately from the time spent on treatment in emergency medical care. Furthermore, the aggregated information storage unit 621 may store statistical data on the time taken from the time an emergency transport inquiry is made until the patient is transported.

[0216] In other words, the aggregated results include statistical data on the number of patients and symptoms for each period, statistical data on symptoms and the duration of medical treatment, and statistical data on patient information and the duration of medical treatment.

[0217] The learning data storage unit 622 stores various types of learning data for executing machine learning processing. For example, the learning data storage unit 622 stores learning data that has been processed to generate the target learning model based on recorded information and aggregated results.

[0218] The learning result storage unit 623 stores the learning results obtained when the learning processing unit 632, described later, performs machine learning processing using the learning data stored in the learning data storage unit 622. The learning result storage unit 623 stores various learning models, such as a predictive model for predicting busy periods in medical institutions, a predictive model for predicting the time required for treatment for each symptom, an estimation model for estimating candidate treatments, candidate medications, candidate tools to be used, or treatment timing (including treatment procedures) for each symptom.

[0219] The regional information storage unit 624 stores regional information that includes at least one of the following: information about the region's weather, information about the region's traffic volume, information about events held in the region, information about the region's population, information about schools in the region, information about the region's industries, and information about other medical institutions located in the region. The regional information stored in the regional information storage unit 624 is obtained from an external server via the network NW1. In addition, the weather information includes weather, temperature, humidity, wind speed, etc.

[0220] The prediction result storage unit 625 stores the prediction results predicted by the prediction processing unit 633, which will be described later. The personnel allocation information storage unit 626 stores the personnel allocation information generated by the personnel allocation generation unit 634, which will be described later.

[0221] The labor management information storage unit 627 stores various information for managing the work schedules of healthcare professionals who are users of the system. For example, the labor management information storage unit 627 stores information such as each healthcare professional's work performance evaluation and working hours management information.

[0222] The external information storage unit 628 stores external information generated by the external information processing unit 637, described later, based on the recorded information. External information is information provided from the medical support system 1b to external organizations. The external information stored in the external information storage unit 628 includes, for example, information that can be used as data for academic presentations on cases by providing it to medical associations or other hospitals, information that can be used to calculate medical fees determined according to the difficulty of treatment by providing it to government-related organizations such as the Ministry of Health, Labour and Welfare, and information that can be used as research and development data by providing it to pharmaceutical companies or medical device manufacturers. Information that can be used as research and development data includes, for example, information such as the type and amount of drugs used in treatment and information on the progress after treatment.

[0223] The server control unit 63 is a functional unit that is realized, for example, by causing a processor (not shown) including a CPU to execute a program stored in the server storage unit 62. The server control unit 63 comprehensively controls the information support server 60 and executes various processes. The server control unit 63 comprises an aggregation processing unit 631, a learning processing unit 632, a prediction processing unit 633, a personnel allocation generation unit 634, a labor management processing unit 635, a proposal processing unit 636, and an external provision processing unit 637.

[0224] The aggregation processing unit 631 aggregates the recorded information based on the information collected during medical procedures. Here, the recorded information is the information stored in the recorded information storage unit 327 of the information collection server 30, and includes, for example, patient information indicating the patient, medical professional information indicating the medical professional who performed the medical procedure, and medical procedure information indicating the content of the medical procedure associated with date and time information.

[0225] The aggregation processing unit 631 aggregates, for example, statistical data on the number of patients and symptoms per period (month, day), statistical data on patient information (age, gender) and the time spent on treatment, statistical data on symptoms and the time spent on treatment, and statistical data showing the relationship between the treatment performed, the type and amount of medication administered, the tools used, and the post-treatment progress information for each symptom, based on the recorded information (information showing the content of medical procedures at medical institutions). The aggregation information storage unit 621 may, when aggregating the time spent on treatment, aggregate the time spent on treatment in emergency medical care separately from the time spent on treatment in other normal medical care. This is because in emergency medical care, patients in urgent need are often the target, so the information on the time spent on treatment may be particularly important. The aggregation information storage unit 621 may also aggregate statistical data on the time taken from the time an emergency transport inquiry is made until the patient is transported.

[0226] Furthermore, the aggregation processing unit 631, for example, uses the recorded information to aggregate the content of medical procedures and the duration required for those procedures for each medical professional, thereby generating statistical data on the work of medical professionals.

[0227] Furthermore, the aggregation processing unit 631 may, for example, perform an analysis of the workload of each medical professional based on the time of collection of the recorded information. In addition, the aggregation processing unit 631 may, for example, aggregate the procedure and timing of the procedure for each of the multiple hospitals and multiple doctors.

[0228] Furthermore, the aggregation processing unit 631 may calculate the difficulty of treatment based on the diagnosis result (disease name, etc.) and the time of collection of the recorded information. Also, the aggregation processing unit 631 may aggregate data by associating, for example, medical devices (jigs) with the time of collection of the recorded information (duration) or the collected information (treatment effect: for example, "improvement in blood data, changes in vital signs (blood pressure, pulse, consciousness, respiration, body temperature, oxygen saturation, etc.)").

[0229] Furthermore, the aggregation processing unit 631 may determine the correlation between regional information and specific symptoms by aggregating the recorded information in association with the regional information stored in the regional information storage unit 624. Possible correlations between regional information and specific symptoms include, for example, "when the temperature is below aa degrees and the humidity is below bb%, the number of influenza patients increases by cc% compared to normal times," "when the temperature is above dd degrees and the humidity is above ee%, the number of heatstroke patients increases by ff% compared to normal times," and "when the traffic volume is above gg% compared to normal times, the number of patients due to traffic accidents increases by hh%." The aggregation processing unit 631 stores the aggregation results in the aggregation information storage unit 621.

[0230] The learning processing unit 632 generates learning data corresponding to the target of machine learning based on recorded information or aggregated results, and stores it in the learning data storage unit 622. Furthermore, the learning processing unit 632 generates various learning models using the learning data stored in the learning data storage unit 622. Through machine learning processing, the learning processing unit 632 generates, for example, a predictive model for predicting busy periods in medical institutions, a predictive model for predicting the time required for treatment for each symptom, a predictive model for estimating candidate treatments, candidate medications, candidate jigs to be used, or treatment timing (including treatment procedures) for each symptom.

[0231] Furthermore, when generating these models, the learning processing unit 632 may perform machine learning while also including patient information (age, gender, etc.). Including patient information can improve the prediction (estimation) accuracy of the model. The learning processing unit 632 stores the learning results of the machine learning process, such as a predictive model, in the learning result storage unit 623.

[0232] The prediction processing unit 633 predicts the busy seasons for medical institutions based on the aggregation results compiled by the aggregation processing unit 631. For example, the aggregation processing unit 631 aggregates the number of patients for each symptom based on recorded information, and the prediction processing unit 633 predicts the busy seasons for each medical department based on the aggregated number of patients for each symptom.

[0233] Furthermore, the prediction processing unit 633 predicts busy periods for medical institutions based, for example, on the aggregated results stored in the aggregated information storage unit 621 and the regional information stored in the regional information storage unit 624. Specifically, the prediction processing unit 633 predicts the increase or decrease in the number of patients with a particular symptom in accordance with the current regional information, based on the correlation between the regional information obtained by the aggregated processing unit 631 and that particular symptom.

[0234] Furthermore, for example, the prediction processing unit 633 predicts peak seasons based on information about events held in the region included in the regional information, according to the number of people in the region during the event period (for example, the prediction processing unit 633 predicts that there may be an increase in heatstroke patients if crowded events are held in the summer). Also, for example, the prediction processing unit 633 predicts peak seasons based on information about the region's industries included in the regional information (for example, the prediction processing unit 633 predicts that if a region has many factories, there will be an increase in patients visiting medical institutions due to injuries from accidents within factories).

[0235] Furthermore, for example, the prediction processing unit 633 predicts peak seasons based on information about the population of the region included in the regional information (for example, the prediction processing unit 633 predicts the number of patients with symptoms according to gender, age, etc., taking into account the male-female ratio, age composition, etc., of the regional population (e.g., there is a possibility of an increase in patients with diseases specific to men / women, or an increase in patients with diseases specific to children / elderly)). Furthermore, for example, the prediction processing unit 633 predicts peak seasons based on information about schools within the region included in the regional information, for example, according to the number of elementary schools, junior high schools, high schools, and universities (for example, the prediction processing unit 633 predicts that if there is a large number of elementary schools in a region, there is a possibility of an increase in patients with diseases specific to children).

[0236] Furthermore, for example, the prediction processing unit 633 predicts peak seasons based on information about other medical institutions located in the region included in the regional information (for example, the prediction processing unit 633 predicts the number of patients with specific symptoms who will visit its own medical institution, taking into account the medical departments and size of its own medical institution and the other medical institutions). Furthermore, the prediction processing unit 633 may predict the peak seasons for medical institutions based on the prediction model stored in the learning result storage unit 623. The prediction model here is a prediction model that predicts the peak seasons for medical institutions.

[0237] Furthermore, the prediction processing unit 633 may predict the required treatment time for each symptom based on statistical data of symptoms and the time taken for treatment, which has been compiled by the aggregation processing unit 631. For this prediction, a prediction model that predicts the required treatment time for each symptom may be used, which is the result of machine learning performed by the learning processing unit 632 using symptoms or diagnostic results (such as disease names) included in the recorded information and the required treatment time (based on the time the recorded information was collected) as training data.

[0238] The prediction processing unit 633 may use the information on the time required for treatment for each symptom, as predicted in this way, to predict, for example, the time when an inquiry is made regarding the acceptance of an emergency transport and the number of patients that can be accepted. Specifically, the prediction processing unit 633 receives information via the NW communication unit 61 about the status of medical personnel at the medical institution at the time of the emergency transport (how many medical personnel are working in each department, the status of each person's work at that time (whether they are treating a patient or are free), and if they are treating a patient, when they will be free, etc.), as well as information related to the emergency transport (basic information of the patient involved in the emergency transport (patient's name, height, weight, blood type, symptoms, and biological information (vital information), etc.)). The prediction processing unit 633 receives the information compiled by the aggregation processing unit 631 regarding the time taken for symptoms and treatments. Based on statistical data (more precisely, the estimated time required for treatment for each symptom), information on the status of medical personnel, and information on emergency transport (especially information on the symptoms of patients involved in emergency transport), the system may predict the available time for acceptance of emergency transport and the number of patients that can be accepted, taking into account the availability of medical personnel capable of handling the symptoms of patients involved in emergency transport and the estimated time required for treatment of those symptoms. In this case, the prediction process can be improved by using statistical data on the time spent on treatment in emergency medical care, which has been compiled by the aggregation processing unit 631.

[0239] The prediction processing unit 633 transmits the predicted time and number of patients that can be accepted for emergency transport to any terminal accessible to the user, such as a mobile terminal 10, via the network communication unit 61, and has the terminal output the information to a medical professional. The medical professional can then check the outputted information and respond to inquiries about the acceptance or rejection of emergency transport. In this case, the final decision on acceptance or rejection may be made by considering the time it takes for the patient to arrive, using statistical data compiled by the aggregation processing unit 631, which shows the time elapsed from the time of the emergency transport inquiry until the patient is transported.

[0240] Furthermore, the prediction processing unit 633 may predict the optimal treatment for each symptom (optimal treatment, optimal drug dosage, optimal tool used, and optimal treatment timing (including treatment procedure), etc.) based on statistical data compiled by the aggregation processing unit 631, which shows the relationship between the treatment performed, the type and amount of medication administered, the tool used, and post-treatment progress information for each symptom. For this prediction, an estimation model may be used that estimates candidate treatments, candidate medications, candidate tools used, or candidate treatment timings for each symptom, which is the result of machine learning performed by the learning processing unit 632 using the symptoms or diagnostic results (disease name, etc.) included in the recorded information, the treatment performed, the type and amount of medication administered, the tool used, and post-treatment progress information as training data.

[0241] Furthermore, in the prediction processing using the results of the machine learning described above, the prediction processing unit 633 may also use the learning results obtained by the learning processing unit 632, which includes patient information (age, gender, etc.).

[0242] The prediction processing unit 633 stores the various prediction results described above (busy periods for medical institutions, time required for treatment for each symptom, acceptable times and number of patients for emergency transport, and optimal treatment for each symptom (optimal procedure, optimal amount of medication, optimal tools used, and optimal timing for treatment, etc.) in the prediction result storage unit 625.

[0243] The personnel allocation generation unit 634 generates a personnel allocation of medical personnel capable of handling the busy season of a medical institution, based on the busy season predicted by the prediction processing unit 633. For example, the personnel allocation generation unit 634 generates a personnel allocation plan for medical departments corresponding to a particular symptom, based on the increase or decrease in the number of patients for each symptom predicted by the prediction processing unit 633, so as to be able to handle that increase in the number of patients. For example, a user can send a request to generate a personnel allocation (personnel allocation request) to the personnel allocation generation unit 634 of the information support server 60 from the input unit 42 of the management terminal 40. Personnel allocation requests from users may be entered from any terminal. Upon receiving a personnel allocation request, the personnel allocation generation unit 634 executes the personnel allocation generation process described above.

[0244] The personnel allocation generation unit 634 stores the generated personnel allocation results (personnel allocation proposals) in the personnel allocation information storage unit 626. The personnel allocation generation unit 634 also outputs the generated personnel allocation results to the requesting party (the terminal that input the personnel allocation request). For example, the personnel allocation generation unit 634 transmits the personnel allocation results to the requesting party, the management terminal 40, via the NW communication unit 61.

[0245] The labor management processing unit 635 performs labor management processing for medical personnel. For example, labor management includes evaluating the work content of medical personnel and managing their working hours. The labor management processing unit 635 performs labor management processing based on statistical data regarding the work of medical personnel compiled by the aggregation processing unit 631, for example. Specifically, in evaluating work content, information such as the number of patients treated, the symptoms of the treated patients, the difficulty of the treatment, the time taken for the treatment, and the post-treatment progress, which are included in the record information, are considered. In managing working hours, information such as the time taken for the treatment, which is included in the record information, is considered.

[0246] The labor management processing unit 635 stores the results of labor management in the labor management information storage unit 627. The labor management information stored in the labor management information storage unit 627 may be output via the NW communication unit 61 from any terminal connected to the network NW1 (for example, a mobile terminal 10). However, from the standpoint of confidentiality of information, it is desirable that only authorized persons be able to access the labor management information. For example, access rights can be set so that only the person in charge of labor management or the supervisor of the person subject to labor management can access the labor management information stored in the labor management information storage unit 627.

[0247] The suggestion processing unit 636 proposes treatment options for a new patient based on the predictions made by the prediction processing unit 633 using past recorded information. Specifically, for example, the suggestion processing unit 636 proposes candidate treatments, medications, tools, or treatment timings for a new patient based on the optimal treatment for each symptom predicted by the prediction processing unit 633 (optimal procedure, optimal drug dosage, optimal tool, and optimal treatment timing (including treatment procedure)). This makes it possible for the suggestion processing unit 636 to propose appropriate treatment when a patient with symptoms similar to those included in past recorded information is brought to the emergency room.

[0248] Furthermore, the suggestion processing unit 636 may, in real time, suggest treatment methods for similar past cases while a medical professional is performing treatment on a patient. For example, a user can send a suggestion request (suggestion request) from the input unit 42 of the management terminal 40 to the suggestion processing unit 636 of the information support server 60. Note that the suggestion request by the user may be entered from any terminal. Upon receiving a suggestion request, the suggestion processing unit 636 executes the suggestion processing.

[0249] Furthermore, when the suggestion processing unit 636 proposes a treatment that can be provided to a new patient, it may also suggest candidates for medical professionals who can provide that treatment. For example, the suggestion processing unit 636 may search for a suitable physician and suggest calling them.

[0250] The suggestion processing unit 636 may also make these suggestions using the learning results stored in the learning result storage unit 623. The suggestion processing unit 636 may, for example, make various suggestions based on the results of various prediction processes using the learning results from the prediction processing unit 633 described above. Specifically, the suggestion processing unit 636 may, for example, suggest the optimal treatment for each symptom (optimal treatment, optimal drug dosage, optimal tool, and optimal treatment timing (including treatment procedure), etc.) predicted by the prediction processing unit 633 based on an estimation model that estimates candidate treatments, candidate medications, candidate tools for use, or treatment timing (including treatment procedure) for each symptom.

[0251] Furthermore, the suggestion processing unit 636 transmits these suggestions via the NW communication unit 61 to any terminal that the user can access, such as the mobile terminal 10, and has the terminal output them to the medical professional. Furthermore, the suggestion processing unit 636 may refer to the recorded information and, if there is a discrepancy between the actual treatment currently being performed (for example, the content of the procedure, the timing, the type of jig, the medication administered, etc.) and the prediction made by the prediction processing unit 633 (i.e., the suggestion made by the suggestion processing unit 636), it may output a warning to the mobile terminal 10.

[0252] Whether or not there is a discrepancy may be determined according to predetermined rules. These rules can be arbitrarily set by the user. For example, these rules may be set when the content of the proposed treatment differs from the actual treatment, when the type of drug proposed differs from the type of drug to be used, when the dosage of the drug proposed differs from the actual dosage by more than a predetermined threshold, and when the timing (including procedure) of the proposed treatment / medication differs from the timing (including procedure) of the actual treatment / medication. Warnings only need to be recognizable to the user, and their format, such as display or audio, is arbitrary. Warnings may also be output from any device other than the mobile terminal 10 that the user can access.

[0253] The external provision processing unit 637 performs a pre-configured processing on the recorded information or the aggregated results of the recorded information stored in the aggregated information storage unit 621 to generate information to be provided to external organizations. Here, the information to be provided to external organizations (external provision information) is, for example, information that can be used as data for academic presentations on cases by providing it to medical associations or other hospitals, information that can be used for calculating medical fees determined according to the difficulty of treatment by providing it to government-related organizations such as the Ministry of Health, Labour and Welfare, and information that can be used as research and development data by providing it to pharmaceutical companies or medical device manufacturers.

[0254] In the generation of externally provided information, from the perspective of protecting patient privacy, processing for avoiding individual identification in accordance with relevant laws, regulations, guidelines, etc., such as anonymization / pseudonymization of patient names and masking of faces in image data, may be appropriately performed. For example, the user can send a request (request for externally provided information) to generate externally provided information from the input unit 42 of the management terminal 40 to the externally provided processing unit 637 of the information support server 60. Note that the request for externally provided information by the user may be input from an arbitrary terminal. The request for externally provided information includes information about the destination of the externally provided information. The externally provided processing unit 637 that has received the request for externally provided information executes the above-described generation process of the externally provided information by performing processing in a format corresponding to the destination of the externally provided information.

[0255] The externally provided processing unit 637 stores the provided information (externally provided information) generated for the external institution in the externally provided information storage unit 628. The information stored in the externally provided information storage unit 628 is transmitted to an external institution (not shown) designated by the user via, for example, the NW communication unit 61.

[0256] Next, referring to the drawings, the operation of the medical support system 1b according to the present embodiment will be described. FIG. 20 is a flowchart showing an example of the learning process of the medical support system 1b according to the present embodiment.

[0257] As shown in FIG. 20, the aggregation processing unit 631 of the information support server 60 acquires the record information and executes aggregation processing (step S401). The aggregation processing unit 631 acquires the record information from the information collection server 30 via the NW communication unit 61 and executes various aggregation processes according to the purpose. The aggregation processing unit 631 stores the aggregation result in the aggregation information storage unit 621.

[0258] Next, the learning processing unit 632 of the information support server 60 generates learning data (step S402). The learning processing unit 632 generates learning data using the aggregation result by the aggregation processing unit 631 and stores the learning data in the learning data storage unit 622.

[0259] Next, the learning processing unit 632 executes learning processing based on the learning data (step S403). The learning processing unit 632 executes machine learning processing using the learning data stored in the learning data storage unit 622.

[0260] Next, the learning processing unit 632 stores the learning result in the learning result storage unit 623 (step S404). After the processing of step S404, the learning processing unit 632 ends the learning processing.

[0261] Next, referring to FIG. 21, the personnel allocation process of the medical support system 1b according to the present embodiment will be described. FIG. 21 is a flowchart showing an example of the personnel allocation process of the medical support system 1b according to the present embodiment.

[0262] As shown in FIG. 21, the server control unit 63 of the information support server 60 determines whether a personnel allocation request has been received (step S501). The server control unit 63 determines whether a personnel allocation request has been received from, for example, the management terminal 40 via the NW communication unit 61. When the server control unit 63 receives a personnel allocation request (step S501: YES), the process proceeds to step S502. Also, when the server control unit 63 has not received a personnel allocation request (step S501: NO), the process returns to step S501.

[0263] In step S502, the prediction processing unit 633 of the server control unit 63 predicts the peak period of the medical institution. For example, the prediction processing unit 633 predicts the peak period of the medical institution using the statistical data of the past record information aggregated by the aggregation processing unit 631. Alternatively, the prediction processing unit 633 may predict the peak period of the medical institution using the learning result (prediction model) stored in the learning result storage unit 623. The prediction processing unit 633 stores the prediction result in the prediction result storage unit 625.

[0264] Next, the personnel allocation generation unit 634 of the server control unit 63 generates a personnel allocation plan based on the prediction results (step S503). The personnel allocation generation unit 634 generates a personnel allocation plan that, for example, increases the number of medical personnel during the predicted busy period. The personnel allocation generation unit 634 stores the generated personnel allocation plan in the personnel allocation information storage unit 626.

[0265] Next, the personnel allocation generation unit 634 outputs the personnel allocation plan to the requester (step S504). The personnel allocation generation unit 634 transmits the personnel allocation plan to, for example, the requester, the management terminal 40, via the NW communication unit 61. After processing in step S504, the personnel allocation generation unit 634 terminates the personnel allocation process.

[0266] Next, with reference to Figure 22, the labor management processing of the medical support system 1b according to this embodiment will be described. Figure 22 is a flowchart showing an example of the labor management process of the medical support system 1b according to this embodiment.

[0267] As shown in Figure 22, the aggregation processing unit 631 of the information support server 60 acquires recorded information and performs aggregation processing for each medical professional (step S511). The aggregation processing unit 631 acquires recorded information from the information collection server 30 via the NW communication unit 61 and performs aggregation processing for each medical professional. Specifically, in the aggregation processing for each medical professional, the content of medical procedures and the period required for medical procedures are aggregated for each medical professional based on the recorded information, and statistical data regarding the medical professionals' work is generated. The aggregation processing unit 631 stores the aggregation results in the aggregation information storage unit 621.

[0268] Next, the labor management processing unit 635 of the information support server 60 executes labor management processing for each medical professional based on the aggregated results (step S512). For example, the labor management processing includes evaluating the work content of medical professionals and managing their working hours.

[0269] Next, the labor management processing unit 635 stores information about the results of labor management in the labor management information storage unit 627 (step S513). After the processing in step S513, the labor management processing unit 635 terminates the labor management process.

[0270] Next, with reference to Figure 23, the proposed processing of the medical support system 1b according to this embodiment will be described. Figure 23 is a flowchart illustrating an example of the proposal processing of the medical support system 1b according to this embodiment. Although not shown in the diagram, the medical support system 1b may also execute the series of processes shown in Figure 23 upon receiving a proposal request (proposal request) from a user. For example, a user can send a proposal request to the information support server 60 from the input unit 42 of the management terminal 40. Personnel allocation requests by users may be entered from any terminal.

[0271] As shown in Figure 23, the server control unit 63 of the information support server 60 acquires recorded information (step S521). The server control unit 63 acquires recorded information from the information collection server 30 via the NW communication unit 61.

[0272] Next, the prediction processing unit 633 of the server control unit 63 predicts the optimal treatment for a new patient based on past recorded information (step S522). Specifically, the prediction processing unit 633 predicts the optimal treatment for each symptom (optimal treatment, optimal drug dosage, optimal tool used, and optimal treatment timing (including treatment procedure), etc.) based on statistical data compiled by the aggregation processing unit 631, which shows the relationship between the treatment performed for each symptom, the type and amount of medication administered, the tools used, and the progress information after treatment. The prediction processing unit 633 may also predict the optimal treatment for a new patient using learning results (estimation models that estimate candidate treatments, candidate medications, candidate tools used, or candidate treatment timings for each symptom) that have been learned in advance and stored in the learning result storage unit 623. The prediction processing unit 633 stores the prediction results in the prediction result storage unit 625.

[0273] Next, the suggestion processing unit 636 proposes the optimal course of action based on the prediction results (step S523). The suggestion processing unit 636 transmits the content of the proposed optimal course of action to the mobile terminal 10 or any terminal that the user can access via the NW communication unit 61.

[0274] Next, the proposal processing unit 636 determines whether there is a discrepancy between the predicted and proposed action and the actual action (step S524). In this determination, the presence or absence of a discrepancy between the two can be determined according to predetermined rules. If there is a discrepancy between the two (step S524: YES), the proposal processing unit 636 proceeds to step S525. If there is no discrepancy between the two (step S524: NO), the proposal processing unit 636 terminates the process.

[0275] In step S525, the suggestion processing unit 636 outputs a warning. That is, the suggestion processing unit 636 sends a warning to the mobile terminal 10 via the NW communication unit 61, causing the display unit 13 or speaker 15 to output the warning. The warning may be output from any terminal other than the mobile terminal 10 that the user can see. After processing in step S525, the suggestion processing unit 636 terminates its processing.

[0276] Next, with reference to Figure 24, the external provision process of the medical support system 1b according to this embodiment will be described. Figure 24 is a flowchart showing an example of the external provision process of the medical support system 1b according to this embodiment.

[0277] As shown in Figure 24, the server control unit 63 of the information support server 60 determines whether or not it has received a request for externally provided information (step S531). The server control unit 63 determines whether or not it has received a request for externally provided information from, for example, the management terminal 40 via the NW communication unit 61. If the server control unit 63 has received a request for externally provided information (step S531: YES), it proceeds to step S532. If the server control unit 63 has not received a request for externally provided information (step S531: NO), it returns to step S531.

[0278] In step S532, the external provision processing unit 637 of the server control unit 63 generates external provision information corresponding to a request from the recording information or the aggregation result of the recording information.

[0279] Next, the external provision processing unit 637 stores the generated external provision information in the external provision information storage unit 628 (step S533). After the processing of step S533, the external provision processing unit 637 ends the external provision processing.

[0280] As described above, the information support server 60 according to the present embodiment includes a aggregation processing unit 631 and a prediction processing unit 633. The aggregation processing unit 631 aggregates the required time for treatment of patients for each symptom based on the medical act information indicating the content of medical acts in a medical institution. When there is an inquiry about the acceptability of emergency transportation, the prediction processing unit 633 predicts the acceptable time and the number of acceptable patients for emergency transportation in the medical institution based on the aggregation result aggregated by the aggregation processing unit 631, the situation of medical staff in the medical institution, and the symptoms of the patient related to the emergency transportation.

[0281] Thereby, the information support server 60 according to the present embodiment aggregates the required time for treatment of patients for each symptom, and predicts the acceptable time and the number of acceptable patients for emergency transportation in the medical institution based on the aggregation result, the situation of medical staff in the medical institution, and the symptoms of the patient related to the emergency transportation, so that the data obtained during the medical act can be utilized more effectively.

[0282] Also, the information support server 60 of the medical support system 1b according to the present embodiment includes a aggregation processing unit 631, a prediction processing unit 633, and a staffing generation unit 634. The aggregation processing unit 631 aggregates the medical act information indicating the content of medical acts in a medical institution. The prediction processing unit 633 predicts the busy period of the medical institution based on the aggregation result aggregated by the aggregation processing unit 631. The staffing generation unit 634 generates the staffing of medical staff that can cope with the busy period of the medical institution predicted by the prediction processing unit 633.

[0283] As a result, the information support server 60 according to this embodiment can use medical procedure information obtained during medical procedures for personnel allocation and can effectively utilize recorded information. Therefore, the medical support system 1b according to this embodiment can reduce the burden on medical personnel and improve work efficiency and accuracy.

[0284] In this embodiment, the aggregation processing unit 631 aggregates the number of patients for each symptom based on medical treatment information. The prediction processing unit 633 predicts the busy season for each medical department based on the number of patients for each symptom aggregated by the aggregation processing unit 631. The staffing allocation generation unit 634 generates staffing allocations for medical personnel in medical departments that can handle the busy season, based on the busy season for each medical department predicted by the prediction processing unit 633. As a result, the information support server 60 according to this embodiment can generate appropriate staffing levels for each medical department.

[0285] In this embodiment, the prediction processing unit 633 predicts the peak season for a medical institution based on the aggregated results and regional information regarding the area in which the medical institution is located. The regional information includes at least one of the following: information about the local weather, information about traffic volume in the area, information about events held in the area, information about the local population, information about local industries, and information about other medical institutions located in the area. As a result, the information support server 60 in this embodiment can more accurately predict the peak seasons for medical institutions based on regional information regarding the areas in which medical procedures are performed.

[0286] Furthermore, in this embodiment, the aggregated results include statistical data on the number of patients and symptoms for each period, statistical data on symptoms and the period required for medical treatment, and statistical data on patient information and the period required for medical treatment.

[0287] As a result, the information support server 60 according to this embodiment can more accurately predict the peak seasons for medical institutions by using statistical data for each period and statistical data that takes into account the time required for medical procedures as aggregated results.

[0288] Furthermore, the information support server 60 according to this embodiment includes a labor management processing unit 635. The labor management processing unit 635 performs labor evaluation processing and working time management processing for medical personnel. The aggregation processing unit 631 aggregates the content of medical procedures and the duration required for medical procedures for each medical personnel based on the recorded information, and generates statistical data regarding the work of medical personnel. Then, the labor management processing unit 635 performs labor evaluation processing and labor management processing based on the statistical data regarding the work of medical personnel.

[0289] As a result, the information support server 60 according to this embodiment can use the recorded information obtained during medical procedures to perform labor evaluation processing and labor management processing for medical personnel, thereby making more effective use of the recorded information.

[0290] Furthermore, the information support server 60 according to this embodiment includes a suggestion processing unit 636. Based on past recorded information, the suggestion processing unit 636 suggests possible treatment options or medication options for a new patient. As a result, the medical support system 1b according to this embodiment can make more effective use of recorded information.

[0291] Furthermore, in this embodiment, the recorded information includes progress information showing the course of events after a medical procedure. Based on the prediction results from the prediction processing unit 633, the suggestion processing unit 636 proposes treatments that can be applied to a new patient, based on the past record information including progress information. The treatments proposed for a new patient include candidates for procedures, medications, tools to be used, or treatment timings.

[0292] As a result, the information support server 60 according to this embodiment can suggest candidates for applicable treatments or medications, and can make more effective use of recorded information.

[0293] Furthermore, in this embodiment, when the suggestion processing unit 636 proposes a treatment for a new patient, it suggests candidates for medical professionals who are capable of handling the content of that treatment. As a result, the information support server 60 according to this embodiment can further improve operational efficiency and accuracy.

[0294] Furthermore, the information support server 6 according to this embodiment includes an external provision processing unit 637. The external provision processing unit 637 performs pre-configured processing on medical procedure information indicating the content of medical procedures at a medical institution to generate provision information to be provided to an external organization. Here, the external organization is another medical institution or a medical association, and the provision information provided to the external organization is information that can be used as data for academic presentations on cases. Alternatively, the external organization may be a government-related organization, and the provision information provided to the external organization may be information that can be used for calculating or revising medical fees determined according to the difficulty of treatment. Alternatively, the external organization may be a pharmaceutical company or a medical device manufacturer, and the provision information provided to the external organization may be information that can be used as research and development data. As a result, the information support server 6 in this embodiment can make more effective use of medical procedure information.

[0295] Furthermore, the information support server 60 according to this embodiment includes a learning processing unit 632. The learning processing unit 632 uses machine learning on the learning data based on the aggregation results aggregated by the aggregation processing unit 631 to generate a predictive model that predicts the peak seasons of medical institutions as a learning result. The prediction processing unit 633 predicts the peak seasons of medical institutions based on the predictive model.

[0296] As a result, the information support server 60 according to this embodiment can accurately and efficiently predict peak seasons for medical institutions by using machine learning.

[0297] Furthermore, the information support server 60 according to this embodiment comprises a collection processing unit 191 and a recording processing unit 333. The collection processing unit 191 collects sound information and image information associated with medical procedures. The recording processing unit 333 extracts keywords contained in the collected sound information associated with medical procedures and stores information as recorded information in the recording information storage unit 327, as information associating the keywords with the date and time information in which the keywords were spoken, and information associating the collected image information associated with medical procedures with the date and time information in which the image information was acquired. Recorded information is used as medical procedure information. The aggregation processing unit 631 aggregates the recorded information stored in the recording information storage unit 327.

[0298] As a result, the information support server 60 according to this embodiment, similar to the first embodiment described above, can automatically collect and store more accurate records of medical procedures, further reducing the burden on medical professionals and improving work efficiency and accuracy.

[0299] Furthermore, the information support method according to this embodiment includes an aggregation processing step, a prediction processing step, and a personnel allocation generation step. In the aggregation processing step, the aggregation processing unit 631 aggregates record information recorded based on information collected during medical procedures, which includes patient information indicating the patient, medical professional information indicating the medical professional who performed the medical procedure, and medical procedure information indicating the content of the medical procedure associated with date and time information. In the prediction processing step, the prediction processing unit 633 predicts the busy season of the medical institution based on the aggregation results aggregated by the aggregation processing unit 631. In the personnel allocation generation step, the personnel allocation generation unit 634 generates a personnel allocation of medical professionals capable of handling the busy season of the medical institution based on the busy season of the medical institution predicted by the prediction processing unit 633.

[0300] As a result, the information support method according to this embodiment has the same effects as the information support server 60 described above, reducing the burden on medical professionals and improving work efficiency and accuracy.

[0301] Furthermore, the information support method according to this embodiment includes an aggregation processing step and a prediction processing step. In the aggregation processing step, the aggregation processing unit 631 aggregates the time required for treatment of patients for each symptom based on medical procedure information indicating the content of medical procedures at the medical institution. In the prediction processing step, when an inquiry is made regarding the acceptance of emergency transport, the prediction processing unit 633 predicts the time when the medical institution can accept emergency transport and the number of patients that can be accepted, based on the aggregation results aggregated by the aggregation processing unit 631, the status of medical personnel at the medical institution, and the symptoms of the patients involved in the emergency transport.

[0302] As a result, the information support method according to this embodiment has the same effect as the information support server 60 described above, and the data obtained during medical procedures can be utilized more effectively.

[0303] Furthermore, in the labor management processing performed by the labor management processing unit 635, the time and mental burden (for example, reflecting the fact that a procedure that normally takes this long was performed in this short time) may be estimated by comparing the time required for the procedure with a standard value. In addition, the labor management processing unit 635 may also perform labor management using a point system, taking into account the estimated time and mental burden in addition to indicators such as the number of emergency patients treated and the time taken for treatment.

[0304] Furthermore, the labor management processing unit 635 may, in its labor management processing, add a bonus to the doctors' and nurses' remuneration if the points exceed a certain threshold. Furthermore, the external provision processing unit 637 may provide the statistical data on the time required for each symptom / disease type, compiled by the aggregation processing unit 631, to government-related organizations such as the Ministry of Health, Labour and Welfare, for use in revising medical fees.

[0305] Figures 25 and 26 illustrate the hardware configuration of each device in the medical support system 1 (1a, 1b) according to the embodiment. Figure 25 shows the hardware configuration of each server device: the information collection server 30 (30a), the hospital management server 50, and the information support server 60.

[0306] As shown in Figure 25, each server device of the information collection server 30 (30a), hospital management server 50, and information support server 60 is equipped with a communication device H31, memory H32, and processor H33.

[0307] Communication device H31 is a communication device that can connect to network NW1, such as a LAN card. Memory H32 is a storage device such as RAM, flash memory, or HDD, and stores various information and programs used by the server devices of the information collection server 30 (30a), the hospital management server 50, and the information support server 60.

[0308] Processor H33 is a processing circuit that includes, for example, a CPU. Processor H33 executes various processes of the information collection server 30 (30a), hospital management server 50, and information support server 60 by executing programs stored in memory H32.

[0309] Figure 26 also shows the hardware configuration of each terminal device, the mobile terminal 10 and the management terminal 40.

[0310] As shown in Figure 26, each terminal device of the mobile terminal 10 and the management terminal 40 includes a communication device H11, an input device H12, a display H13, a wireless communication device H14, a microphone H15, a speaker H16, a camera H17, a memory H18, and a processor H19.

[0311] Communication device H11 is a communication device that can connect to network NW1, such as a LAN card, wireless LAN card, or mobile communication device. Input device H12 is, for example, an input device such as a switch, button, touch sensor, or non-contact sensor.

[0312] Display H13 refers to display devices such as liquid crystal displays and organic electro-luminescence (OLED) displays. Wireless communication device H14 is, for example, a Bluetooth® communication device.

[0313] The microphone H15 is a device that picks up sound. The speaker H16 is a device that emits sound. Camera H17, for example, has a CCD image sensor and implements the imaging unit 16 described above.

[0314] Memory H18 is a storage device such as RAM, flash memory, or HDD, and stores various information and programs used by each terminal device, the mobile terminal 10 and the management terminal 40.

[0315] Processor H19 is a processing circuit that includes, for example, a CPU. Processor H26 executes various processes for each terminal device, the mobile terminal 10 and the management terminal 40, by executing programs stored in memory H25.

[0316] This disclosure is not limited to the embodiments described above and may be modified without departing from the spirit of this disclosure. For example, in each of the embodiments described above, an example was described in which the information collection server 30 (30a) comprises a speech recognition processing unit 331, an image recognition processing unit 332, a recording processing unit 333 (333a), a support processing unit 334, an editing processing unit 335, and a medical document generation unit 336. However, this disclosure is not limited to such examples. It is sufficient that the medical support system 1 (1a) as a whole realizes the functions described above, and the device configuration may be arbitrary, and each function may be mounted on any device. For example, some or all of the functions of the information collection server 30 (30a) may be provided by the mobile terminal 10. For example, in the first embodiment, if the mobile terminal 10 is equipped with all of the functions of the information collection server 30, even if a medical procedure is performed during a power outage or in a location where communication with the information collection server 30 is not good, the mobile terminal 10 can execute the functions of the medical support system 1 described above, such as displaying a timeline of recorded information, thereby realizing smooth recording of medical procedures. In that case, after communication with the information collection server 30 is restored, various types of information, such as recorded information, generated and stored by the mobile terminal 10 may be transmitted to and stored in the information collection server 30. Similarly, with respect to medical support system 1b, the device configuration and the functions installed in each device may be arbitrarily modified so that each of the functions described above can be realized. Furthermore, some or all of the matters described in each of the above embodiments may be combined and implemented to the extent possible.

[0317] Furthermore, in each of the above embodiments, examples of application to a medical support system 1 (1a, 1b) in the medical field were described as examples of a support system, but the system is not limited to this, and the support system may be applied to, for example, the manufacturing industry or the construction industry (work in factories or on site).

[0318] When the support system disclosed herein is applied to the manufacturing and construction industries, it is useful in situations where it is not possible to operate a terminal by hand, such as on a job site where thick gloves are worn, on a job site where hands get dirty, or in high places, underwater, or in dark places. The support system disclosed herein allows for recording descriptions of the situation using voice, and for taking photographs using wearable devices such as smart glasses via voice control, thereby reducing the burden on workers and improving work efficiency and accuracy.

[0319] Furthermore, in each of the above embodiments, the collected image data may be image data of the drug pack. In this case, the image recognition processing unit 332 may recognize the type and remaining amount (i.e., dosage) of the drug from the image data of the drug pack, and the recording processing unit 333 (333a) may record the recognition result as recorded information.

[0320] In other words, in the embodiments described above, the image recognition processing unit 332 extracted text data contained in the image data. However, this disclosure is not limited to such examples, and the image recognition processing unit 332 may recognize various situations in a medical setting using various known image recognition technologies. For example, as described above, the image recognition processing unit 332 recognizes the type and amount of drug (i.e., dosage) from image data of a drug pack, and the recording processing unit 333 (333a) records the recognition result as recording information, making it possible to monitor whether the correct drug is being administered in the appropriate amount.

[0321] As a result, the recording processing unit 333 (333a) can also output a warning if the type, quantity, and flow rate of the medication actually used differ from the medication order given by voice. In addition to the above example, the image recognition processing unit 332 may, for example, identify a medical professional involved in a medical procedure using facial recognition technology from image data showing the face of a medical professional, or identify a symptom by comparing image data showing the affected area of ​​a patient with a database or a learning model generated by machine learning.

[0322] In the embodiments described above, keywords to be extracted (i.e., keywords to be included in the recorded information) are stored in advance in the keyword information storage unit 326, and the recording processing unit 333 (333a) extracts keywords to be included in the recorded information by comparing the text data converted from sound data by the speech recognition processing unit 331 and the text data detected from image data by the image recognition processing unit 332 with the keywords stored in the keyword information storage unit 326. However, this disclosure is not limited to such examples. The recording processing unit 333 (333a) may also extract keywords using a machine learning model that has learned the keywords to be included in the recorded information. In that case, each time keywords are extracted and recorded information is created in the medical support system 1 (1a), the machine learning model may be updated using that recorded information as training data. This can improve the accuracy of the machine learning model each time recorded information is created using the medical support system 1 (1a).

[0323] Furthermore, the timeline display of recorded information is not limited to the examples described above. The timeline display may be appropriately designed to improve visibility for medical professionals reviewing it. For example, when keywords are displayed in the timeline, the source of the keyword (i.e., information indicating the speaker if the keyword is extracted from audio data, or the name of the device shown in the image if the keyword is extracted from image data) may be included. Also, for example, if the keyword is extracted from audio data, information that can identify the speaker (such as the speaker's name or nickname, or an icon or ID indicating the speaker) may be included along with the keyword. This makes it easy to check the source of the keyword (for example, who made the statement related to the keyword) on the timeline display.

[0324] Furthermore, for example, keywords may be categorized, and in the timeline display, keywords may be displayed in different colors for each category. For example, keywords may be categorized into those related to vital signs (blood pressure, pulse, etc.), drug names, names of medical materials used in treatments, etc., and displayed in different colors for each category. Also, when the editing processing unit 335 searches for and outputs arbitrary information from the recorded information at the instruction of the user, it may be searchable by category. In this case, the user may be able to specify the period of the recorded information to be searched. For example, the user can specify a period for each category, such as a list of drugs used today or a list of medical devices used in the past year, and search for and display the desired information in a list. This increases the accessibility of the information desired by the user and further improves its convenience. Note that categorization may be performed automatically by the recording processing unit 333 (333a). For example, the recording processing unit 333 (333a) may perform categorization by matching with a predetermined dictionary, or by using machine learning.

[0325] Furthermore, each component of the medical support system 1 (1a, 1b) described above has a computer system inside. The processing in each component of the medical support system 1 (1a, 1b) described above may be performed by recording a program for realizing the functions of each component of the medical support system 1 (1a, 1b) onto a computer-readable recording medium, loading the program recorded on this recording medium into the computer system, and executing it. Here, "loading the program recorded on the recording medium into the computer system and executing it" includes installing the program into the computer system. Here, "computer system" includes the OS and hardware such as peripheral devices.

[0326] Furthermore, "computer system" may include multiple computer devices connected via a network, including communication lines such as the Internet, WAN, LAN, and dedicated lines. Also, "computer-readable recording medium" refers to portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into the computer system. Thus, the recording medium storing the program may also be a non-transient recording medium such as a CD-ROM.

[0327] Furthermore, the recording medium also includes internal or external recording media accessible from the distribution server for distributing the program. The program may be divided into multiple parts, downloaded at different times, and then combined in the respective configurations of the medical support system 1 (1a, 1b) described above, or each of the divided programs may be distributed by a different distribution server. Additionally, "computer-readable recording media" includes volatile memory (RAM) within a computer system that acts as a server or client when the program is transmitted over a network, which retains the program for a certain period of time. Moreover, the program may be intended to implement only a part of the functions described above. Furthermore, the program may be a so-called differential file (differential program) that can implement the functions described above in combination with a program already recorded in the computer system. [Explanation of Symbols]

[0328] 1,1a,1b...Medical support system, 10...Mobile terminal, 11,31,41,61...Network communication unit, 12,42...Input unit, 13,43...Display unit, 14,22...Microphone, 15...Speaker, 16,23...Imaging unit, 17,21...Wireless communication unit, 18,44...Terminal storage unit, 19,45...Terminal control unit, 20...Headset device, 25...External imaging device, 30,30a...Information collection server, 32, 62...Server storage unit, 33, 33a, 63...Server control unit, 40...Management terminal, 50...Hospital management server, 55...External display device, 60...Information support server, 181...Setting information storage unit, 191...Collection processing unit, 192...Output control unit, 193...Setting processing unit, 321...Terminal information storage unit, 322...User information storage unit, 323...Learning result storage unit, 324...Collected sound information storage unit, 325...Collected image information Reporting memory unit, 326... Keyword information storage unit, 327... Recording information storage unit, 328... Format storage unit, 329... Medical document storage unit, 331... Speech recognition processing unit, 332... Image recognition processing unit, 333, 333a... Recording processing unit, 334... Support processing unit, 335... Editing processing unit, 336... Medical document generation unit, 337, 632... Learning processing unit, 451, 551... Display control unit, 621... Aggregation information storage unit, 622... Learning data storage unit, 623... Learning result storage unit, 624... Regional information storage unit, 625... Prediction result storage unit, 626... Personnel allocation information storage unit, 627... Labor management information storage unit, 628... External provision information storage unit, 631... Aggregation processing unit, 633... Prediction processing unit, 634... Personnel allocation generation unit, 635... Labor management processing unit, 636... Proposal processing unit, 637... External provision processing unit, NW1... Network

Claims

1. A collection and processing unit that collects sound and image information from users, A recording processing unit extracts a first keyword which is a keyword contained in the sound information collected by the collection processing unit, and stores information relating the first keyword to the date and time information in which the first keyword was spoken, and information relating the image information collected by the collection processing unit to the date and time information in which the image information was acquired, as recording information in a recording information storage unit. An output control unit causes a display unit to display a timeline display in which the first keyword and the image information are arranged in chronological order based on the recorded information stored in the recorded information storage unit. An input unit that receives information entered by the user by touching the display unit, Equipped with, When the display unit is displaying the image information, the output control unit adds and displays text based on the user's voice on top of the trajectory traced by the user's finger. Support system.

2. The aforementioned collection processing unit, When a hands-free operation performed by the user without physical contact is detected, and an initiation operation to start collecting the sound information or image information is initiated, the collection of the sound information or image information is initiated. The support system according to claim 1.

3. The aforementioned collection processing unit, In the hands-free operation performed by the user without contact, if an termination operation is detected that ends the collection of the sound information or the image information, the collection of the sound information or the image information will be terminated. The support system according to claim 1.

4. A sound collection unit that collects sounds associated with medical procedures and outputs the sound information, An imaging unit that captures images associated with the aforementioned medical procedure and outputs the image information. Equipped with, The hands-free operation includes at least one of the following: voice operation by the user detected based on the sound information output by the sound pickup unit; motion operation by the user detected based on the image information output by the imaging unit; and gaze operation by the user detected by the headset equipped with the imaging unit. The support system according to claim 2.

5. The recording processing unit extracts a second keyword which is a keyword included in the image information collected by the collection processing unit, and stores information relating the image information, the date and time information on which the image information was acquired, and the second keyword in the recording information storage unit as recording information. The output control unit, based on the recorded information, displays the image information, the date and time information on which the image information was acquired, and the second keyword in association with each other on the timeline display of the display unit. The support system according to any one of claims 1 to 4.

6. The recording processing unit causes the recording information created for each period during which the sound information and the image information were collected to be stored in the recording information storage unit. The output control unit causes the timeline display to be displayed on the display unit for each of the specified periods. The support system according to any one of claims 1 to 4.

7. The first and second keywords mentioned above include at least one of the following: terms related to treatment, terms related to jigs used in the treatment, terms related to drugs, and terms related to the patient's biological information. The support system according to claim 5.

8. The aforementioned users include multiple healthcare professionals, The recording processing unit identifies the speaker among the multiple medical professionals who uttered the first keyword based on the sound information, and stores the information relating the identification information for the speaker, the first keyword, and the date and time information as the recording information in the recording information storage unit. The support system according to any one of claims 1 to 4.

9. The recording processing unit outputs a warning when it detects an abnormality in a medical procedure based on the recording information stored in the recording information storage unit. The support system according to any one of claims 1 to 4.

10. The system includes a support processing unit that performs support processing in response to a user's request for support regarding medical procedures and outputs the support processing results to the user. The support system according to any one of claims 1 to 4.

11. The support processing unit generates a proposal regarding medical procedures based on the past recorded information, and outputs the generated proposal regarding medical procedures to the user as the support processing result. The support system according to claim 10.

12. The recording information storage unit includes an editing processing unit that edits the recorded information stored in it and outputs the edited recorded information to the user. The support system according to any one of claims 1 to 4.

13. The system includes a medical document generation unit that generates medical documents in a predetermined format based on the recorded information stored in the recorded information storage unit. The support system according to any one of claims 1 to 4.

14. The aforementioned display unit is a display unit of a mobile terminal carried by the user. The support system according to any one of claims 1 to 4.

15. The aforementioned display unit is a display device installed in a place where medical procedures are performed. The recording processing unit, via the output control unit, causes the timeline display to be updated with the latest information and displayed on the display device. The support system according to any one of claims 1 to 4.

16. A collection and processing unit that collects sound and image information from users, A recording processing unit extracts a first keyword which is a keyword contained in the sound information collected by the collection processing unit, and stores information relating the first keyword to the date and time information in which the first keyword was spoken, and information relating the image information collected by the collection processing unit to the date and time information in which the image information was acquired, as recording information in a recording information storage unit. Equipped with, The recording processing unit causes the display device to display a timeline display in which the first keyword and the image information are arranged in chronological order, based on the recording information stored in the recording information storage unit. The recording processing unit, when the display device is displaying the image information, receives information input by the user through a touch operation on the display device, and adds and displays text based on the user's voice on top of the trajectory traced by the user's finger. Server device.

17. The collection processing unit collects sound and image information from the user. The recording processing unit extracts a first keyword which is a keyword contained in the sound information collected by the collection processing unit, and stores information relating the first keyword to the date and time information in which the first keyword was spoken, and information relating the image information collected by the collection processing unit to the date and time information in which the image information was acquired, as recorded information in the recording information storage unit. The output control unit causes the display unit to display a timeline display in which the first keyword and the image information are arranged in chronological order, based on the recorded information stored in the recorded information storage unit. The input unit receives information entered by the user by touching the display unit. When the display unit is displaying the image information, the output control unit adds and displays text based on the user's voice on top of the trajectory traced by the user's finger. How to help.

Images