Information processing method, program, and information processing device

Abstract

To provide an information processing method, etc. for recording the communication performed in a hospital room.SOLUTION: The information processing method causes a computer to execute processing for: detecting entry of a person into a hospital room where a camera and a microphone are installed; determining whether or not the person having entered the room is a healthcare personnel; acquiring conversation data 62 during conversion between a patient hospitalized in a hospital room 45 and the healthcare personnel, which includes images captured by the camera and voices recorded by the microphone; inputting conversation data 62 to a model 51 that receives input of the conversation data 62 and outputs category information related to the conversation, and acquiring category information; and recording the conversation data 62 in association with the acquired category information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing method, a program, and an information processing device. [Background technology]

[0002] BACKGROUND ART An information processing method has been proposed that provides various information to guests by utilizing a user interface installed in a guest room of an accommodation facility (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2019 / 244365 Summary of the Invention [Problem to be solved by the invention]

[0004] In medical institutions such as hospitals, communication may take place within the hospital among people of various positions, such as patients, medical staff, patients' families, and non-medical staff who are employees of the medical institution. However, the information processing method of Patent Document 1 cannot record the content of communication.

[0005] In one aspect, an object of the present invention is to provide an information processing method for recording communications carried out within a hospital or the like. [Means for solving the problem]

[0006] The information processing method involves a computer executing a process in which it detects a person entering a hospital room in which a camera and a microphone are placed, determines whether the person who entered the room is a medical professional, acquires dialogue data including images captured by the camera and audio recorded by the microphone during a dialogue between a patient hospitalized in the hospital room and the medical professional, inputs the dialogue data into a model that accepts input of dialogue data and outputs type information regarding the dialogue, acquires type information, and records the dialogue data in association with the acquired type information. [Effects of the Invention]

[0007] In one aspect, an information processing method for recording communications that take place in a hospital room can be provided. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 2 is an explanatory diagram illustrating an outline of the operation of the information processing system. [Figure 2] FIG. 1 is an explanatory diagram illustrating a configuration of an information processing system. [Figure 3] FIG. 2 is an explanatory diagram illustrating input and output of a model. [Figure 4] FIG. 10 is an explanatory diagram illustrating a record layout of a dialogue DB. [Figure 5] 10 is a flowchart illustrating the flow of processing of a program that is started when entry of a medical professional into a room is detected. [Figure 6] 10 is a flowchart illustrating the flow of program processing when a dialogue DB is utilized. [Figure 7] 10 is a flowchart illustrating the flow of processing of a program that is started when the entry of a cleaning staff member into a room is detected. DETAILED DESCRIPTION OF THE INVENTION

[0009] [Embodiment 1] Fig. 1 is an explanatory diagram illustrating an overview of the operation of the information processing system 40. The information processing method shown in Fig. 1 is used in a medical facility. In the following explanation, an example will be given in which a patient is hospitalized in a private room.

[0010] A camera 28 is placed in a hospital room 45 where a patient is hospitalized. The camera 28 constantly photographs and records images inside the hospital room 45. The image photographed by the camera 28 or a sensor attached to the door or the like detects that a person has entered the hospital room 45. Using means such as facial recognition, it is determined whether the person who has entered the room is a medical professional or cleaning staff member.

[0011] If the person entering the room is a medical professional, dialogue data 62 relating to communication between the patient and the medical professional is recorded until the medical professional's exit is detected. The dialogue data 62 includes audio and video recorded during the communication. Note that the dialogue data 62 may be audio only.

[0012] Dialogue data 62 is input to the model 51, and type information relating to the type of dialogue is output from the model 51. Details of the model 51 will be described later. The dialogue data 62 and type information are associated with each other and recorded in a dialogue DB (Database) 55 (see FIG. 2), which will be described later. The type information includes information relating to the type of dialogue and the patient's emotions.

[0013] If, for example, a problem occurs between a patient and a medical professional at a later date, a user such as a hospital director or head nurse can use the type information to inquire about the recorded content. Dialogue data 62 related to the type information is returned. By checking the dialogue data 62, it is possible to make an objective judgment based on the record, for example, as to whether the medical professional's explanation to the patient was sufficient.

[0014] To protect privacy and prevent storage capacity from becoming too large, data recorded in the dialogue data 62 is automatically deleted after a predetermined period of time, such as two weeks. The user can prevent automatic deletion by, for example, flagging data recording a dispute that occurred between a patient and a medical professional as non-deletable.

[0015] If the person entering the room is a cleaning staff member, information about areas that need to be cleaned intensively is output based on images captured by the camera 28 since the previous cleaning. Areas that need to be cleaned intensively are, for example, areas that have been touched by a patient. The cleaning staff will give particular attention to cleaning the areas that need to be cleaned intensively.

[0016] After the series of cleaning tasks is completed, information about areas that were not cleaned is output based on images taken by the camera 28 during cleaning. The cleaning staff then re-cleans the areas that were not cleaned. The images taken by the camera 28 or a sensor attached to the door detects the cleaning staff's departure from the room.

[0017] The images taken by the camera 28 are deleted after a predetermined period of time, such as two weeks, has passed, or after the cleaning staff has left the facility.

[0018] 2 is an explanatory diagram illustrating the configuration of information processing system 40. Information processing system 40 includes information processing device 10, user interface 20, and second camera 282. User interface 20 and second camera 282 are placed in patient room 45.

[0019] The information processing device 10 includes a first control unit 11, a main memory device 12, an auxiliary memory device 13, a communication unit 14, a reading unit 19, and a bus. The first control unit 11 is an arithmetic and control device that executes a program 97 of this embodiment. The first control unit 11 uses one or more central processing units (CPUs), graphics processing units (GPUs), multi-core CPUs, etc. The first control unit 11 is connected to each hardware unit that constitutes the information processing device 10 via the bus.

[0020] The main memory device 12 is a memory device such as an SRAM (Static Random Access Memory), a DRAM (Dynamic Random Access Memory), a flash memory, etc. The main memory device 12 temporarily stores information required during processing performed by the first control unit 11 and a program 97 currently being executed by the first control unit 11.

[0021] The auxiliary storage device 13 is a storage device such as an SRAM, a flash memory, a hard disk, or a magnetic tape. The auxiliary storage device 13 stores the model 51, the conversation DB 55, the recorded data 63, the program 97 to be executed by the first control unit 11, and various data required for executing the program 97. The conversation DB 55 and the recorded data 63 may be stored in an external mass storage device connected to the information processing device 10.

[0022] The communication unit 14 is an interface for communicating with the network. The reading unit 19 is a device for reading the portable recording medium 96 on which the program 97 is recorded. The first control unit 11 reads the program 97 recorded on the portable recording medium 96, the semiconductor memory 98, or another server computer connected via the network, and stores it in the auxiliary storage device 13.

[0023] The information processing device 10 is a general-purpose personal computer, a tablet, a mainframe computer, or a virtual machine running on a mainframe computer. The information processing device 10 may be configured with hardware such as multiple personal computers or mainframe computers that perform distributed processing. The information processing device 10 may also be configured with a cloud computing system. The information processing device 10 may also be configured with hardware such as multiple personal computers or mainframe computers that operate in cooperation with each other.

[0024] The user interface 20 includes a second control unit 21, a main memory device 22, an auxiliary memory device 23, a communication unit 24, a touch panel 25, a microphone 26, a speaker 27, a first camera 281, and a bus. The second control unit 21 is an arithmetic and control device that executes the program 97 of this embodiment. The second control unit 21 uses one or more CPUs, GPUs, multi-core CPUs, etc. The second control unit 21 is connected to each hardware unit that constitutes the user interface 20 via the bus.

[0025] The main memory device 22 is a memory device such as an SRAM, a DRAM, a flash memory, etc. The main memory device 22 temporarily stores information required during the processing performed by the second control unit 21 and a program 97 being executed by the second control unit 21.

[0026] The auxiliary storage device 23 is a storage device such as an SRAM, a flash memory, a hard disk, a magnetic tape, etc. The auxiliary storage device 23 stores a program 97 to be executed by the second control unit 21 and various data required for executing the program 97.

[0027] The communication unit 24 is an interface for communicating with a network. The user interface 20 is an information device such as a smart speaker, tablet, or smartphone. Although not shown, the hospital room 45 is equipped with a television, lighting, air conditioner, or the like that are controlled by radio waves or infrared rays emitted from the user interface 20.

[0028] The program 97 executed by the second control unit 21 is a part of the program 97 read by the first control unit 11 from the portable recording medium 96, and is stored in the auxiliary storage device 23 via the network. The second control unit 21 may read the program 97 recorded in another server computer connected via the network and store it in the auxiliary storage device 23.

[0029] Touch panel 25, microphone 26, speaker 27, and first camera 281 are either built into user interface 20 or attached externally. Second camera 282 is positioned so as to capture the entire interior of hospital room 45, including areas that would be blind spots of first camera 281. Second camera 282 is preferably positioned so as to capture the interior of hospital room 45 without any blind spots, even when people such as patients or staff are moving around within hospital room 45. Multiple second cameras 282 may be positioned in hospital room 45 so as to cover each other's blind spots.

[0030] The second camera 282 is a network camera, and transmits captured images to the information processing device 10 in real time. If there is an area that is in the blind spot of the second camera 282 but is captured by the first camera 281, the second control unit 21 also transmits the images captured by the first camera 281 to the information processing device 10 in real time. The images transmitted to the information processing device 10 are recorded in the recording data 63 regardless of whether or not a person has entered the hospital room 45. That is, the camera 28 described using FIG. 1 may include the first camera 281 in addition to the second camera 282.

[0031] 3 is an explanatory diagram illustrating input and output of the model 51. The model 51 is a multimodal large language model (LLM) such as GPT-4 or Gemini, which receives input of dialogue data 62 and a prompt, and outputs type information regarding the dialogue data 62.

[0032] The dialogue data 62 is the voice captured by the microphone 26, i.e., the recorded voice, and the image captured by the first camera 281 during communication between the healthcare professional and the patient. The dialogue data 62 may also include an image captured by the second camera 282 at the same time. By including the image captured by the second camera 282 in the dialogue data 62, an image including the patient's facial expressions and gestures is included in the dialogue data 62, even if the patient has their back turned to the user interface 20, for example.

[0033] The prompt may be, for example, "Please tell us the type of dialogue and the patient's emotions during the dialogue in 20 characters or less each." The type information is the type of dialogue and the patient's emotions during the dialogue. Specific examples of the dialogue type and the patient's emotions output by the model 51 will be described later with reference to FIG. 4. The model 51 may be a machine learning model generated by machine learning.

[0034] 4 is an explanatory diagram illustrating the record layout of the dialogue DB 55. The dialogue DB 55 has a patient name field, a hospital room number field, a dialogue start date and time field, a medical staff field, a type information field, and a file field. The type information field has a type field and a patient emotion field.

[0035] The patient name field records the name of the patient. The room number field records the number of the hospital room 45. The dialogue start date and time field records the date and time when communication between the medical professional and the patient started. The dialogue start date and time field may also record the time when the medical professional entered the hospital room 45.

[0036] The medical professional field records the medical professional who was present during the communication with the patient. The type field records the type of dialogue output from Model 51. The type of dialogue is a short sentence that indicates the purpose or overview of the communication between the patient and the medical professional, such as "explanation of future policy," "preoperative explanation," or "explanation regarding payment."

[0037] The patient's emotion may be, for example, "resignation" or "anxiety." The patient's emotion may be, for example, the emotion that the patient felt for the longest period of time during the dialogue, or the emotion at the end of the dialogue. Changes in the patient's emotion over time may be recorded in the dialogue DB 55.

[0038] 4 are merely examples. The type information field may include a field for recording the patient's emotions as well as a field for recording the healthcare professional's emotions. The dialogue DB 55 may also include a field for recording people who were present during the dialogue, such as the patient's family members.

[0039] The file field stores the file in which the dialogue data 62 is recorded, or information indicating the location where the file is saved. The dialogue DB 55 has one record for each communication between a patient and a medical staff member. Note that the dialogue DB 55 may also store records related to communication between a patient and a visitor, and communication between a patient and a non-medical staff member, such as a cleaning staff member.

[0040] 5 is a flowchart illustrating the processing flow of the program 97 that is activated when the entry of a medical professional into the room is detected. The second control unit 21 starts transmitting, in real time, the voice acquired via the microphone 26 and the image acquired via the first camera 281 to the information processing device 10 (step S541).

[0041] The first control unit 11 starts receiving audio and images (step S641). The first control unit 11 temporarily records the received audio and images in the auxiliary storage device 13.

[0042] The second control unit 21 detects the exit of the medical staff (step S542). If multiple medical staff have entered the room, the second control unit 21 detects the exit of all of the medical staff. The second control unit 21 ends the transmission of the voice and image, and transmits a signal notifying the end of transmission to the information processing device 10 (step S543).

[0043] The first control unit 11 receives the notification of the end of transmission (step S642). The first control unit 11 creates dialogue data 62 based on the voice and images recorded between steps S641 and S642. As described with reference to FIG. 3, the first control unit 11 inputs the dialogue data 62 and the prompt into the model 51 to acquire type information (step S643). The first control unit 11 creates a new record in the dialogue DB 55 and records information corresponding to each field (step S644).

[0044] 6 is a flowchart illustrating the processing flow of the program 97 when the dialogue DB 55 is utilized. The following explanation will be given taking as an example a case where the information processing device 10 is used. The dialogue DB 55 may be utilized using the user interface 20 or any hardware capable of acquiring the dialogue DB 55 via a network.

[0045] The first control unit 11 accepts conditions specified by the user (step S651). The conditions include, for example, the number of the hospital room 45, the date, the names of the medical staff involved, the type of dialogue, and the patient's emotions. The conditions may be some of the items exemplified above. The first control unit 11 searches the dialogue DB 55 using each of the accepted conditions as a key to extract records (step S652). The first control unit 11 outputs the dialogue data 62 recorded in the file field of the extracted record (step S653).

[0046] The program 97 described with reference to FIG. 6 can provide an information processing system 40 that can easily confirm what actually happened when, for example, a problem occurs between a patient and a medical professional.

[0047] FIG. 7 is a flowchart illustrating the flow of processing by the program 97 that is activated when the entry of a cleaning staff member into the room is detected.

[0048] The second control unit 21 acquires the voice of the cleaning staff via the microphone 26 (step S551). The second control unit 21 analyzes the meaning of the voice (step S552). The second control unit 21 may analyze the meaning of the voice in cooperation with a voice recognition server, a language analysis server, etc. (not shown).

[0049] The second control unit 21 determines whether the voice is a question about the area to be cleaned (step S553). If it is determined that the voice is not a question about the area to be cleaned (NO in step S553), the second control unit 21 returns to step S551. If it is determined that the voice is a question about the area to be cleaned (YES in step S553), the first control unit 11 transmits information about the content of the question to the information processing device 10 (step S554). The information about the content of the question is, for example, data recording the voice of the cleaning staff. The information about the content of the question may be information about whether the content of the question is a question about an area to be cleaned that needs to be cleaned or an area that has not been cleaned.

[0050] The first control unit 11 receives information about the question content (step S661). The first control unit 11 determines whether the question content is a question about an area to be intensively cleaned or a question about an area that has not been cleaned (step S662).

[0051] If it is determined that the question is about areas that need to be cleaned (areas that need to be cleaned in step S662), the first control unit 11 acquires the video recording data 63 recorded since the previous cleaning (step S663). The first control unit 11 inputs the acquired video recording data 63 into an object detection model such as YOLO, and determines areas that need to be cleaned, such as areas that the patient touches frequently or areas that the patient touches for a long time (step S664).

[0052] The first control unit 11 may analyze the image captured by the camera 28 in real time to detect the location touched by the patient and record it together with the video data 63. This makes it possible to realize an information processing system 40 that requires less time to determine areas that need to be cleaned.

[0053] The first control unit 11 generates a response sentence indicating the areas to be cleaned with priority (step S665). Specifically, for example, the first control unit 11 inputs an image with the areas to be cleaned with priority markings and a prompt such as "The markings indicate areas to be cleaned with priority. Please explain the areas to be cleaned with priority in 50 characters or less" into a language model such as LLM, and obtains the response sentence output from the language model.

[0054] If it is determined that the question is about an area not cleaned (area not cleaned in step S662), the first control unit 11 acquires video recording data 63 recorded after the cleaning staff entered the room (step S666). The first control unit 11 inputs the acquired video recording data 63 into an object detection model such as YOLO to detect areas that have been cleaned. The first control unit 11 determines areas in the hospital room 45 that have not been cleaned, i.e., areas that have not been cleaned (step S667).

[0055] The first control unit 11 may input the images captured by the camera 28 into an object detection model such as YOLO in real time to detect in real time the areas that have been cleaned, i.e., the areas that have been touched by cleaning tools such as a dustcloth or mop. The first control unit 11 can realize an information processing system 40 that requires less time to determine areas that have not been cleaned by recording the areas that have been cleaned in a predetermined checklist.

[0056] The first control unit 11 generates a response sentence indicating the missed cleaning areas (step S668). Specifically, for example, the first control unit 11 inputs an image in which the missed cleaning areas are marked and a prompt such as "The markings indicate missed cleaning areas. Please explain the missed cleaning areas in 50 characters or less" into a language model such as LLM, and obtains a response sentence output from the language model. If there are no missed cleaning areas, the first control unit 11 obtains a default response sentence such as "There are no missed cleaning areas."

[0057] After step S665 or step S668 is completed, the first control unit 11 transmits the generated response sentence to the user interface 20 (step S669). The second control unit 21 receives the response sentence (step S555). The second control unit 21 outputs the response sentence by voice via the speaker 27 (step S556).

[0058] The second control unit 21 may output a response text via the touch panel 25. The first control unit 11 may transmit an image indicating areas that need to be cleaned or areas that have not been cleaned, in addition to the response text, in step S669, and the second control unit 21 may display the received image on the touch panel 25 in step S556.

[0059] The second control unit 21 determines whether the cleaning staff member has left (step S557). If it is determined that the cleaning staff member has not left (NO in step S557), the second control unit 21 returns to step S551. If it is determined that the cleaning staff member has left (YES in step S557), the second control unit 21 ends the process.

[0060] This embodiment provides an information processing system 40 that records conversations between medical professionals and patients and allows for prompt confirmation when a problem occurs. By checking the conversation data 62, a person in charge, such as the hospital director, can take appropriate action to deal with the problem based on objective records. By making it known that the conversation data 62 is being recorded, the occurrence of so-called monster patients can be prevented.

[0061] The first control unit 11 may be configured to, for example, execute step S664 and then execute step S667, and if it determines in step S667 that the same area determined to be a priority cleaning area in step S664 has not been cleaned, output a warning such as "Despite being pointed out as a priority cleaning area, cleaning has not been carried out."

[0062] To protect privacy and prevent storage capacity from becoming too large, data recorded in the dialogue data 62 is automatically deleted after a predetermined period of time, such as two weeks. The user can prevent automatic deletion by, for example, flagging data recording a dispute that occurred between a patient and a medical professional as non-deletable.

[0063] According to this embodiment, it is possible to provide an information processing system 40 that supports cleaning staff by indicating areas that need to be cleaned and areas that have not been cleaned. Because audio is not recorded in the video recording data 63, it is possible to provide an information processing system 40 that supports cleaning staff while protecting patient privacy to the maximum extent possible.

[0064] Program 97 is an example of a program product. A computer program can be deployed to be executed on a single computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0065] The technical features (constituent elements) described in each embodiment can be combined with each other, and by combining them, new technical features can be formed. The embodiments disclosed herein are to be considered as illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims.

[0066] Independent and dependent claims may be combined with each other in any combination, regardless of the reference format. Furthermore, while the claims may be written in a format in which a claim references two or more other claims (multiple claim format), this is not a limitation. Multiple claims that reference at least one other claim (multiple multiple claim format) may also be written. [Explanation of symbols]

[0067] 10. Information processing equipment 11 First Control Section 12 Main storage 13 Auxiliary storage device 14 Communications Department 19 Reading unit 20 User Interface 21 Second Control Section 22 Main storage 23 Auxiliary storage device 24 Communications Department 25 Touch Panel 26. Mike 27 Speaker 28 Camera 281 Camera 1 282 Second Camera 40 Information Processing Systems 45 Hospital Room 51 Model 55 Dialogue DB 62 Interaction Data 63 Recorded Data 96 Portable recording media 97 Programs 98 Semiconductor Memory

Claims

1. Detecting the entry of a person into a patient room where a camera and a microphone are installed; determining whether the person entering the room is a medical professional; acquiring conversation data including an image captured by the camera and a voice recorded by the microphone during a conversation between the patient hospitalized in the hospital room and the medical staff; inputting the dialogue data into a model that receives input of the dialogue data and outputs type information relating to the dialogue, thereby acquiring type information; The dialogue data and the acquired type information are recorded in association with each other. An information processing method in which processing is performed by a computer.

2. The camera records video data regardless of whether or not a person enters the patient room, Determine whether the person who entered the room is a cleaning staff member; When a question about a priority cleaning area is received from the cleaning staff, information about the priority cleaning area is output based on the video recording data, When a question about a spot that has not been cleaned is received from the cleaning staff, information about the spot that has not been cleaned is output based on a portion of the video data that has been filmed after the cleaning staff has entered the room. The information processing method according to claim 1 .

3. When type information is received, the recorded dialogue data is output in association with the received type information. The information processing method according to claim 1 .

4. Detecting the entry of a person into a patient room where a camera and a microphone are installed; determining whether the person entering the room is a medical professional; acquiring conversation data including an image captured by the camera and a voice recorded by the microphone during a conversation between the patient hospitalized in the hospital room and the medical staff; inputting the dialogue data into a model that receives input of the dialogue data and outputs type information relating to the dialogue, thereby acquiring type information; The dialogue data and the acquired type information are recorded in association with each other. A program that causes a computer to perform a process.

5. An information processing device including a control unit, The control unit Detecting the entry of a person into a patient room where a camera and a microphone are installed; determining whether the person entering the room is a medical professional; acquiring conversation data including an image captured by the camera and a voice recorded by the microphone during a conversation between the patient hospitalized in the hospital room and the medical staff; inputting the dialogue data into a model that receives input of the dialogue data and outputs type information relating to the dialogue, thereby acquiring type information; The dialogue data and the acquired type information are recorded in association with each other. Information processing device.

Images