Program and information processing method

Abstract

To provide a program and an information processing method for generating a conversation log with annotations inserted thereto.SOLUTION: In an information processing system, a server, a remote diagnosis support apparatus, a first communication device, and a second communication device are connected to communicate each other over a network. The server which executes a program acquires call information from the first communication device and the second communication device, generates conversation log information based on the call information and stores the log information, outputs the conversation log information to at least one of the first communication device and the second communication device 4, determines whether an annotation setting instruction has been acquired from the second communication device, acquires, when the instruction is acquired, information on an annotation from the second communication device, updates the conversation log information based on the acquired information on the annotation, and outputs the updated conversation log information to the first and second communication devices.SELECTED DRAWING: Figure 13

Description

[Technical Field]

[0001] FIELD An embodiment of the present invention relates to a program and an information processing method. [Background technology]

[0002] In recent years, in addition to "face-to-face treatment" in which patients visit the hospital and meet with doctors face-to-face, "telemedicine" has become a common medical treatment method in hospitals, where doctors and patients use communication devices to receive treatment without actually visiting a medical institution. In telemedicine, medical equipment is installed in the patient's home or a designated location, and vital data obtained by the medical equipment is transmitted via communication to a doctor in a remote location, who then makes a diagnosis based on the data.

[0003] One type of remote medical care is an information processing system known as a remote medical care booth, box, or kiosk, which is equipped with equipment capable of communicating with medical devices at a designated location and allows for real-time medical treatment via video calls between patients and doctors. This allows for not only interactive medical treatment via communication devices, but also medical treatment using blood pressure monitors, stethoscopes, and other medical equipment, enabling more detailed examination of symptoms.

[0004] In remote medical care, conversations conducted via communication devices are sometimes converted into text data, and conversation logs are used that doctors, patients, and the patient's family can review after the conversation has ended.

[0005] However, in a typical conversation log, conversations that are unrelated to the main topic, such as communication problems, are also recorded as they are, which can lead to misunderstandings when the person in question or their family members check the conversation log at a later date.

[0006] Therefore, there is a need for a call processing system that can create a conversation log that does not cause misunderstandings by inserting annotations, etc. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Publication No. 2020-149072 Summary of the Invention [Problem to be solved by the invention]

[0008] The problem to be solved by the embodiments of the present invention is to provide a program and an information processing method that can create a conversation log with annotations inserted therein. [Means for solving the problem]

[0009] In one embodiment, the program is capable of causing a computer to perform the following functions: displaying a conversation log based on calls between multiple terminals; obtaining information regarding annotations to be set on spoken sentences included in the conversation log based on user operation; and displaying an updated conversation log based on the information regarding the annotations. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is an external view illustrating a telemedicine booth included in an information processing system according to an embodiment. [Figure 2] FIG. 2 is a block diagram illustrating an information processing system according to the embodiment. [Figure 3] FIG. 3 is a diagram illustrating an example of a data structure of conversation log information according to the embodiment. [Figure 4] FIG. 4 is a schematic diagram illustrating an example of a conversation log image according to the embodiment. [Figure 5] FIG. 5 is a schematic diagram illustrating an example of an editing instruction image according to the embodiment. [Figure 6] FIG. 6 is a schematic diagram illustrating an example of a notification image according to the embodiment. [Figure 7] FIG. 7 is a schematic diagram illustrating an example of a conversation log image according to the embodiment. [Figure 8] FIG. 8 is a schematic diagram illustrating an example of an edited input image according to the embodiment. [Figure 9] FIG. 9 is a schematic diagram illustrating an example of a notification image according to the embodiment. [Figure 10] FIG. 10 is a schematic diagram illustrating an example of a notification image according to the embodiment. [Figure 11] FIG. 11 is a schematic diagram illustrating an example of a conversation log image according to the embodiment. [Figure 12] FIG. 12 is a sequence diagram illustrating an example of a procedure for information processing by the information processing system according to the embodiment. [Figure 13] FIG. 13 is a flowchart illustrating an example of a procedure for information processing by the server according to the embodiment. [Figure 14] FIG. 14 is a flowchart illustrating an example of a procedure for information processing by the second communication device according to the embodiment. [Figure 15] FIG. 15 is a flowchart showing another example of the procedure of information processing by the second communication device according to the embodiment. [Figure 16] FIG. 16 is a flowchart illustrating an example of a procedure for information processing by the first communication device according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] (Embodiment) (Configuration example) Hereinafter, an embodiment will be described with reference to the drawings.

[0012] In the drawings, the same components are denoted by the same reference numerals whenever possible, and redundant explanations will be omitted. FIG. 1 is an external view illustrating a telemedicine booth included in an information processing system 100 according to an embodiment. A remote consultation booth is a facility surrounded by walls, preventing conversations from being overheard and ensuring privacy during consultations. A remote consultation booth is also called a consultation booth or simply a booth. Inside the booth, chairs, desks, and other equipment are installed, along with medical equipment for measurements and a communication device for communicating with a doctor outside the booth. For example, multiple consultation booths are installed in each area, such as a city, ward, town, or village. A patient makes a reservation for a remote consultation at a consultation booth in advance and enters the reserved booth at the scheduled time. The patient communicates with the doctor via a video call or other means via the communication device in the consultation booth, and uses medical equipment based on the doctor's instructions to measure biometric data and other data. Measurement data from the medical equipment can be viewed in real time on the doctor's equipment or other devices used by the doctor. The doctor can then perform a consultation based on the measurement data. Each consultation booth may be installed for each hospital, or may be used by multiple doctors from multiple hospitals.

[0013] FIG. 2 is a block diagram illustrating an information processing system 100 according to the embodiment. The information processing system 100 includes a server 1, a second communication device 4, and at least one remote medical examination booth. The remote medical examination booth includes a remote medical examination support device 2 and a first communication device 3. The server 1, the remote medical examination support device 2, the first communication device 3, and the second communication device 4 are communicably connected to each other via a network. For example, the network may be one or more of various networks such as the Internet, a mobile communication network, and a LAN (Local Area Network). The one or more networks may include a wireless network or a wired network. The remote medical examination support device 2 and the first communication device 3 are communicably connected to each other via a network. The network is a LAN or the like. The LAN may be a wireless LAN or a wired LAN. The remote medical examination support device 2 is communicably connected to at least one medical device. The remote medical examination support device 2 and at least one medical device 251-1 to 251-m are directly communicably connected via a wired or wireless connection. The remote medical care support device 2 and at least one of the medical devices 251-1 to 251-m may be connected via, for example, a LAN, Bluetooth (registered trademark), Wi-Fi (registered trademark), etc. Note that the information processing system 100 may also refer to a system including the server 1, the remote medical care support device 2, the first communication device 3, the second communication device 4, and at least two of the medical devices 251-1 to 251-m.

[0014] The server 1 is an electronic device that collects and processes data. The electronic device includes a computer. The server 1 is communicatively connected to a remote medical consultation support device 2, a first communication device 3, and a second communication device 4 via a network. The server 1 receives various data from the remote medical consultation support device 2, the first communication device 3, and the second communication device 4, and outputs various data to the remote medical consultation support device 2, the first communication device 3, and the second communication device 4. The server 1 may be a server used in a cloud service.

[0015] The server 1 may provide a call service for performing video communication or the like between the first communication device 3 and the second communication device 4. The call service may be a call service based on voice communication. The call service may not involve communication using video images. The server 1 may provide a remote medical consultation service between the remote medical consultation support device 2, the first communication device 3, and the second communication device 4. An example configuration of the server 1 will be described later.

[0016] The remote medical care support device 2 is an electronic device capable of communicating with other electronic devices. The remote medical care support device 2 is a device installed in a medical examination booth. For example, the remote medical care support device 2 is a PC (Personal Computer), a smartphone, or a tablet terminal. The participant may be interpreted as a user or a person. An example configuration of the remote medical care support device 2 will be described later.

[0017] The first communication device 3 is an electronic device capable of communicating with other electronic devices. The first communication device 3 is a device installed in a medical examination booth. The first communication device 3 is, for example, a device used by a patient receiving remote medical treatment. For example, the first communication device 3 is a PC, a smartphone, a tablet terminal, or the like. The patient may be read as a user or a person. An example configuration of the first communication device 3 will be described later. The first communication device 3 is an example of a terminal.

[0018] The second communication device 4 is an electronic device capable of communicating with other electronic devices. The second communication device 4 is, for example, a device used by a doctor who performs remote medical treatment. For example, the second communication device 4 is a PC, a smartphone, a tablet terminal, or the like. The doctor may be interpreted as a medical professional, a user, or a person. The second communication device 4 is installed, for example, in a location different from the examination booth. The location where the examination booth is installed is an example of a first base. The location where the second communication device 4 is installed is an example of a second base. The second base is, for example, a medical institution such as a hospital. An example configuration of the second communication device 4 will be described later. The second communication device 4 is an example of a terminal.

[0019] An example of the configuration of the server 1 will be described. The server 1 is an electronic device that includes a processor 11, a main memory 12, an auxiliary storage device 13, and a communication interface 14. The components that make up the server 1 are connected to each other so that signals can be input and output. In FIG. 1, the interface is indicated as "I / F."

[0020] The processor 11 corresponds to the central part of the server 1. The processor 11 is an element that constitutes the computer of the server 1. For example, the processor 11 is a CPU (Central Processing Unit), but is not limited to this. The processor 11 may be composed of various circuits. The processor 11 loads a program that is pre-stored in the main memory 12 or the auxiliary storage device 13 into the main memory 12. The program is a program that causes the processor 11 of the server 1 to realize or execute each unit described below. The processor 11 performs various operations by executing the program loaded into the main memory 12.

[0021] The main memory 12 corresponds to the main storage portion of the server 1. The main memory 12 is an element that constitutes the computer of the server 1. The main memory 12 includes a non-volatile memory area and a volatile memory area. The main memory 12 stores an operating system or programs in the non-volatile memory area. The main memory 12 uses the volatile memory area as a work area where data is rewritten by the processor 11 as appropriate. For example, the main memory 12 includes a ROM (Read Only Memory) as a non-volatile memory area. For example, the main memory 12 includes a RAM (Random Access Memory) as a volatile memory area. The main memory 12 stores programs.

[0022] The auxiliary storage device 13 corresponds to the auxiliary storage portion of the server 1. The auxiliary storage device 13 is an element that constitutes the computer of the server 1. The auxiliary storage device 13 is an EEPROM (registered trademark) (Electric Erasable Programmable Read-Only Memory), an HDD (Hard Disc Drive), an SSD (Solid State Drive), or the like. The auxiliary storage device 13 stores the above-mentioned programs, data used by the processor 11 when performing various processes, and data generated by the processes in the processor 11. The auxiliary storage device 13 stores the above-mentioned programs. The auxiliary storage device 13 is an example of a storage unit.

[0023] The auxiliary storage device 13 stores conversation log information. The conversation log information is information recording a conversation between a user of a first communication device and a user of a second communication device. The conversation log information includes at least speaker information and conversation information. The speaker information is, for example, speaker identification information that can identify the speaker. The speakers include the user of the first communication device and the user of the second communication device. The speaker identification information includes a user ID, a speaker name, etc. The user ID is information that can individually identify each speaker. The speaker name may be the speaker's name or username, etc. The speaker identification information may be information that indicates the speaker's attributes, such as "doctor" or "patient." The conversation information is associated with the speaker identification information. The conversation information indicates the content of the utterance spoken by the speaker. The conversation information is information obtained by performing speech recognition processing on speech information related to the utterance. Speech recognition includes, for example, converting the speech information related to the utterance into text data and segmenting it using known technology. The conversation information includes at least one spoken sentence. The spoken sentence is, for example, segmented based on divisions in the user's speech. The spoken sentence may include identification information that identifies each spoken sentence. The spoken sentence includes at least one word. A word is the smallest linguistic unit that has meaning. A word includes parts of speech such as nouns, verbs, and exclamations.

[0024] The conversation log information will be described with reference to FIG. FIG. 3 shows a data structure of conversation log information according to an embodiment. FIG. FIG. 3 shows conversation log information when a conversation takes place between a doctor and a patient. The conversation log information has records linked to a conversation log ID, speaker identification information, date and time, annotation flag, conversation information, etc. The conversation log shows data in which spoken sentences are arranged in chronological order. The conversation log ID is information that can individually identify each spoken sentence. Spoken sentences are also called conversation sentences. The conversation log ID may also be information that can individually identify each segment. The speaker identification information may include a user ID and a speaker name. In the example of FIG. 3, the speaker name is "doctor" or "patient." The speaker identification information is linked to each spoken sentence. The speaker identification information may be information based on information that can identify the communication device that output the audio information. The speaker identification information may be information based on information that can identify the user of the communication device that output the audio information. The date and time indicate speech time information. The speech time information may include time information indicating the time when the speech was made, or elapsed time information indicating the time elapsed since the conversation started. In the example of Figure 3, the date and time indicates time information indicating the time when the speech was made. The time when the speech was made may correspond to the time when the server 1 obtained the call information from the first communication device 3 or the second communication device 4. The conversation log information may also be simply referred to as log information.

[0025] The annotation flag is information indicating whether an annotation has been set based on a user operation. The annotation indicates additional information to be recorded for a selected utterance sentence or range of utterance sentences when the user selects an utterance sentence or range of utterance sentences from the conversation log. The range of utterance sentences may include multiple utterance sentences. The range of utterance sentences may be part of a single utterance sentence. The additional information includes text information input based on a user operation. The additional information may include information such as images, audio, and video. The annotation includes information about the utterance sentence. The information about the utterance sentence includes information such as descriptions, references, and explanations to be added to the utterance sentence. The information about the utterance sentence includes information explaining the content, etc., of the utterance sentence. The information about the utterance sentence includes information explaining the situation, etc., of the utterance sentence. For example, when a conversation is held about a topic different from the topic of the conversation, the annotation is information explaining the different topic. For example, the annotation may be set on the utterance sentence or range of the utterance sentence selected by the user.

[0026] A case will be described where the topic of a conversation between a user of a first communication device 3 and a user of a second communication device 4 is "about treatment." Topics different from the topic of the conversation "about treatment" include, for example, "about the topic and another conversation," "about communication troubles," "about different symptoms," etc. For example, assume that a communication trouble occurs during a conversation between a user of a first communication device 3 and a user of a second communication device 4. In this case, the conversation about the communication trouble is a conversation with a different topic. The different topic could be "about the topic and another conversation," or "about communication troubles," etc. It may be assumed that the user sets an annotation to indicate that the conversation about the communication trouble is a conversation with a different topic from the topic of the conversation.

[0027] The annotation may include the reason for the conversation, such as "due to communication trouble," "due to a conversation about a different symptom," etc. The reason for the conversation may be the same as or different from the topic of the conversation.

[0028] An annotation flag can be set for each utterance sentence or for each range of utterance sentences. The annotation flag includes "YES" indicating that an annotation has been set, and "NO" indicating that an annotation has not been set. The annotation flag may be 1-bit information. The annotation flag can be set by the user selecting an utterance sentence or a range of utterance sentences. The user selects an utterance sentence or a range of utterance sentences to set an annotation. Setting an annotation flag may be interpreted as adding an annotation.

[0029] Conversation information is, for example, text information indicating the content of an utterance. Conversation information is linked to each utterance sentence. Conversation information may include a main text and annotation information. The main text indicates information obtained by performing speech recognition processing based on audio information related to the utterance. Annotation information is information related to annotations input based on user operations. The information related to annotations includes at least one of information indicating the content of the annotation and information indicating the reason for the annotation. User operations include operations to edit utterance sentences on a conversation log screen on which the conversation log is displayed. Editing operations include operations to select an edit button or the like by touch input or the like on the screen on which the conversation log is displayed. The edit button is a button operated by the user. The edit button is a software key. The edit button is an operator for instructing editing. The edit button may include an annotation setting button and an annotation deletion button. Editing operations include operations to set an annotation and an annotation deletion operation. The operation of setting an annotation includes an operation of selecting an utterance sentence or a range of an utterance sentence, an operation of inputting text information, an operation of rewriting part of an annotation, an operation of duplicating part or all of an annotation, an operation of moving part or all of an annotation, etc. The operation of deleting an annotation includes an operation of selecting an utterance sentence or a range of an utterance sentence, an operation of deleting text information, an operation of deleting part or all of an annotation, etc.

[0030] In the example of FIG. 3, the conversation log ID "1001" is a sentence spoken by a doctor with a user ID of "A." The time when the sentence with the conversation log ID "1001" was spoken was "17:00:35." The annotation flag for the conversation log ID "1001" indicates that an annotation has been set for the sentence with the conversation log ID "1001." The conversation content of the sentence with the conversation log ID "1001" is "Are you OK?" When an annotation is set by the user for the sentence with the conversation log ID "1001," information about the annotation is saved in the annotation information field. For example, the example of FIG. 3 indicates that the user set information about the annotation for the sentences with the conversation log ID "1001" and the conversation log ID "1002" as "Content: conversation other than the topic, Reason: communication trouble." Information about the annotation is not limited to the above example and may be input arbitrarily by the user.

[0031] The user who sets the annotation may include the user of the first communication device 3 and the user of the second communication device 4. The user who sets the annotation may be a pre-set user. For example, the user who sets the annotation may be only the user of the second communication device 4. The user who sets the annotation may be a doctor in remote medical care. The user who sets the annotation may be set as appropriate by the speaker, administrator, setter, etc. In the following example, the user who sets the annotation will be described as being the user (doctor) of the second communication device 4.

[0032] The conversation log information is information in which at least a conversation log ID, speaker identification information, date and time, an annotation flag, and conversation information are linked together.

[0033] The auxiliary storage device 13 may store information about the examination booth. The information about the examination booth includes information that can identify the examination booth. The information about the examination booth may also include information about medical equipment installed in the examination booth.

[0034] The auxiliary storage device 13 may store user information. The user information is information about users (patients) who use the examination booth. The user information includes user identification information. The user identification information is unique identification information assigned to each user to individually identify the user. The user information may include information such as the user's location information and medical history information. The user information may include measurement data from medical equipment.

[0035] The communication interface 14 includes various interfaces that connect the server 1 to other electronic devices via a network in accordance with a predetermined communication protocol so that they can communicate with each other.

[0036] The hardware configuration of the server 1 is not limited to the above configuration, and the server 1 allows the omission or modification of the above components and the addition of new components as appropriate.

[0037] Each unit implemented in the above-mentioned processor 11 will now be described. The processor 11 realizes a call information acquisition unit 110, a log creation unit 111, a storage control unit 112, a log update unit 113, and an output unit 114. Each unit realized in the processor 11 can also be referred to as each function. Each unit realized in the processor 11 can also be referred to as being realized in a control unit including the processor 11 and the main memory 12. The processor 11 is an example of a processing circuit.

[0038] The call information acquisition unit 110 acquires call information from the first communication device 3 and the second communication device 4 via the communication interface 14. The call information includes voice information. The voice information includes voice information related to the user's speech. The voice information may include information indicating voice characteristics. The voice characteristics may include voice pitch, timbre, etc. The call information includes speaker information and speaking time information. The call information acquisition unit 110 may acquire biometric data of the user from the first communication device 3 and the second communication device 4. The biometric data may be measurement data obtained by medical devices 251-1 to 251-m. The biometric data may be data measured by sensors connected to the first communication device 3 and the second communication device 4.

[0039] The log creation unit 111 creates a conversation log based on the call information. The log creation unit 111 creates the conversation log in chronological order based on audio information related to the user's utterance, speaker information, and speech time information. Information related to the conversation log created by the log creation unit 111 is also referred to as conversation log information. The conversation log information is also simply referred to as a conversation log. The log creation unit 111 may stop creating the conversation log based on an annotation setting instruction for setting an annotation. The log creation unit 111 may resume creating the conversation log based on an annotation confirmation instruction for confirming the annotation.

[0040] The storage control unit 112 stores the information acquired by the call information acquisition unit 110 in the auxiliary storage device 13. The storage control unit 112 stores the conversation log information. The storage control unit 112 updates the conversation log information. The storage control unit 112 sets or updates an annotation flag based on an annotation flag set by the log update unit 113 described below. The storage control unit 112 stores the conversation log information updated by the log update unit 113 in the auxiliary storage device 13. The storage control unit 112 updates the conversation log information.

[0041] The log update unit 113 acquires an annotation setting instruction from the second communication device 4 via the communication interface 14. The annotation setting instruction includes an instruction to set an annotation for an utterance sentence or a range of an utterance sentence included in the conversation log, and an instruction to delete an annotation set for an utterance sentence. Setting an annotation is also referred to as inserting an annotation. The log update unit 113 acquires information about the annotation from the second communication device 4 via the communication interface 14. The information about the annotation includes information indicating the range in which the annotation is set. The range in which the annotation is set includes an utterance sentence or a range of an utterance sentence. The range in which the annotation is set is also referred to as the annotation target. The information about the annotation includes information indicating the content of the annotation. The content of the annotation may include a subject of the utterance sentence that is the annotation target. The information about the annotation may include the reason for the annotation. The information about the annotation may include information about the time the annotation was set. The information about the annotation may include information about the user who set the annotation. The log update unit 113 sets or updates an annotation flag based on acquiring the annotation setting instruction. The log update unit 113 may receive an instruction to edit the annotation from the second communication device 4 via the communication interface 14. The log update unit 113 updates the conversation log information based on the information about the annotation. The log update unit 113 may also update the conversation log information based on an instruction to confirm the annotation.

[0042] The output unit 114 outputs the conversation log information to the first communication device 3 and the second communication device 4 via the communication interface 14. The output unit 114 may output the conversation log information every time the conversation log information is created. The output unit 114 may output the conversation log information based on a conversation log display request from the first communication device 3 and the second communication device 4. The output unit 114 outputs updated conversation log information to the first communication device 3 and the second communication device 4 via the communication interface 14. The output unit 114 may output the updated conversation log information every time the conversation log information is updated. The output unit 114 may output the updated conversation log information based on a display request from the first communication device 3 and the second communication device 4 for the updated conversation log.

[0043] The output unit 114 outputs information indicating various notifications to the first communication device 3 and the second communication device 4 via the communication interface 14. The output unit 114 outputs various requests to the first communication device 3 and the second communication device 4 via the communication interface 14. The output unit 114 outputs information related to annotations to the first communication device 3 via the communication interface 14.

[0044] An example of the configuration of the remote medical care support device 2 will be described. The remote medical consultation support device 2 is an electronic device including a processor 21, a main memory 22, an auxiliary storage device 23, a communication interface 24, an input / output interface 25, a display device 26, a speaker 27, and an input device 28. The components constituting the remote medical consultation support device 2 are connected to each other so that signals can be input and output.

[0045] The processor 21 corresponds to the central part of the remote medical consultation support device 2. The processor 21 is an element that constitutes the computer of the remote medical consultation support device 2. For example, the processor 21 has the same hardware configuration as the above-mentioned processor 11. The processor 21 performs various operations by executing programs deployed in the main memory 22. The processor 21 is an example of a processing circuit.

[0046] The main memory 22 corresponds to the main storage portion of the remote medical consultation support device 2. The main memory 22 is an element that constitutes the computer of the remote medical consultation support device 2. The main memory 22 has the same hardware configuration as the above-mentioned main memory 12. The main memory 22 stores programs.

[0047] The auxiliary storage device 23 corresponds to the auxiliary storage portion of the remote medical consultation support device 2. The auxiliary storage device 23 is an element that constitutes the computer of the remote medical consultation support device 2. The auxiliary storage device 23 has the same hardware configuration as the above-mentioned auxiliary storage device 13. The auxiliary storage device 23 stores the above-mentioned programs.

[0048] The communication interface 24 includes various interfaces that connect the remote medical consultation support device 2 to other devices so that they can communicate with each other via a network in accordance with a predetermined communication protocol.

[0049] The input / output interface 25 is an interface for connecting the remote medical consultation support device 2 with external devices. The external devices include at least one medical device 251-1 to 251-m (m is an integer equal to or greater than 1). The medical devices 251-1 to 251-m include, for example, an electrocardiograph, a blood pressure monitor, a digital stethoscope, a pulse oximeter, a slit lamp, an external otoscope, etc. The medical devices 251-1 to 251-m have a communication function and output measurement data to the remote medical consultation support device 2.

[0050] The display device 26 is a device capable of displaying various screens under the control of the processor 21. For example, the display device 26 is a liquid crystal display or an EL display.

[0051] The speaker 27 is a device that can output sound under the control of the processor 21. The speaker 27 is an example of an output device.

[0052] The input device 28 is a device that can input data or instructions to the remote medical consultation support apparatus 2. For example, the input device 28 includes a built-in microphone that can input voice and a built-in camera that can acquire image data of the image capture range. The input device 28 may also include a keyboard, a touch panel, or the like.

[0053] The hardware configuration of the remote medical care support device 2 is not limited to the above-described configuration. The remote medical care support device 2 allows the omission or modification of the above-described components and the addition of new components as appropriate.

[0054] An example of the configuration of the first communication device 3 will be described. The first communication device 3 is an electronic device including a processor 31, a main memory 32, an auxiliary storage device 33, a communication interface 34, an input / output interface 35, and an input device 38. The components constituting the first communication device 3 are connected to each other so that signals can be input and output to and from each other.

[0055] The processor 31 corresponds to the central part of the first communication device 3. The processor 31 is an element that constitutes the computer of the first communication device 3. The processor 31 has the same hardware configuration as the above-mentioned processor 11. The processor 31 performs various operations by executing programs that are pre-stored in the main memory 32 or the auxiliary storage device 33. The processor 31 is an example of a processing circuit.

[0056] The main memory 32 corresponds to the main storage portion of the first communication device 3. The main memory 32 is an element that constitutes the computer of the first communication device 3. The main memory 32 has the same hardware configuration as the above-mentioned main memory 12. The main memory 32 stores programs.

[0057] The auxiliary storage device 33 corresponds to the auxiliary storage portion of the first communication device 3. The auxiliary storage device 33 is an element that constitutes the computer of the first communication device 3. The auxiliary storage device 33 has the same hardware configuration as the above-mentioned auxiliary storage device 13. The auxiliary storage device 33 stores the above-mentioned programs.

[0058] The communication interface 34 includes various interfaces that connect the first communication device 3 to other devices so that they can communicate with each other via a network in accordance with a predetermined communication protocol.

[0059] The input / output interface 35 is an interface for connecting the first communication device 3 with external devices. The external devices include a display device 351-1, a camera 351-2, a microphone 351-3, and a speaker 351-4. The display device 351-1 is a device capable of displaying various screens under the control of the processor 31. For example, the display device 351-1 is a liquid crystal display or an EL display. The camera 351-2 is a device capable of acquiring image capture data of the image capture range under the control of the processor 31. The microphone 351-3 is a device capable of inputting audio under the control of the processor 31. The speaker 351-4 is a device capable of outputting audio under the control of the processor 31.

[0060] The input device 38 is a device that can input data or instructions to the first communication device 3. For example, the input device 38 includes a keyboard, a touch panel, or the like.

[0061] The hardware configuration of the first communication device 3 is not limited to the above configuration, and the first communication device 3 may appropriately omit or modify the above components and add new components.

[0062] An example of the configuration of the second communication device 4 will be described. The second communication device 4 is an electronic device including a processor 41, a main memory 42, an auxiliary storage device 43, a communication interface 44, an input / output interface 45, and an input device 48. The components constituting the second communication device 4 are connected to each other so as to be able to input and output signals to and from each other.

[0063] The processor 41 corresponds to the central part of the second communication device 4. The processor 41 is an element that constitutes the computer of the second communication device 4. The processor 41 has the same hardware configuration as the above-mentioned processor 11. The processor 41 performs various operations by executing programs that are pre-stored in the main memory 42 or the auxiliary storage device 43. The processor 41 is an example of a processing circuit.

[0064] The main memory 42 corresponds to the main storage portion of the second communication device 4. The main memory 42 is an element that constitutes the computer of the second communication device 4. The main memory 42 has the same hardware configuration as the above-mentioned main memory 12. The main memory 42 stores programs.

[0065] The auxiliary storage device 43 corresponds to the auxiliary storage portion of the second communication device 4. The auxiliary storage device 43 is an element that constitutes the computer of the second communication device 4. The auxiliary storage device 43 has the same hardware configuration as the above-mentioned auxiliary storage device 13. The auxiliary storage device 43 stores the above-mentioned programs.

[0066] The communication interface 44 includes various interfaces that connect the second communication device 4 to other devices so that they can communicate with each other via a network in accordance with a predetermined communication protocol.

[0067] The input / output interface 45 is an interface for connecting the second communication device 4 with external devices. The external devices include a display device 451-1, a camera 451-2, a microphone 451-3, and a speaker 451-4. The display device 451-1 is a device capable of displaying various screens under the control of the processor 41. For example, the display device 451-1 is a liquid crystal display or an EL display. The camera 451-2 is a device capable of acquiring image capture data of the image capture range under the control of the processor 41. The microphone 451-3 is a device capable of inputting audio under the control of the processor 41. The speaker 451-4 is a device capable of outputting audio under the control of the processor 41.

[0068] The input device 48 is a device that can input data or instructions to the second communication device 4. For example, the input device 48 includes a keyboard, a touch panel, or the like.

[0069] The hardware configuration of the second communication device 4 is not limited to the above-described configuration, and the second communication device 4 may appropriately omit or modify the above-described components and add new components.

[0070] Each unit implemented in the above-mentioned processor 41 will now be described. The processor 41 realizes a log processing unit 410 and a display processing unit 411. Each unit realized by the processor 41 can also be referred to as each function. Each unit realized by the processor 41 can also be referred to as being realized by a control unit including the processor 41 and the main memory 42.

[0071] The log processing unit 410 outputs call information to the server 1 via the communication interface 44. The log processing unit 410 acquires conversation log information from the server 1 via the communication interface 44. The log processing unit 410 acquires updated conversation log information from the server 1 via the communication interface 44.

[0072] The log processing unit 410 performs processing related to editing the conversation log. The log processing unit 410 acquires an annotation setting instruction based on a user operation. The user operation includes a touch input operation such as an edit button. The log processing unit 410 outputs an annotation setting instruction to the server 1 via the communication interface 44. The log processing unit 410 acquires information about the annotation based on the user operation. The information about the annotation includes information indicating the range in which the annotation is to be set. The range in which the annotation is to be set can be specified by the user selecting an utterance sentence by a touch operation or the like, or selecting a range of an utterance sentence by a slide operation, swipe operation, or the like on the conversation log screen on which the conversation log is displayed. The information about the annotation can be arbitrarily input by a user operation. The user operation includes an operation of inputting information about the annotation by text input or the like. The log processing unit 410 outputs the information about the annotation to the server 1 via the communication interface 44. The log processing unit 410 outputs an annotation confirmation instruction to confirm the annotation via the communication interface 44.

[0073] The log processing unit 410 outputs various requests to the server 1 via the communication interface 44. The requests include a stop request to request a restart to resume. The requests include a resume request to resume recording of the conversation log.

[0074] The log processing unit 410 acquires information regarding various notifications from the server 1 via the communication interface 44. The notifications include notifications regarding recording of conversation logs. The log processing unit 410 acquires various requests from the server 1 via the communication interface 44. The requests include confirmation requests for confirming that annotations are reflected.

[0075] (Display example) Display examples of the conversation log and the annotation log will be described. FIG. 4 is a schematic diagram illustrating an example of a conversation log image according to the embodiment.

[0076] FIG. 4 shows a conversation log image Ima. The conversation log image Ima may be displayed on the display device 351-1 of the first communication device 3 and the display device 451-1 of the second communication device 4. The conversation log image Ima is a conversation log image displayed on a communication device used by a user who can set annotations in the conversation log. A user who can set annotations indicates a user who has annotation setting authority. The annotation setting authority may be set in advance by, for example, an administrator, a host of a call between the communication devices, a setter, etc. The annotation setting authority may be updated as appropriate by, for example, an administrator, a host of a call between the communication devices, a setter, etc. The annotation setting authority may be granted to at least one of the speakers. The annotation setting authority may also be granted to all of the speakers. In the following example, the annotation setting authority is granted to the user of the second communication device 4.

[0077] The conversation log image Ima is displayed in real time each time a conversation log is created. Note that the conversation log image Ima may be displayed based on a conversation log display request from the user of the second communication device 4.

[0078] The conversation log image Ima includes at least speaker identification marks Ma and Mb, a spoken sentence La, and an edit button Ba. The speaker identification marks Ma and Mb indicate marks that can identify the speaker. The speaker identification marks Ma and Mb may include the speaker's name. In the example of FIG. 4, the speaker identification mark Ma indicates "doctor" or "patient." The speaker identification mark Ma may have a different shape, color, or size for each speaker. The spoken sentence La is text information indicating the content of the utterance. The spoken sentence La may include a balloon-shaped frame, for example, as shown in FIG. 4. The spoken sentence La is arranged in chronological order based on the utterance time information. The edit button Ba is a button selected by the user via touch input or the like. The edit button Ba is, for example, an operator that can input an annotation setting instruction. In response to the user selecting the edit button Ba, the display processing unit 411 displays the edit instruction image shown in FIG. 5 on the display device 451-1. In response to the user selecting the edit button Ba, the log processing unit 410 may output an annotation setting instruction to the server 1. In response to the user selecting the edit button Ba, the processor 41 may transition to the edit mode.

[0079] Note that the conversation log image Ima excluding the edit button Ba may be displayed on the first communication device 3. The conversation log image may be displayed on the first communication device 3 in real time each time a conversation log is created. The conversation log image may be displayed based on a conversation log display request from the user of the first communication device 3.

[0080] FIG. 5 is a schematic diagram illustrating an example of an editing instruction image according to the embodiment.

[0081] FIG. 5 shows an editing instruction image Imb. The editing instruction image Imb is displayed on the display device 451-1 of the second communication device 4. The editing instruction image Imb is displayed on the display device 451-1 in response to the user selecting the edit button Ba in the conversation log image Ima. The editing instruction image Imb may be superimposed on or combined with the conversation log image Ima. The editing instruction image Imb may be displayed on the display device 451-1 as a pop-up display. In the following description, "in response" may be read as "based on."

[0082] The editing instruction image Imb includes an annotation setting button Bb and an annotation deletion button Bc. The annotation setting button Bb is an operator that can input an instruction to set an annotation. The annotation deletion button Bc is an operator that can input an instruction to delete an annotation. To set an annotation, the user selects the annotation setting button Bb by touch input or the like. To delete an annotation, the user selects the annotation deletion button Bc by touch input or the like.

[0083] FIG. 6 is a schematic diagram illustrating an example of a notification image according to the embodiment. 6 shows a notification image Imc. The notification image Imc is displayed on the display device 451-1 of the second communication device 4. For example, the notification image Imc is displayed on the display device 451-1 in response to the user selecting the annotation setting button Bb or the annotation deletion button Bc in the edit instruction image Imb. The notification image Imc may also be displayed on the display device 451-1 in response to the user selecting the edit button Ba in the conversation log image Ima. The notification image Imc may be superimposed on or combined with the conversation log image Ima. The notification image Imc may also be displayed on the display device 451-1 as a pop-up display.

[0084] The notification image Imc includes text information indicating that recording of the conversation log will be stopped while the annotation process is being performed. The text information includes, for example, information such as "The conversation during this time will not be recorded." The annotation process includes, for example, processing from when the user inputs an annotation setting instruction to when information related to the annotation is recorded in or updated in the conversation log information. The annotation process includes processing in edit mode. The notification image Imc can be displayed on the display device 351-1 of the first communication device 3 and the display device 451-1 of the second communication device 4.

[0085] This allows the user of the first communication device 3 and the user of the second communication device 4 to know that their conversation during the annotation process will not be recorded.

[0086] FIG. 7 is a schematic diagram illustrating an example of a conversation log image according to the embodiment. 7 shows a conversation log image Imd. The conversation log image Imd is displayed on the display device 451-1 of the second communication device 4. The conversation log image Imd may be displayed on a communication device used by a user who has annotation setting authority.

[0087] Like the conversation log image Ima, the conversation log image Imd includes at least a speaker identification mark and a spoken sentence. The conversation log image Imd also includes an annotation target area Ara. The annotation target area Ara indicates a spoken sentence or a range of spoken sentences to which an annotation selected by the user is to be set. In this example, multiple spoken sentences are selected as annotation targets by the user's slide operation. The range of annotation targets is not limited to selection by a slide operation, but may also be selected for each spoken sentence by a touch operation or the like. The range of annotation targets may be displayed in a manner that allows the range to be recognized as other spoken sentences. The display processing unit 411 may display the range of annotation targets in a different display manner from other spoken sentences. The display manner may include character decoration such as character color, size, font, brightness or luminance, background color, background brightness or luminance, background pattern, frame color, or blinking display. In this example, the display processing unit 411 displays the range of annotation targets by enclosing it with a dashed line. The range of annotation targets corresponds to information indicating the range to be annotated.

[0088] FIG. 8 is a schematic diagram illustrating an example of an edited input image according to the embodiment. 8 shows an edited input image Ime. The edited input image Ime is displayed on the display device 451-1 of the second communication device 4. The edited input image Ime may be displayed on a communication device used by a user authorized to set annotations. The edited input image Ime may be displayed on the display device 451-1 in response to the user selecting a range to be annotated.

[0089] The edited input image Ime includes an area in which information about the annotation can be input. The area in which information about the annotation can be input may include an annotation content input area Arb in which the annotation content can be input, and an annotation purpose input area Arc in which the annotation purpose can be input. The annotation content input area Arb is an area in which the annotation content can be arbitrarily input. The annotation purpose input area Arc is an area in which the annotation purpose can be arbitrarily input. In this example, the edited input image Ime shows that the user has input "conversation other than the main topic" as the annotation content in the annotation content input area Arb, and "communication trouble" as the annotation purpose in the annotation purpose input area Arc.

[0090] The edit input image Ime includes an OK button Bd and a cancel button Be. The OK button Bd is an operator that allows the user to input information indicating that the user has confirmed the input content. The cancel button Be is an operator that allows the user to input information indicating that the user wants to cancel the input content. In response to the user selecting the OK button Bd by touch input or the like, the log processing unit 410 outputs information regarding the annotation to the server 1. In response to the user selecting the cancel button Be by touch input or the like, the display processing unit 411 may transition the displayed image from the edit input image Ime to a conversation log image Imd. Note that in response to the user selecting the cancel button Be by touch input or the like, the display processing unit 411 may transition the displayed image to a conversation log image Ima or an edit instruction image Imb.

[0091] FIG. 9 is a schematic diagram illustrating an example of a notification image according to the embodiment. FIG. 9 shows a notification image Imf. The notification image Imf is displayed on the display device 351-1 of the first communication device 3. The notification image Imf may be displayed on a communication device used by a user other than the user with annotation setting authority. A user other than the user with annotation setting authority is, for example, a user who can accept annotations. A user who can accept annotations indicates a user with annotation acceptance authority. The annotation acceptance authority may be updated as appropriate by an administrator, a host of a call between communication devices, a setter, etc. The annotation acceptance authority may be granted to at least one of the speakers, for example. The annotation acceptance authority may also be granted to a user other than the speaker. In the following example, the annotation acceptance authority is assumed to be granted to the user of the first communication device 3.

[0092] The notification image Imf includes information notifying that the server 1 has acquired information related to the annotation. The notification image Imf may be superimposed on or combined with the conversation log image. The notification image Imf may be displayed on the display device 351-1 as a pop-up display.

[0093] The notification image Imf includes a notification area Ard, an annotation area Are, an OK button Bf, and a reject button Bg. The notification area Ard includes information indicating that an annotation has been set. The notification area Ard may include information related to the annotation. The notification area Ard includes text information such as, "An annotation has been added to the following conversation record. Content: Conversation other than the main topic, Reason: Communication trouble." The annotation area Are includes information indicating the range of the annotation target. The annotation area Are indicates the range of the annotation target in the conversation log in a recognizable manner. In this example, the spoken sentences included in the range of the annotation target are indicated by dashed speech bubbles.

[0094] The OK button Bf is an operator that allows the user to input an acceptance instruction indicating that the user accepts the annotation. Accepting the annotation includes at least one of accepting the annotation to be set and accepting the content of the annotation. The reject button Bg is an operator that allows the user to input a rejection instruction indicating that the user rejects the annotation. Rejecting the annotation includes at least one of refusing to set the annotation and rejecting the content of the annotation. The acceptance instruction and the rejection instruction are examples of information regarding acceptance. In response to the user selecting the OK button Bf by touch input or the like, the processor 31 outputs information indicating that the annotation is accepted to the server 1. Information regarding the annotation is output to the server 1. In response to the user selecting the reject button Bg by touch input or the like, the processor 31 outputs information indicating that the annotation is rejected to the server 1.

[0095] According to this example, the information processing system 100 can confirm whether or not users other than the user who set the annotation have permission to set the annotation and the content of the annotation. Users other than the user who set the annotation can express their intention to set the annotation and whether or not they have permission to set the annotation and the content of the annotation. This allows the information processing system 100 to reach an agreement regarding the annotation between the person who set the annotation and other users. In this way, the information processing system 100 can create a conversation log in which annotations are inserted based on the intentions of the users.

[0096] FIG. 10 is a schematic diagram illustrating an example of a notification image according to the embodiment. FIG. 10 shows a notification image Img. The notification image Img is displayed on the display device 451-1 of the second communication device 4. The notification image Img may be displayed on a communication device used by a user who has the authority to set an annotation. The notification image Img includes information notifying that the server 1 has acquired information indicating that the annotation is accepted. The notification image Img may be superimposed or combined with the conversation log image. The notification image Img may be displayed on the display device 451-1 as a pop-up display.

[0097] The notification image Img includes a notification area Arf and an OK button Bh. The notification area Arf includes information indicating that the annotation has been accepted. The notification area Arf includes text information such as, for example, "The annotation content has been accepted. The annotation content will be reflected." The OK button Bh is an operator that allows the user to input a confirmation instruction indicating that the annotation content will be reflected. In response to the user selecting the OK button Bh by touch input or the like, the log processing unit 410 outputs information indicating that the annotation content will be reflected to the server 1. The notification image Img is an example of a confirmation request.

[0098] According to this example, a user can confirm that a user other than the user who set the annotation has set an annotation and that the user has consented to the content of the annotation. The user can input an instruction to confirm the annotation based on the consent of the user other than the user who set the annotation. This allows the information processing system 100 to create a conversation log with annotations inserted based on the intentions of the users.

[0099] FIG. 11 is a schematic diagram illustrating an example of a conversation log image according to the embodiment. FIG. 11 shows a conversation log image Imh. The conversation log image Imh can be displayed on the display device 351-1 of the first communication device 3 and the display device 451-1 of the second communication device 4. The conversation log image Imh includes an annotation area Arg and an annotation target area Arh. The annotation area Arg can include information related to the annotation. For example, the annotation area Arg includes text information such as "conversation other than the main topic (reason: communication trouble)." The text information is not limited to the display format shown in FIG. 11. The annotation target area Arh indicates the range of the annotation target in a recognizable manner. In this example, spoken sentences included in the range of the annotation target are indicated by dashed line speech bubbles.

[0100] The display processing unit 411 displays the annotation in a recognizable manner based on the annotation area Arg and the annotation target area Arh. Displaying the annotation in a recognizable manner includes displaying at least one of the range in which the annotation is set and the content of the annotation in a recognizable manner.

[0101] According to this example, the information processing system 100 can display the presence or absence of annotations and the contents of the annotations on the display screen of the conversation log in a recognizable manner, thereby allowing the user to easily recognize the presence or absence of annotations and the contents of the annotations.

[0102] (Example of operation) The processing procedure performed by the information processing system 100 will be described. In the following description focusing on server 1, server 1 may be read as processor 11. In the following description focusing on first communication device 3, first communication device 3 may be read as processor 31. In the following description focusing on second communication device 4, second communication device 4 may be read as processor 41. The processing procedures described below are merely examples, and each process may be modified as much as possible. Furthermore, steps may be omitted, replaced, or added as appropriate depending on the embodiment.

[0103] FIG. 12 is a sequence diagram showing an example of a procedure of information processing by the information processing system 100 according to the embodiment.

[0104] The following process is assumed to be initiated when a call is started between the first communication device 3 and the second communication device 4. In the following example, it is assumed that annotation setting authority is set for the user of the second communication device 4. It is assumed that annotation acceptance authority is set for the user of the first communication device 3.

[0105] The first communication device 3 and the second communication device 4 output the call information to the server 1. The server 1 acquires the call information. The server 1 creates conversation log information based on the call information. The server 1 outputs the conversation log information to the first communication device 3 and the second communication device 4.

[0106] Based on the user's operation, the second communication device 4 outputs an annotation setting instruction to the server 1. The second communication device 4 outputs a stop request to the server 1 to stop recording the conversation log.

[0107] The server 1 acquires an annotation setting instruction from the second communication device 4. The server 1 acquires a stop request from the second communication device 4. Based on acquiring the stop request, the server 1 outputs information related to the notification to the first communication device 3 and the second communication device 4. The notification is a notification indicating that recording of the conversation log will be stopped. The server 1 stops recording the conversation log. The server 1 may also stop creating conversation log information.

[0108] The first communication device 3 and the second communication device 4 acquire information about the notification from the server 1. The first communication device 3 and the second communication device 4 display a notification image on a display device based on the information about the notification.

[0109] Based on a user operation, the second communication device 4 outputs information related to the annotation to the server 1. The second communication device 4 may output a stop request after outputting the information related to the annotation to the server 1. Based on receiving the stop request, the server 1 may output information related to the notification to the first communication device 3 and the second communication device 4.

[0110] The server 1 acquires information about the annotation from the second communication device 4. The server 1 outputs the information about the annotation to the first communication device 3. The first communication device 3 acquires the information about the annotation from the server 1. The first communication device 3 displays a notification image on the display device 351-1 based on the information about the annotation. The notification image includes information indicating that the annotation has been set.

[0111] Based on a user operation, the first communication device 3 outputs an acceptance instruction indicating acceptance of the content of the annotation to the server 1. Note that the first communication device 3 may also output a rejection instruction indicating rejection of the content of the annotation to the server 1 based on a user operation.

[0112] The server 1 acquires an acceptance instruction from the first communication device 3. The server 1 outputs a confirmation request to the second communication device 4 based on the acceptance instruction. The confirmation request is a request to confirm that the annotation will be reflected. If the server 1 acquires a rejection instruction from the first communication device 3, the server 1 may output a confirmation request to the second communication device 4 based on the rejection instruction. In this case, the confirmation request is, for example, a request to confirm that the annotation content will not be reflected. The confirmation request may include text information such as, for example, "The annotation content has been rejected. The annotation content will not be reflected."

[0113] Based on the user's operation, the second communication device 4 outputs a confirmation instruction to the server 1. The second communication device 4 outputs to the server 1 a resume request to resume recording of the conversation log.

[0114] The server 1 receives a confirmation instruction to confirm the annotation setting from the second communication device 4. The server 1 updates the conversation log information based on the confirmation instruction. The server 1 adds information about the annotation to the conversation log information. The server 1 resumes recording the conversation log based on a resume request. The server 1 may resume creating the conversation log information.

[0115] The server 1 outputs the updated conversation log information to the first communication device 3 and the second communication device 4. The first communication device 3 and the second communication device 4 display a conversation log image on a display device based on the updated conversation log information.

[0116] The above process may be repeated until the call between the first communication device 3 and the second communication device 4 is completed.

[0117] According to this example, the information processing system 100 can set annotations for utterances included in a conversation log based on a user's operation. The user can add information to the conversation log that cannot be recognized from the utterances alone. In this way, the information processing system 100 can create a conversation log with annotations inserted. The information processing system 100 can assist in creating a conversation log that will not cause misunderstandings to viewers of the conversation log, for example, when viewing the conversation log after the conversation has ended. Furthermore, the information processing system 100 can display the presence or absence of annotations and the contents of the annotations on the display screen of the conversation log in a recognizable manner. This allows the user to easily recognize the presence or absence of annotations and the contents of the annotations. Furthermore, the information processing system 100 can reflect the annotation settings in the conversation log information when an instruction of consent is received from a user other than the user who set the annotation. Therefore, the information processing system 100 can set annotations in the conversation log based on the consent of the other users.

[0118] FIG. 13 is a flowchart showing an example of a procedure for information processing by the server 1 according to the embodiment.

[0119] The following processing is assumed to start when a call between the first communication device 3 and the second communication device 4 is started.

[0120] The call information acquisition unit 110 acquires call information from the first communication device 3 and the second communication device 4. (ACT 1) In ACT 1, for example, the call information acquisition unit 110 acquires voice information from the first communication device 3 and the second communication device 4. The voice information may include speaker identification information for identifying the user of the first communication device 3 or the second communication device 4. The storage control unit 112 may store the acquired call information in the auxiliary storage device 13.

[0121] The log creation unit 111 creates conversation log information based on the call information (ACT2). In ACT2, for example, the log creation unit 111 performs speech recognition processing based on audio information related to the user's utterance. The log creation unit 111 segments text information based on the audio information related to the user's utterance for each spoken sentence. The log creation unit 111 links the segmented text information to speaker identification information. The log creation unit 111 creates conversation log information including the spoken sentences linked to the speaker identification information. The log creation unit 111 may create conversation log information including speech time information related to each spoken sentence.

[0122] The storage control unit 112 stores the conversation log information in the auxiliary storage device 13 (ACT 3).

[0123] The output unit 114 outputs the conversation log information to at least one of the first communication device 3 and the second communication device 4 (ACT4). At least one of the first communication device 3 and the second communication device 4 that has acquired the conversation log information displays the conversation log on a display device based on the conversation log information.

[0124] The log update unit 113 determines whether or not an annotation setting instruction has been received from the second communication device 4 (ACT5). If the log update unit 113 has received an annotation setting instruction (ACT5: YES), the process proceeds from ACT5 to ACT6. If the log update unit 113 has not received an annotation setting instruction (ACT5: NO), the process proceeds from ACT5 to ACT9.

[0125] The log update unit 113 acquires information about the annotation from the second communication device 4 (ACT6). In ACT6, for example, the log update unit 113 acquires information indicating the range in which the annotation is to be set and information indicating the content of the annotation. The log update unit 113 saves the information about the annotation. The output unit 114 outputs the information about the annotation to the first communication device 3. Note that the log creation unit 111 stops creating the conversation log information based on having acquired a stop request to stop recording the conversation log from the second communication device 4.

[0126] The log update unit 113 updates the conversation log information based on the information about the annotation (ACT7). In this example, it is assumed that an acceptance instruction indicating acceptance of the information about the annotation is output from the first communication device 3. In ACT7, for example, the log update unit 113 acquires the acceptance instruction from the first communication device 3. Based on the acquisition of the acceptance instruction, the output unit 114 outputs a confirmation request to the second communication device 4. The log update unit 113 acquires a confirmation instruction from the second communication device 4. Based on the confirmation instruction, the log update unit 113 adds the information about the annotation to the conversation log information.

[0127] The output unit 114 outputs the updated conversation log information to the first communication device 3 and the second communication device 4 (ACT8). Note that the log creation unit 111 resumes creation of the conversation log information based on receiving a resume request to resume recording of the conversation log from the second communication device 4.

[0128] The processor 11 determines whether the call between the first communication device 3 and the second communication device 4 has ended (ACT9). If the processor 11 determines that the call has ended (ACT9: YES), the processing ends. If the processor 11 determines that the call has not ended (ACT9: NO), the processing transitions from ACT9 to ACT1. The processor 11 repeats the processing of ACT1 to ACT8 until the call between the first communication device 3 and the second communication device 4 has ended.

[0129] Processor 11 may perform the processes from ACT2 to ACT8 after the call between the first communication device 3 and the second communication device 4 has ended. For example, call information acquisition unit 110 may acquire call information while the call between the first communication device 3 and the second communication device 4 is ongoing. Log creation unit 111 may create conversation log information after the call between the first communication device 3 and the second communication device 4 has ended.

[0130] In addition, the output unit 114 may output the conversation log information to at least one of the first communication device 3 and the second communication device 4 based on the conversation log display request. In this example, the processor 11 acquires the conversation log display request from at least one of the first communication device 3 and the second communication device 4. For example, the processor 11 acquires the conversation log display request based on a user operation of the user of the first communication device 3 or the second communication device 4. The user operation may include, for example, a click operation or a touch operation on a display button to display the conversation log displayed on the display device. Based on the conversation log display request, the output unit 114 may output the conversation log information to at least one of the first communication device 3 and the second communication device 4, which are the source of the conversation log display request.

[0131] The conversation log display request may be output from an electronic device other than the first communication device 3 and the second communication device 4. For example, the conversation log display request may be output from a user terminal (not shown). In this case, the output unit 114 may output the conversation log information to the user terminal that has output the conversation log display request.

[0132] FIG. 14 is a flowchart showing an example of a procedure for information processing by the second communication device 4 according to the embodiment.

[0133] The following process is assumed to be started when a call between the first communication device 3 and the second communication device 4 is started. The following process may also be started when a conversation log is displayed on at least one of the first communication device 3 and the second communication device 4. In the following example, it is assumed that annotation setting authority is set for the user of the second communication device 4. The following process illustrates the process when an annotation is set by the user of the second communication device 4.

[0134] The display processing unit 411 displays a conversation log based on a call between the first communication device 3 and the second communication device 4 on the display device 451-1.

[0135] The log processing unit 410 acquires an instruction to set an annotation based on a user operation (ACT 101). In ACT 101, for example, the log processing unit 410 acquires an instruction to set an annotation based on a user operation.

[0136] The log processing unit 410 outputs an instruction to the server 1 to set an annotation (ACT 102). The log processing unit 410 outputs a stop request to the server 1.

[0137] The display processing unit 411 displays the notification on the display device 451-1 (ACT103). In ACT103, for example, the log processing unit 410 acquires information about the notification indicating that recording of the conversation log will be stopped from the server 1. The display processing unit 411 displays a notification image including information indicating that recording of the conversation log will be stopped on the display device 451-1 based on the information about the notification.

[0138] The log processing unit 410 acquires information about an utterance sentence included in the conversation log or an annotation to be set in a range of an utterance sentence based on a user operation (ACT104). In ACT104, for example, the log processing unit 410 acquires information indicating a range in which an annotation is to be set based on a user operation. The log processing unit 410 acquires information indicating the content of the annotation based on a user operation.

[0139] The log processing unit 410 outputs information related to the annotation to the server 1 (ACT105). Note that the log processing unit 410 may output a stop request to the server 1 in ACT105. In this case, after ACT105, the display processing unit 411 may display a notification image on the display device 451-1 based on the information related to the notification.

[0140] The log processing unit 410 determines whether or not a confirmation request for confirming that the annotation content is to be reflected has been received from the server 1 (ACT106). If the log processing unit 410 determines that a confirmation request has been received (ACT106: YES), the process proceeds from ACT106 to ACT107. If the log processing unit 410 determines that a confirmation request has not been received (ACT106: NO), the process ends. In this case, the log processing unit 410 may also receive a confirmation request for confirming that the annotation content is not to be reflected.

[0141] The display processing unit 411 displays a notification on the display device 451-1 based on a confirmation request for finalizing the reflection of the annotation content (ACT107). In ACT107, for example, the display processing unit 411 displays a notification image on the display device 451-1 based on the confirmation request. The notification image includes information confirming that the annotation content is to be reflected. Note that, when the log processing unit 410 acquires a confirmation request for confirming that the annotation content is not to be reflected, the display processing unit 411 may display a notification image including information confirming that the annotation content is not to be reflected on the display device 451-1.

[0142] The log processing unit 410 outputs a confirmation instruction to the server 1 (ACT108). In ACT108, for example, the log processing unit 410 acquires a confirmation instruction for confirming that the annotation content is to be reflected in the conversation log information based on a user operation. The log processing unit 410 outputs the confirmation instruction to the server 1. The log processing unit 410 outputs a resume request to the server 1 to resume recording of the conversation log. The server 1 resumes recording of the conversation log. The server 1 may output information indicating that recording of the conversation log will be resumed to the second communication device 4. In this case, the display processing unit 411 may display a notification image including information indicating that recording of the conversation log will be resumed on the display device 451-1.

[0143] The log processing unit 410 acquires the conversation log information updated based on the information about the annotation from the server 1 (ACT 109).

[0144] The display processing unit 411 displays the updated conversation log on the display device 451-1 based on the updated conversation log information (ACT110). The display processing unit 411 displays the annotation on the display device 451-1 in a recognizable manner. Displaying the annotation in a recognizable manner includes displaying at least one of the range in which the annotation is set and the content of the annotation in a recognizable manner.

[0145] The processor 41 may perform the above-described process after the call between the first communication device 3 and the second communication device 4 is completed.

[0146] According to this example, the second communication device 4 can set annotations for spoken sentences included in the conversation log based on a user's operation. The user can add information to the conversation log that cannot be recognized from the spoken sentences alone. In this way, the second communication device 4 can create a conversation log with annotations inserted. The second communication device 4 can assist in creating a conversation log that will not cause misunderstandings to viewers of the conversation log, for example, when viewing the conversation log after the conversation has ended. Furthermore, the second communication device 4 can display the presence or absence of annotations and the contents of the annotations in a recognizable manner on the display screen of the conversation log. This allows the user of the second communication device 4 to easily recognize the presence or absence of annotations and the contents of the annotations. Furthermore, the second communication device 4 can reflect the annotation settings in the conversation log information when it receives an instruction of consent from another user regarding the setting of the annotations. This allows the user of the second communication device 4 to obtain consent from another user regarding the setting of the annotations.

[0147] FIG. 15 is a flowchart showing another example of the procedure of information processing by the second communication device 4 according to the embodiment.

[0148] The following process is assumed to be started when a call between the first communication device 3 and the second communication device 4 is started. The following process may also be started when a conversation log is displayed on at least one of the first communication device 3 and the second communication device 4. In the following example, it is assumed that annotation setting authority is set for the user of the second communication device 4. The following process illustrates the process when an annotation is deleted by the user of the second communication device 4.

[0149] The display processing unit 411 displays a conversation log based on a call between the first communication device 3 and the second communication device 4 on the display device 451-1.

[0150] The log processing unit 410 acquires an instruction to set an annotation based on a user's operation (ACT 201). In ACT 201, for example, the log processing unit 410 acquires an instruction to delete an annotation based on a user's operation.

[0151] The log processing unit 410 outputs an annotation setting instruction to the server 1 (ACT202). The log processing unit 410 may output a stop request to the server 1. When the log processing unit 410 outputs the stop request, the display processing unit 411 may display a notification on the display device 451-1, similar to ACT103.

[0152] The log processing unit 410 acquires information regarding the deletion of an utterance sentence included in the conversation log or an annotation set in a range of an utterance sentence based on a user operation (ACT203). In ACT203, for example, the log processing unit 410 acquires information indicating a range from which the annotation is to be deleted based on a user operation. The log processing unit 410 acquires information indicating the content of the deletion based on the user operation. The content of the deletion may include information indicating that part or all of the annotation is to be deleted.

[0153] The log processing unit 410 outputs information related to the deletion to the server 1 (ACT204). Note that the server 1 may output the information related to the deletion to the first communication device 3. The server 1 may obtain an instruction of consent for the information related to the deletion from the first communication device 3.

[0154] The display processing unit 411 displays a notification on the display device 451-1 to confirm that the deletion content is to be reflected (ACT205). In ACT205, for example, the log processing unit 410 acquires a confirmation request from the server 1. The display processing unit 411 displays the notification on the display device 451-1 based on the confirmation request.

[0155] The log processing unit 410 outputs a confirmation instruction to the server 1 (ACT206). In ACT206, for example, the log processing unit 410 acquires a confirmation instruction for confirming that the deletion of the annotation will be reflected in the conversation log information based on a user operation. The log processing unit 410 outputs the confirmation instruction to the server 1. The log processing unit 410 may output a resume request to the server 1 to resume recording of the conversation log. The server 1 resumes recording of the conversation log. The server 1 may output information indicating that recording of the conversation log will be resumed to the second communication device 4. In this case, the display processing unit 411 may display a notification image including information indicating that recording of the conversation log will be resumed on the display device 451-1.

[0156] The log processing unit 410 acquires the conversation log information updated based on the information regarding the deletion of the annotation from the server 1 (ACT 207).

[0157] The display processing unit 411 displays the updated conversation log on the display device 451-1 based on the updated conversation log information (ACT208).

[0158] The processor 41 may perform the above-described process after the call between the first communication device 3 and the second communication device 4 is completed.

[0159] FIG. 16 is a flowchart showing an example of a procedure for information processing by the first communication device 3 according to the embodiment.

[0160] In the following example, it is assumed that annotation setting authority is set for the user of the second communication device 4. It is assumed that annotation acceptance authority is set for the user of the first communication device 3. It is assumed that the following processing is started when an instruction to set an annotation is output from the second communication device 4 to the server 1. It is assumed that the processor 31 is displaying a conversation log based on a call between the first communication device 3 and the second communication device 4 on the display device 351-1.

[0161] The following process shows a process when an annotation is set by the user of the second communication device 4. Note that the following process can also be applied when an annotation is deleted by the user of the second communication device 4.

[0162] The processor 31 displays a notification (ACT301). In ACT301, for example, the processor 31 acquires information about the notification from the server 1. The processor 31 acquires information about the notification indicating that recording of the conversation log will be stopped from the server 1. Based on the information about the notification, the processor 31 displays a notification image including information indicating that recording of the conversation log will be stopped on the display device 351-1.

[0163] The processor 31 acquires information about the annotation from the server 1 (ACT 302). In ACT 302, for example, the processor 31 acquires from the server 1 information indicating the range in which the annotation is set and information indicating the content of the annotation.

[0164] The processor 31 displays a notification (ACT303). In ACT303, for example, the processor 31 displays a notification image including information indicating that the annotation has been set on the display device 351-1 based on information about the annotation. The notification image may include a button or the like that can instruct whether or not to accept the annotation content.

[0165] The processor 31 determines whether or not an acceptance instruction to accept the annotation content has been acquired based on the user's operation (ACT304). If the processor 31 determines that an acceptance instruction has been acquired (ACT304: YES), the process transitions from ACT304 to ACT305. If the processor 31 determines that an acceptance instruction has not been acquired (ACT304: NO), the process transitions from ACT304 to ACT308. The fact that an acceptance instruction has not been acquired may be interpreted as the fact that a rejection instruction indicating that the annotation content is rejected has been acquired.

[0166] The processor 31 outputs an acceptance instruction to the server 1 (ACT305). Based on the acceptance instruction, the server 1 outputs a confirmation request to the second communication device 4. The server 1 acquires a confirmation instruction for confirming the annotation setting from the second communication device 4. Based on the confirmation instruction, the server 1 updates the conversation log information.

[0167] The processor 31 acquires the updated conversation log information from the server 1 (ACT 306).

[0168] The processor 31 displays the updated conversation log on the display device 351-1 based on the updated conversation log information (ACT307). The processor 31 displays the annotation on the display device 351-1 in a recognizable manner. Displaying the annotation in a recognizable manner includes displaying at least one of the range in which the annotation is set and the content of the annotation in a recognizable manner.

[0169] Server 1 may output information indicating that recording of the conversation log will be resumed to first communication device 3. In this case, processor 31 may display, on display device 351-1, a notification image including information indicating that recording of the conversation log will be resumed.

[0170] The processor 31 may perform the above-described process after the call between the first communication device 3 and the second communication device 4 is completed.

[0171] According to this example, the first communication device 3 can display the presence or absence of annotations and the contents of the annotations on the display screen of the conversation log in a recognizable manner. This allows the user of the first communication device 3 to easily recognize the presence or absence of annotations and the contents of the annotations. Furthermore, the first communication device 3 can output an acceptance instruction to the server 1 for annotations set by other users based on the user's operation. The user can indicate whether or not they accept the annotations.

[0172] (Other embodiments) In the above-described embodiment, an example has been described in which the examination booth includes the remote medical consultation support device 2 and the first communication device 3, but this is not limited to this. The examination booth may include either the remote medical consultation support device 2 or the first communication device 3. When the examination booth does not include the first communication device 3 but includes the remote medical consultation support device 2, the remote medical consultation support device 2 may realize the functions of the first communication device 3. When the examination booth does not include the remote medical consultation support device 2 but includes the first communication device 3, the first communication device 3 may realize the functions of the remote medical consultation support device 2.

[0173] In the above-described embodiment, a call between a doctor and a patient in a remote medical consultation has been described as an example, but the present invention is not limited to this. The above-described embodiment can be applied to calls between users of multiple communication devices in web conferences, video conferences, distance learning, and the like.

[0174] The information processing device may be realized by one device such as the server 1, or may be realized by a plurality of devices with distributed functions.

[0175] The above-described embodiments may be applied not only to an apparatus but also to a method executed by the apparatus. The above-described embodiments may be applied to a program that can cause a computer of the apparatus to execute each function. The above-described embodiments may be applied to a recording medium that stores a program. The above-described embodiments may be applied not only to a system but also to a method executed by multiple elements included in the system.

[0176] The processing circuitry includes one or more circuits that perform multiple processes for multiple functions, such as, but not limited to, a processor, an ASIC (Application Specific Integrated Circuit), or an FPGA (Field-Programmable Gate Array).

[0177] Each of the one or more circuits constituting the processing circuit performs one or more of the multiple processes. When the processing circuit is composed of a single circuit, the single circuit performs all of the multiple processes. When the processing circuit is composed of multiple circuits, each of the multiple circuits performs a part of the multiple processes. The part of the multiple processes may be one of the multiple processes, or two or more of the multiple processes. When the processing circuit is composed of multiple circuits, the multiple circuits may be included in a single device or may be distributed across multiple devices.

[0178] The program may be transferred in a state where it is stored in a device, or in a state where it is not stored in a device. In the latter case, the program may be transferred via a network, or in a state where it is recorded on a recording medium. The recording medium is a non-transitory tangible medium. The recording medium is a computer-readable medium. The form of the recording medium is not important as long as it is a medium that can store the program and is computer-readable, such as a CD-ROM or a memory card.

[0179] Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations according to the embodiments may be appropriately adopted.

[0180] In short, this invention is not limited to the above-described embodiments, and the components can be modified and embodied in practice without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0181] (Addendum) The above embodiment may be expressed as follows: (1) To the computer, The ability to view conversation logs based on calls between multiple devices, a function of acquiring information about annotations to be set on utterances included in the conversation log based on user operations; displaying an updated conversation log based on information about the annotations; A program that can be executed. (2) To the computer, The program according to (1), further capable of executing a function of outputting a stop request for stopping recording of the conversation log based on an operation by the user. (3) To the computer, The program according to (1), further capable of executing a function of outputting a resume request for resuming recording of the conversation log based on an operation by the user. (4) The program described in (1) can further execute a function to output a confirmation request to confirm that the annotation is reflected based on obtaining consent information for the annotation from another terminal. (5) The program according to (1), wherein the function of displaying the updated conversation log includes a function of displaying the annotations in a recognizable manner. (6) displaying a conversation log based on calls between multiple devices; and acquiring information about annotations to be set on utterances included in the conversation log based on a user operation; displaying an updated conversation log based on information about the annotations; and An information processing method comprising: [Explanation of symbols]

[0182] 1...server, 2...remote medical care support device, 3...first communication device, 4...second communication device, 11...processor, 12...main memory, 13...auxiliary storage device, 14...communication interface, 21...processor, 22...main memory, 23...auxiliary storage device, 24...communication interface, 25...input / output interface, 26...display device, 27...speaker, 28...input device, 31...processor, 32...main memory, 33...auxiliary storage device, 34...communication interface, 35...input / output interface, 38...input device, 41...processor, 42...main memory, 43...auxiliary storage device, 44...communication interface, 45...input / output interface, 48...input device, 100...information processing system, 110...call information acquisition unit, 111...log creation unit, 112...memory control unit, 113...log update unit, 114...output unit, 251-1 to 251-m...medical device, 351- 1...display device, 351-2...camera, 351-3...microphone, 351-4...speaker, 410...log processing unit, 411...display processing unit, 451-1...display device, 451-2...camera, 451-3...microphone, 451-4...speaker, Ara...annotation target area, Arb...annotation content input area, Arc...annotation purpose input area, Ard...notification area, Are...annotation area, Arf...notification area, Arg...annotation area, Arh...annotation target area A, Ba...Edit button, Bb...Annotation setting button, Bc...Annotation deletion button, Bd...OK button, Be...Cancel button, Bf...OK button, Bg...Reject button, Bh...OK button, Ima...Conversation log image, Imb...Edit instruction image, Imc...Notification image, Imd...Conversation log image, Ime...Edit input image, Imf...Notification image, Img...Notification image, Imh...Conversation log image, La...Spoken text, Ma...Speaker identification mark, Mb...Speaker identification mark.

Claims

1. On the computer, The ability to view conversation logs based on calls between multiple devices, a function of acquiring information about annotations to be set on utterances included in the conversation log based on user operations; displaying an updated conversation log based on information about the annotations; a function of outputting a confirmation request for confirming that the annotation is to be reflected based on information regarding consent for the annotation obtained from another terminal; A program that can be executed.

2. On the computer, 2. The program according to claim 1, further capable of executing a function of outputting a stop request for stopping recording of the conversation log based on an operation by the user.

3. On the computer, 2. The program according to claim 1, further capable of executing a function of outputting a resume request for resuming recording of the conversation log based on an operation by the user.

4. The program of claim 1 , wherein the function of displaying the updated conversation log includes a function of displaying the annotations in a recognizable manner.

5. Displaying a conversation log based on calls between multiple devices; acquiring information about annotations to be set on utterances included in the conversation log based on a user operation; displaying an updated conversation log based on information about the annotations; and outputting a confirmation request for confirming that the annotation is to be reflected based on the information regarding the consent for the annotation obtained from the other terminal; An information processing method comprising:

Images