Program, displaying method and terminal

Abstract

To facilitate an input of alphabetic information on the basis of sound information into a chat room.SOLUTION: A program that is executed by a user terminal which displays a chat room involving a user and a user of a first terminal different from the user causes the terminal to execute: obtaining sound information by an obtaining unit of the terminal; and causing alphabetic information on the basis of the sound information obtained by the obtaining unit to be input into a selected chat room, and causing the chat room containing the alphabetic information to be displayed on a display unit of the terminal.SELECTED DRAWING: Figure 1-1

Description

[Technical field]

[0001] The present disclosure relates to a program, a display method, and a terminal. [Background technology]

[0002] 2. Description of the Related Art Information processing systems relating to sound recognition (eg, voice recognition) are known (eg, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2014-206961 A Summary of the Invention

[0004] According to a first aspect of the present invention, a program executed by a user's terminal displaying a chat room including the user and a user of a first terminal different from the user is executed by the terminal to acquire sound information by an acquisition unit of the terminal, input text information based on the sound information acquired by the acquisition unit into a selected chat room, and display the chat room including the text information on a display unit of the terminal. According to a second aspect of the present invention, a display method of a user's terminal displaying a chat room including the user and a user of a first terminal different from the user includes acquiring sound information by an acquisition unit of the terminal, inputting character information based on the sound information acquired by the acquisition unit into a selected chat room, and displaying the chat room including the character information on a display unit of the terminal. According to a third aspect of the present invention, a user terminal displaying a chat room including the user and a user of a first terminal different from the user includes an acquisition unit that acquires sound information, and a display unit that inputs text information based on the sound information acquired by the acquisition unit into a selected chat room and displays the chat room including the text information. [Brief description of the drawings]

[0005] [Figure 1-1] FIG. 1 is a diagram showing an example of the configuration of a communication system according to an embodiment. [Figure 1-2] 4 is a flowchart showing an example of the flow of processes executed by each device according to the first embodiment. [Figure 2-1] FIG. 11 is a diagram showing an example of information stored in a storage unit of a server according to the second embodiment. [Figure 2-2] FIG. 11 is a diagram showing an example of messaging application user registration data according to the second embodiment. [Figure 2-3] FIG. 11 is a diagram showing an example of a group management database according to the second embodiment. [Figure 2-4] FIG. 11 is a diagram showing an example of a calendar information management database according to the second embodiment. [Figure 2-5] FIG. 11 is a diagram showing an example of information stored in a storage unit of a terminal according to a second embodiment. [Figure 2-6] FIG. 11 is a diagram showing an example of a screen displayed on a display unit of a terminal according to a second embodiment. [Figure 2-7] FIG. 11 is a diagram showing an example of a screen displayed on a display unit of a terminal according to a second embodiment. [Figure 2-8] FIG. 11 is a diagram showing an example of a screen displayed on a display unit of a terminal according to a second embodiment. [Figure 2-9] FIG. 11 is a diagram showing an example of a screen displayed on a display unit of a terminal according to a second embodiment. [Figure 2-10] FIG. 11 is a diagram showing an example of a screen displayed on a display unit of a terminal according to a second embodiment. [Figure 2-11] FIG. 11 is a diagram showing an example of a screen displayed on a display unit of a terminal according to a second embodiment. [Figure 2-12] FIG. 11 is a diagram showing an example of a screen displayed on a display unit of a terminal according to a second embodiment. [Figure 2-13] FIG. 11 is a diagram showing an example of a screen displayed on a display unit of a terminal according to a second embodiment. [Figure 2-14] FIG. 11 is a diagram showing an example of a screen displayed on a display unit of a terminal according to a second embodiment. [Figure 2-15] FIG. 11 is a diagram showing an example of a screen displayed on a display unit of a terminal according to a second embodiment. [Figure 2-16]10 is a flowchart showing an example of the flow of processes executed by each device according to the second embodiment. [Figure 3-1] FIG. 11 is a diagram showing an example of a screen displayed on a display unit of a terminal according to a third embodiment. [Figure 3-2] FIG. 11 is a diagram showing an example of a screen displayed on a display unit of a terminal according to a third embodiment. [Figure 3-3] FIG. 11 is a diagram showing an example of a screen displayed on a display unit of a terminal according to a third embodiment. [Diagram 3-4] FIG. 11 is a diagram showing an example of a screen displayed on a display unit of a terminal according to a third embodiment. [Figure 3-5] FIG. 11 is a diagram showing an example of a screen displayed on a display unit of a terminal according to a third embodiment. [Diagram 3-6] FIG. 11 is a diagram showing an example of a screen displayed on a display unit of a terminal according to a third embodiment. [Diagram 3-7] FIG. 11 is a diagram showing an example of a screen displayed on a display unit of a terminal according to a third embodiment. [Diagram 3-8] FIG. 11 is a diagram showing an example of a screen displayed on a display unit of a terminal according to a third embodiment. [Diagram 3-9] FIG. 11 is a diagram showing an example of a screen displayed on a display unit of a terminal according to a third embodiment. [Figure 3-10] 13 is a flowchart showing an example of the flow of processes executed by each device according to the third embodiment. [Figure 4-1] FIG. 13 is a diagram showing an example of a screen displayed on a display unit of a terminal according to a fourth embodiment. [Figure 4-2] FIG. 13 is a diagram showing an example of a screen displayed on a display unit of a terminal according to a fourth embodiment. [Figure 4-3] FIG. 13 is a diagram showing an example of a screen displayed on a display unit of a terminal according to a fourth embodiment. [Figure 4-4] FIG. 13 is a diagram showing an example of a screen displayed on a display unit of a terminal according to a fourth embodiment. [Figure 4-5] 13 is a flowchart showing an example of the flow of processes executed by each device according to the fourth embodiment. [Figure 5-1] FIG. 13 is a diagram showing an example of a screen displayed on a display unit of a terminal according to a fifth embodiment. [Figure 6-1] FIG. 23 is a diagram showing an example of a display format table according to the sixth embodiment. [Figure 7-1] FIG. 23 is a diagram showing an example of a calendar information management database according to the seventh embodiment. [Figure 7-2] FIG. 23 is a diagram showing an example of a screen displayed on a display unit of a terminal according to the seventh embodiment. [Figure 7-3] FIG. 23 is a diagram showing an example of a screen displayed on a display unit of a terminal according to the seventh embodiment. [Figure 7-4] FIG. 23 is a diagram showing an example of a screen displayed on a display unit of a terminal according to the seventh embodiment. [Figure 7-5] 13 is a flowchart showing an example of the flow of processes executed by each device according to the fourth embodiment. [Figure 7-6] FIG. 23 is a diagram showing an example of a screen displayed on a display unit of a terminal according to a seventh modified example. [Figure 7-7] FIG. 23 is a diagram showing an example of a screen displayed on a display unit of a terminal according to a seventh modified example. [Figure 7-8] 13 is a flowchart showing an example of the flow of processes executed by each device according to the seventh modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] <Compliance with legal matters> It should be noted that the disclosures described herein are subject to compliance with the laws of the country of implementation, such as communications secrecy, as necessary for the implementation of the disclosure.

[0007] An embodiment for implementing a program or the like according to the present disclosure will be described with reference to the drawings.

[0008] In this specification, the expression "through a communication I / F" is used as appropriate. This indicates that, by way of example and not limitation, a device transmits and receives various types of information and data through a communication I / F (through a communication unit) based on the control of a control unit (such as a processor).

[0009] <System configuration> FIG. 1-1 is a diagram illustrating an example of a system configuration of a communication system 1 according to the present embodiment. In the communication system 1, as an example and not a limitation, a server 10 and a plurality of terminals 20 (terminal 20A, terminal 20B, terminal 20C, . . . ) are connected via a network 30.

[0010] The server 10 has a function of providing a messaging service to a terminal 20 owned by a user via a network 30. The server 10 can also be expressed as a messaging server or a messaging service server.

[0011] The terminal 20 (terminal 20A, terminal 20B, terminal 20C, ...) may be any information processing terminal capable of implementing the functions described in each embodiment. The terminal 20 includes, but is not limited to, a smartphone, a mobile phone (feature phone), a computer (such as, but not limited to, a desktop, laptop, tablet, etc.), a media computer platform (such as, but not limited to, a cable or satellite set-top box, digital video recorder), a handheld computer device (such as, but not limited to, a PDA (personal digital assistant), email client, etc.), a wearable device (such as a glasses-type device, a watch-type device, etc.), a VR (Virtual Reality) terminal, a smart speaker (a device for voice recognition), or other types of computers or communication platforms. The terminal 20 may also be expressed as an information processing terminal.

[0012] The configurations of the terminals 20A, 20B, and 20C are basically the same. In addition, if necessary, the terminal used by the user X may be expressed as the terminal 20X, and the user information in a predetermined service associated with the user X or the terminal 20X may be expressed as the user information X. The user information is information of a user associated with an account used by the user in a specific service. The user information includes, but is not limited to, information associated with a user, such as the user's name, the user's icon image, the user's age, the user's sex, the user's address, the user's hobbies and interests, and the user's identifier, which is input by the user or given by a specific service, and may be any one of these, or a combination of these, or may not be the same.

[0013] The network 30 plays a role of connecting one or more terminals 20 and one or more servers 10. In other words, the network 30 refers to a communication network that provides a connection path so that the above-mentioned various devices can be connected and then transmit and receive data. The number of servers 10 and the number of terminals 20 connected to the network 30 are not limited.

[0014] One or more portions of network 30 may or may not be wired or wireless networks. Network 30 may include, by way of example and not limitation, an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of the Public Switched Telephone Network (PSTN), a cellular network, integrated service digital networks (ISDN), wireless LAN, long term evolution (LTE), code division multiple access (CDMA), Bluetooth, satellite communications, or the like, or a combination of two or more thereof. Network 30 may include one or more networks 30.

[0015] The server 10 (not limited to, but an example of a server, information processing device, or information management device) has a function of providing a predetermined service (messaging service in this embodiment) to the terminal 20. The server 10 may be any device as long as it is an information processing device that can realize the functions described in each embodiment. The server 10 includes, not limited to, a server device, a computer (not limited to, but an example of, a desktop, a laptop, a tablet, etc.), a media computer platform (not limited to, but an example of, a cable, a satellite set-top box, a digital video recorder), a handheld computer device (not limited to, but an example of, a PDA, an email client, etc.), or other types of computers or communication platforms. The server 10 may also be expressed as an information processing device. When it is not necessary to distinguish between the server 10 and the terminal 20, the server 10 and the terminal 20 may or may not be expressed as information processing devices.

[0016] [Hardware (HW) configuration of each device] The hardware configuration of each device included in the communication system 1 will be described.

[0017] (1) Terminal hardware configuration FIG. 1-1 shows an example of the hardware configuration of the terminal 20. In FIG. The terminal 20 includes a control unit 21 (CPU: central processing unit), a storage unit 28, a communication I / F 22 (interface), an input / output unit 23, a clock unit 29A, and a position calculation information detection unit 29B. The HW components of the terminal 20 are connected to each other via a bus B, for example and not for limitation. It is not essential that the HW configuration of the terminal 20 includes all of the components. For example and not for limitation, the terminal 20 may or may not be configured such that individual components or multiple components are removable.

[0018] The communication I / F 22 transmits and receives various data via the network 30. The communication may be performed either wired or wirelessly, and any communication protocol may be used as long as the communication between the two devices is possible. The communication I / F 22 has a function of communicating with various devices such as the server 10 via the network 30. The communication I / F 22 transmits various data to various devices such as the server 10 according to instructions from the control unit 21. The communication I / F 22 also receives various data transmitted from various devices such as the server 10 and transmits it to the control unit 21. The communication I / F 22 may also be simply referred to as a communication unit. When the communication I / F 22 is configured with a physically structured circuit, it may also be referred to as a communication circuit.

[0019] The input / output unit 23 includes a device for inputting various operations to the terminal 20, and a device for outputting the results of processing performed by the terminal 20. The input / output unit 23 may be an integrated input unit and an output unit, or may be separate from the input unit and the output unit, or may not be.

[0020] The input unit is realized by any one or a combination of all kinds of devices that can receive input from a user and transmit information related to the input to the control unit 21. Examples of the input unit include, but are not limited to, a touch panel, a touch display, hardware keys such as a keyboard, a pointing device such as a mouse, a camera (operation input via moving images), and a microphone (operation input by voice).

[0021] The output unit is realized by any one or a combination of all kinds of devices capable of outputting the processing results processed by the control unit 21. Examples of the output unit include, but are not limited to, a touch panel, a touch display, a speaker (audio output), a lens (for example, but not limited to, 3D (three dimensions) output and hologram output), a printer, etc.

[0022] In one embodiment, the input / output unit 23 includes, by way of example and not limitation, a display unit 24, a sound input unit 25, a sound output unit 26, and an imaging unit 27.

[0023] The display unit 24 is realized by any one of all kinds of devices capable of displaying according to the display data written in the frame buffer, or a combination thereof. The display unit 24 includes, but is not limited to, a touch panel, a touch display, a monitor (for example, but not limited to, a liquid crystal display or an organic electroluminescence display (OLED)), a head mounted display (HDM), a projection mapping, a hologram, a device capable of displaying images, text information, and the like in air (which may or may not be a vacuum). Note that these display units 24 may or may not be capable of displaying display data in 3D.

[0024] The sound input unit 25 is used to input sound data (including voice data; the same applies below). The sound input unit 25 includes a microphone and the like. The sound output unit 26 is used to output sound data and includes a speaker and the like. The imaging unit 27 is used to acquire moving image data and includes a camera.

[0025] When the input / output unit 23 is a touch panel, the input / output unit 23 and the display unit 24 may be disposed facing each other and have approximately the same size and shape.

[0026] The clock unit 29A is a built-in clock of the terminal 20, and outputs time information (timekeeping information). The clock unit 29A is configured to include, for example and not limitation, a clock using a crystal oscillator. The clock unit 29A can also be expressed as a timekeeping unit or a time information detection unit, for example and not limitation.

[0027] The clock unit 29A may or may not have a clock that conforms to the Network Identity and Time Zone (NITZ) standard or the like.

[0028] The position calculation information detection unit 29B is a functional unit that detects (measures) information (hereinafter referred to as "position calculation information") necessary for the control unit 21 to calculate (measure) the position of its own terminal 20. The position calculation information detection unit 29B can also be expressed as a position calculation sensor unit, for example and not by way of limitation.

[0029] The position calculation information detection unit 29B includes, by way of example and not limitation, a satellite positioning sensor (satellite positioning unit) which is a sensor or unit for calculating the position of the terminal 20 using a satellite positioning system such as GPS (Global Positioning System), an inertial measurement sensor (inertial measurement unit (IMU (Inertial Measurement Unit))) which is a sensor or unit for calculating the position of the terminal 20 using an inertial navigation system, a UWB positioning sensor (UWB positioning unit) which is a sensor or unit for calculating the position of the terminal 20 using UWB (Ultra Wide Band), and the like.

[0030] The satellite positioning unit includes, by way of example and not limitation, an RF receiving circuit that converts RF (Radio Frequency) signals, including positioning satellite signals transmitted from positioning satellites and received by an antenna (not shown), into digital signals, and a baseband processing circuit that performs correlation calculation processing or the like on the digital signals output from the RF receiving circuit to capture the positioning satellite signals, and outputs information such as satellite orbit data and time data extracted from the positioning satellite signals as information for position calculation.

[0031] The inertial measurement unit has an inertial sensor that is a sensor that detects information required for calculating the position of the terminal 20 by inertial navigation calculation. The inertial sensor includes, but is not limited to, a three-axis acceleration sensor and a three-axis gyro sensor, and outputs the acceleration detected by the acceleration sensor and the angular velocity detected by the gyro sensor as information for position calculation.

[0032] The UWB positioning unit includes, by way of example and not limitation, an ultra-wideband RF receiving circuit that converts an ultra-wideband RF (Radio Frequency) signal, including an ultra-wideband pulse signal for positioning transmitted from a positioning beacon and received by an antenna not shown, into a digital signal, and a relative position calculation processing circuit that calculates the relative position between the terminal 20 and the positioning beacon based on the digital signal output from the ultra-wideband RF receiving circuit. As an example and not by way of limitation, the UWB positioning unit may or may not cause the terminal 20 to function as a positioning beacon by transmitting an ultra-wideband RF signal including an ultra-wideband pulse signal for positioning from an antenna not shown.

[0033] As a non-limiting example, the control unit 21 calculates the position of its own terminal 20 at regular or specific timing based on the position calculation information detected by the position calculation information detection unit 29B. The terminal position is referred to as the "terminal position", and the calculated terminal position is referred to as the "calculated terminal position". Then, the control unit 21 associates the calculated terminal position with the date and time when the calculated terminal position is calculated, and stores the calculated terminal position in the storage unit 28 as calculated terminal position history data.

[0034] The control unit 21 has a circuit physically structured to execute the functions realized by the code or instructions contained in the program, and is realized by a data processing device built into hardware, for example and not by way of limitation. Therefore, the control unit 21 may or may not be expressed as a control circuit.

[0035] The control unit 21 includes, by way of example and not limitation, a central processing unit (CPU), a microprocessor, a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), and a field programmable gate array (FPGA).

[0036] The storage unit 28 has a function of storing various programs and various data required for the operation of the terminal 20. The storage unit 28 includes, by way of example and not limitation, various storage media such as a hard disk drive (HDD), a solid state drive (SSD), a flash memory, a random access memory (RAM), and a read only memory (ROM). In addition, the storage unit 28 may or may not be expressed as a memory.

[0037] Terminal 20 stores program P in storage unit 28, and executes this program P, causing control unit 21 to execute processing as each unit included in control unit 21. In other words, program P stored in storage unit 28 causes terminal 20 to realize each function executed by control unit 21. Also, this program P may or may not be expressed as a program module.

[0038] (2) Server hardware configuration FIG. 1-1 shows an example of the HW configuration of the server 10. In FIG. The server 10 includes a control unit 11 (CPU), a storage unit 15, a communication I / F 14 (interface), an input / output unit 12, and a clock unit 19. The components of the HW of the server 10 are connected to each other via a bus B, for example and not for limitation. It is not essential that the HW of the server 10 includes all the components as the configuration of the HW of the server 10. For example and not for limitation, the HW of the server 10 may or may not be configured such that individual components or multiple components are removable.

[0039] The control unit 11 has circuits physically structured to execute functions realized by codes or instructions contained in a program, and is realized, for example and not by way of limitation, by a data processing device embedded in hardware.

[0040] The control unit 11 is typically a central processing unit (CPU), but may also be a microprocessor, a processor core, a multiprocessor, an ASIC, or an FPGA, but is not limited to these in the present disclosure.

[0041] The storage unit 15 has a function of storing various programs and various data required for the operation of the server 10. The storage unit 15 is realized by various storage media such as an HDD, an SSD, and a flash memory. However, in the present disclosure, the storage unit 15 is not limited to these. Furthermore, the storage unit 15 may or may not be expressed as a memory.

[0042] The communication I / F 14 transmits and receives various data via the network 30. The communication may be performed either wired or wirelessly, and any communication protocol may be used as long as the communication between the two devices is possible. The communication I / F 14 has a function of communicating with various devices such as the terminal 20 via the network 30. The communication I / F 14 transmits various data to various devices such as the terminal 20 according to instructions from the control unit 11. The communication I / F 14 also receives various data transmitted from various devices such as the terminal 20 and transmits it to the control unit 11. The communication I / F 14 may also be simply referred to as a communication unit. When the communication I / F 14 is configured with a physically structured circuit, it may also be referred to as a communication circuit.

[0043] The input / output unit 12 is realized by a device for inputting various operations to the server 10 . The input unit is realized by any one or a combination of all kinds of devices capable of receiving input from a user and transmitting information related to the input to the control unit 11. The input unit is typically realized by hardware keys such as a keyboard or a pointing device such as a mouse. Note that the input unit may or may not include a touch panel, a camera (operation input via moving images), or a microphone (operation input by voice), as examples and without limitation.

[0044] The output unit is realized by any one or a combination of all kinds of devices capable of outputting the processing results processed by the control unit 11. Examples of the output unit include, but are not limited to, a touch panel, a touch display, a speaker (audio output), a lens (for example, but not limited to, 3D (three dimensions) output and hologram output), a printer, etc.

[0045] In the first embodiment, the input / output unit 12 includes a display unit 13 .

[0046] The display unit 13 is typically realized by a monitor (for example, but not limited to, a liquid crystal display or an organic electroluminescence display (OLED)). The display unit 13 may or may not be a head mounted display (HDM) or the like. These display units 13 may or may not be capable of displaying display data in 3D. In the present disclosure, the display unit 13 is not limited to these.

[0047] The clock unit 19 is a built-in clock of the server 10, and outputs time information (timekeeping information). The clock unit 19 is configured to include, for example and not limitation, an RTC (Real Time Clock) as a hardware clock, a system clock, etc. The clock unit 19 can also be expressed as a timekeeping unit or a time information detection unit, for example and not limitation.

[0048] (3)Other The server 10 stores a program P in the storage unit 15, and by executing this program P, the control unit 11 executes the processes of each unit included in the control unit 11. In other words, the program P stored in the storage unit 15 causes the server 10 to realize each function executed by the control unit 11. This program P may or may not be expressed as a program module. The same applies to other devices.

[0049] In each embodiment of the present disclosure, the description will be given assuming that the CPU of the terminal 20 and / or the server 10 executes the program P to realize the present invention. The same applies to other devices.

[0050] The control unit 21 of the terminal 20 and / or the control unit 11 of the server 10 may or may not realize each process by a CPU having a control circuit, but may also realize each process by a logic circuit (hardware) formed in an integrated circuit (IC (Integrated Circuit) chip, LSI (Large Scale Integration)) or a dedicated circuit. These circuits may or may not be realized by one or more integrated circuits, and the multiple processes shown in each embodiment may or may not be realized by one integrated circuit. Furthermore, an LSI may be called a VLSI, a super LSI, an ultra LSI, or the like, depending on the degree of integration. Therefore, the control unit 21 may or may not be expressed as a control circuit. The same applies to other devices.

[0051] In addition, the program P (as an example, but not limited to, a software program, a computer program, or a program module) of each embodiment of the present disclosure may or may not be provided in a state stored in a computer-readable storage medium. The storage medium can store the program P in a "non-transitory tangible medium." In addition, the program P may or may not be for realizing part of the functions of each embodiment of the present disclosure. Furthermore, the program P may or may not be a so-called difference file (difference program) that can realize the functions of each embodiment of the present disclosure in combination with a program P already recorded in a storage medium.

[0052] The storage medium may include one or more semiconductor-based or other integrated circuits (ICs) (such as, by way of example and not limitation, a field programmable gate array (FPGA) or an application specific IC (ASIC)), a hard disk drive (HDD), a hybrid hard drive (HHD), an optical disk, an optical disk drive (ODD), a magneto-optical disk, a magneto-optical drive, a floppy diskette, a floppy disk drive (FDD), a magnetic tape, a solid state drive (SSD), a RAM drive, a secure digital card, or a drive, any other suitable storage medium, or a suitable combination of two or more of these. The storage medium may be volatile, non-volatile, or a combination of volatile and non-volatile, where appropriate. It should be noted that the storage medium is not limited to these examples and may be any device or medium capable of storing the program P. In addition, the storage medium may or may not be referred to as a memory.

[0053] The server 10 and / or the terminal 20 can realize the functions of the multiple functional units shown in each embodiment by reading out the program P stored in a storage medium and executing the read out program P. The same applies to other devices.

[0054] Furthermore, the program P of the present disclosure may or may not be provided to the server 10 and / or the terminal 20 via any transmission medium capable of transmitting a program (such as a communication network or broadcast waves). The server 10 and / or the terminal 20 executes the program P downloaded via the Internet or the like, for example and not for limitation, to realize the functions of the multiple functional units shown in each embodiment. The same applies to other devices.

[0055] In addition, each embodiment of the present disclosure may also be realized in the form of a data signal in which the program P is embodied by electronic transmission. At least a part of the processing in the server 10 and / or the terminal 20 may or may not be realized by cloud computing configured with one or more computers. At least a part of the processing in the terminal 20 may or may not be configured to be performed by the server 10. In this case, at least a part of the processing of each functional unit of the control unit 21 of the terminal 20 may or may not be configured to be performed by the server 10. At least a part of the processing in the server 10 may or may not be configured to be performed by the terminal 20. In this case, at least a part of the processing of each functional unit of the control unit 11 of the server 10 may or may not be configured to be performed by the terminal 20. Unless explicitly stated, the judgment configuration in the embodiments of the present disclosure is not essential, and a specified process may or may not be executed when the judgment condition is satisfied, or a specified process may or may not be executed when the judgment condition is not satisfied.

[0056] Note that the programs of the present disclosure are implemented using, by way of example and not limitation, scripting languages ​​such as ActionScript and JavaScript (registered trademark), compiler languages ​​such as Objective-C and Java (registered trademark), and markup languages ​​such as HTML5.

[0057] <Example> By way of example and not limitation, multiple users in a group, such as colleagues belonging to the same department, may meet in the real world (offline) for a conference or meeting and may wish to share online the content discussed at the conference or meeting. In this case, by way of example and not limitation, it would be desirable to have a system that performs speech recognition based on voice data from real-world conferences and meetings, converts the data into easily viewable and searchable text, etc., and then shares the text, etc., data with other users. An embodiment for achieving this will be described below.

[0058] In this specification, by way of example and not limitation, a messaging application for implementing a messaging service (MS: Messaging Service) including an instant messaging service (IMS) is used to share data (hereinafter referred to as "minutes data") that has been converted from audio data of a conference or meeting. Hereinafter, a service for exchanging minutes data via a messaging service is referred to as a "minutes service."

[0059] In the following description, it is assumed that the server 10 is operated and managed by a messaging service provider. In the following description, the name of the messaging application is appropriately referred to as "Messaging App".

[0060] In this specification, "content" refers to messages including simple text and emoticons that are transmitted and received between multiple terminals 20 in a messaging service, as well as various types of information that can be transmitted and received between terminals 20, such as, but not limited to, image information (including still images, moving images, etc.), operation information (including buttons, icons, etc.), communication information and link information (including URIs, URLs, etc.).

[0061] A "chat room" is a virtual room in which one or more users of terminals 20 can communicate using data communication lines on a computer network. By way of example and not limitation, when content is exchanged in a chat room using a messaging service, the chat room may also be referred to as a "talk room."

[0062] Examples of talk rooms include, but are not limited to, talk rooms in which users can have one-on-one talks, as well as group talk rooms in which a group including multiple users formed in a messaging service can have talks.

[0063] Furthermore, the messaging service may or may not include an instant messaging service (IMS) that enables the sending and receiving of content such as simple messages between terminals 20.

[0064] It should be noted that the messaging service (MS) (including the IMS) can be considered as one form of the social networking service (SNS). For this reason, a distinction may or may not be made between MS and SNS.

[0065] In the following description, the user of terminal 20A is referred to as "user AA", the user of terminal 20B as "user BB", the user of terminal 20C as "user CC", . . . .

[0066] In addition, the messaging application may be provided by the server 10 as a standalone application that does not have the functionality of a so-called schedule management service, or may be provided by the server 10 as a composite application that has the functionality of a schedule management service.

[0067] In the embodiment described below, character information (character data) included in the minutes will be exemplified as character information in the present disclosure. However, the text information in this disclosure is not limited to text information, and other information that can generally be displayed in a chat room, such as, for example and without limitation, symbols, content, and image information, can also be included in the text information in this disclosure.

[0068] There are two main methods for creating and sharing meeting minutes data to realize a meeting minutes service. Each method will be explained in the following examples. (1) A method of performing speech recognition on the server 10 and creating minutes data (2) A method of performing speech recognition on the terminal 20 and creating minutes data

[0069] In addition, there are three main methods for creating and sharing meeting minutes data: (A) Create and share minutes data after a meeting or conference (B) Set up the creation and sharing of minutes data before the meeting / meeting begins, and create and share minutes data in real time during the meeting / meeting. (C) Set up the creation and sharing of minutes data before the meeting begins, and create and share minutes data after the meeting ends.

[0070] <First Example> The first embodiment is an embodiment based on the above method (2). In the first embodiment, in a minutes-of-meeting service, as an example and not a limitation, sound data (voice data, as an example and not a limitation; the same applies below) is acquired in a terminal 20 on which a messaging application runs, and minutes-of-meeting data including character data is created based on the sound data.The created minutes-of-meeting data is then displayed in a chat room (talk room, as an example and not a limitation) of the messaging application. The contents described in the first embodiment are applicable to any of the other embodiments and other modified examples.

[0071] In this embodiment, it is assumed that at least the following people participate in the chat room: user AA of terminal 20A, and a user of a terminal 20 other than terminal 20A, for example and not limitation, user BB of terminal 20B.

[0072] <Processing> FIG. 1-2 is a flowchart showing an example of the flow of processing executed by the control unit 21 of the terminal 20 in this embodiment. This flowchart is an example of a process executed by the control unit 21 of the terminal 20A.

[0073] The processes described below are merely examples of processes for implementing the method of the present disclosure, and the present disclosure is not limited to these processes. Other steps may be added to the processing described below, or some steps may be omitted (deleted). This also applies to each of the flowcharts (processes) described below.

[0074] First, the control unit 21 of the terminal 20A accepts an input to the sound input unit 25, and acquires, by way of example and not limitation, voice data including the contents of the user's dialogue (A100).

[0075] In addition to the voice of the user's dialogue, the voice data may or may not include, for example and without limitation, data on the sound of a bell ringing or the sound of a dog barking "woof woof." In other words, the voice data may or may not include data on sounds other than human voices. Furthermore, a distinction may be made between human voice data and other sound data, but it is not necessary to make such a distinction.

[0076] Next, the control unit 21 of the terminal 20A executes a voice recognition process on the acquired voice data, for example and not for limitation. Then, based on the result of the voice recognition process, the control unit 21 of the terminal 20A converts the voice data into minutes data, which is character (text) data (A120). This process can be said to be a process of generating minutes data, which is character information, from sound information.

[0077] Note that the voice recognition process is not limited to converting the spoken content into a character string. As a non-limiting example, the voice recognition process may or may not include onomatopoeia (mimetic word) conversion processing that converts the sound of a ringing bell into the character "ring ring" and the sound of a barking dog into the character "woof woof."

[0078] In addition, in the minutes data conversion process, speaker recognition processing may be performed on the voice data, and minutes data in which character data and speaker data are linked based on the speaker recognition results may or may not be generated.

[0079] Then, the control unit 21 of the terminal 20A additionally inputs the converted (generated) minutes data into a chat room (talk room) including, for example but not limited to, users AA and BB of the messaging application (A160). That is, the character data included in the minutes data is shared among multiple users through the chat room of the messaging application.

[0080] Finally, the control unit 21 of the terminal 20A causes the display unit 24 to display the minutes data inputted to the chat room (A180). Then, the control unit 21 of the terminal 20A ends the process.

[0081] <Effects of the First Embodiment> In the first embodiment, terminal 20A (not limiting, an example of a terminal) displaying a chat room including user AA (not limiting, an example of a user of a terminal) and user BB (not limiting, an example of a user of a first terminal different from the user of the terminal) acquires voice data and information related to voice (not limiting, an example of sound information) by sound input unit 25 (not limiting, an example of an acquisition unit of a terminal). Then, minutes data (not limiting, an example of text information) based on the acquired voice data is input to a chat room (not limiting, an example of a selected chat room) including selected users AA and BB, and the chat room including the minutes data is displayed on display unit 24 (not limiting, an example of a display unit of a terminal) of terminal 20A. As an example of the effect of the embodiment obtained by such a configuration, the user of the terminal and the user of the first terminal can share character information based on sound information displayed in a chat room. As a result, a plurality of users can refer to the character information based on sound information, improving user convenience. In addition, the user of the terminal and the user of the first terminal can view text information, which is easier to view than audio information. As a result, the user of the terminal can accurately obtain the information he or she needs, improving user convenience.

[0082] <First Modification Example (1)> In the first embodiment, the above method (2) is applied, and the minutes data conversion process is executed in the terminal 20 based on the voice data to create the minutes data, but the present invention is not limited to this. By way of example and not limitation, the above technique (1) may be applied.

[0083] In this case, by way of example and not limitation, when the control unit 21 of the terminal 20 executes step A100 in FIG. Then, the control unit 11 of the server 10 executes a minutes data conversion process based on the received voice data. Then, the control unit 11 of the server 10 transmits the generated minutes data to the terminal 20 via the communication I / F 14.

[0084] As an example of the effect of the modified example obtained by such a configuration, the minutes data conversion process including the speech recognition process, which requires a high calculation cost, can be executed by the control unit of the server, which has fewer constraints such as thermal design power, compared to the control unit of the terminal. This can speed up the processing. Also, it can prevent the battery of the terminal from being worn out.

[0085] <First Modification (2)> In the first embodiment, adding (inputting) the minutes data to the chat room was performed by the terminal 20, but is not limited to this. By way of example and not of limitation, the minutes data may or may not be added to the chat room by the server 10.

[0086] In this case, by way of example and not limitation, when the control unit 21 of the terminal 20 executes step A120 in FIG. 1-2, it transmits the minutes data to the server 10 via the communication I / F 22. Also, by way of example and not limitation, the control unit 21 of the terminal 20 acquires minutes posting destination selection information for specifying the chat room to which the minutes data is to be added, based on a user's input to the input / output unit 23. Then, the control unit 21 of the terminal 20 transmits the minutes posting destination selection information to the server 10 via the communication I / F 22.

[0087] Then, the control unit 11 of the server 10 adds (inputs) the minutes data to the chat room based on the received minutes data and the minutes posting destination selection information. Then, the control unit 11 of the server 10 transmits the contents of the chat room to which the minutes data has been input to the terminal 20 via the communication I / F 14. After that, the terminal 20 executes step A180 in FIG. 1-2.

[0088] In addition, when the minutes data is generated by the server 10 as described in the first modified example (1), the terminal 20 needs to transmit only the minutes posting destination selection information to the server 10.

[0089] In this modified example, a chat room is selected based on an input (minutes posting destination selection information) by a user of a terminal. As an example of the effect of this modified example, the user can specify any chat room as the input destination of character information, thereby improving user convenience.

[0090] <Second Example> The second embodiment is an embodiment using the technique (1) regarding the timing of (A) above. In the second embodiment, in the minutes-of-meeting service, the control unit 21 of the terminal 20 transmits the voice data stored in the storage unit 28 of the terminal 20 to the server 10. The control unit 11 of the server 10 creates minutes-of-meeting data based on the received voice data, and then transmits the minutes-of-meeting data to the terminal 20. Hereinafter, the method according to the second embodiment will be referred to as the "batch append mode." The contents described in the second embodiment are applicable to any of the other embodiments and other modified examples.

[0091] <Data Structure> (1) Server FIG. 2-1 is a diagram illustrating an example of information stored in the storage unit 15 of the server 10 in this embodiment. The memory unit 15 stores, by way of example and not limitation, an MS management processing program 151 which is read by the control unit 11 and executed as messaging application management processing, a voice recognition program 153 which is read by the control unit 11 and executed as voice data recognition processing, MS user registration data 155, a group management database 157, and a calendar information management database 159.

[0092] The MS user registration data 155 is registration data relating to the terminal 20 that uses the messaging application or the user of that terminal 20, and an example of the data configuration is shown in FIG. 2-2. In the MS user registration data 155, for example and not by way of limitation, a user name, an MS application ID, a terminal phone number, and other registration information are stored in association with each other.

[0093] The user name is the name of the user of the terminal 20 who uses the messaging application, and by way of example and not limitation, the name that the user of the terminal 20 registers when using the messaging application is stored.

[0094] The MS application ID is information used to identify an account of a messaging application, or the account itself. This MS application ID is preferably a unique value for each account, and as a non-limiting example, a unique value (proper value) is set and stored for each account by the server 10. The MS application ID is information associated with the terminal 20 or the user of the terminal 20, and is an example of information about the terminal or information about the user of the terminal.

[0095] The terminal phone number is the phone number of the terminal 20 of the user of this username, and by way of example and not limitation, the phone number of the terminal 20 that the user of the terminal 20 registers when using a messaging application is stored.

[0096] Other registration information includes, for example and not limited to, the ID of the terminal 20: terminal ID (for example and not limited to, IMEI ( International Mobile Equipment Identity )), the email address (terminal email address) of the terminal 20 of the user of this username, and authentication information such as passwords (login password, authentication password) used for various authentications in the messaging application.

[0097] The identification information for identifying the terminal 20 may be, for example and without limitation, a terminal ID. Furthermore, the identification information for identifying the user of the terminal 20 can be, for example and without limitation, an MS application ID. Note that this may or may not be a "user ID."

[0098] Furthermore, if the application allows only one account to be registered per terminal 20, then, as a non-limiting example, it may be possible to use "identification information for identifying terminal 20 = identification information for identifying the user of terminal 20 = MS application ID."

[0099] It should be noted that, by way of example and not limitation, it may or may not be possible to assign multiple terminal IDs to one user ID.

[0100] Group management database 157 is a database that cumulatively stores data for managing information about groups created by users in the messaging service, and an example of the data configuration is shown in FIG. 2-3.

[0101] Group management database 157 stores group management data as management data for each group. Each group management data includes, by way of example and not limitation, a group ID, a group name, and group member data.

[0102] The group ID is an ID that functions as identification information for identifying a group. As a non-limiting example, the group ID is set and stored by the server 10 as a unique ID for each group.

[0103] The group name is the name of the group identified by its group ID.

[0104] The group member data is data related to the members (users) belonging to the group, and by way of example and not limitation, the MS application ID and user name, which are information about the members belonging to the group, are stored in association with each other.

[0105] The calendar information management database 159 is a database for managing information related to a schedule management application that functions in cooperation with the messaging application. In this embodiment, the schedule management application service will be described as being provided by the same server as the server 10 of the messaging application, by way of example and not by way of limitation.

[0106] The schedule management application service may or may not be provided by a server different from the server 10. Furthermore, a messaging application may have a schedule management function as one of its functions. The calendar information management database 159 may be excluded from the configuration of this embodiment.

[0107] FIG. 2-4 shows an example of the data structure of the calendar information management database 159A, which is an example of the calendar information management database 159 in this embodiment. Calendar information management database 159A stores calendar information management data as management data for each event registered in the calendar of the schedule management application.

[0108] Each calendar information management data may store, by way of example and not limitation, an event ID, an MS application ID, an event name, an event start date and time, an event end date and time, an event location, an associated URI, and invitee data.

[0109] The event ID is an ID that functions as identification information for identifying an event. As an example, and not a limitation, the event ID is set and stored by the server 10 as a unique ID for each event.

[0110] The MS application ID is an ID that serves as an identification for identifying a registrant of the event, and is, by way of example and not limitation, an identification of a messaging application.

[0111] The event name is the name of the event identified by its event ID.

[0112] The event start date and time is the date and time when the event identified by the event ID is scheduled to start.

[0113] The event end date and time is the date and time when the event identified by the event ID is scheduled to end.

[0114] The event location is the name of the location where the event identified by the event ID is scheduled to take place. The location of the event may or may not be stored with location information identifiable by latitude and longitude, for example and not by way of limitation, together with the name of the location.

[0115] The associated Uniform Resource Identifier (URI) is information that identifies a resource (file) accessible via the Internet, for example and not by way of limitation, where information, such as materials, related to the event identified by the event ID is stored.

[0116] In the invitee data, as information of a user invited to the event identified by the event ID, for example and not limitation, an MS application ID and a user name are stored in association with each other. For example and not limitation, based on an operation input to a schedule management application of an event registrant, user information (MS application ID and user name) invited to the event is added to the invitee data and stored. Then, this calendar information management data is shared with the user registered in the invitee data.

[0117] When all members of a group are invited to the event, the invitee data may or may not store the group ID and group name in association with each other.

[0118] (2) Terminal FIG. 2-5 is a diagram showing an example of information stored in the storage unit 28 of the terminal 20 in this embodiment. In the storage unit 28, for example and not for limitation, an MS processing program 281 that is read by the control unit 21 and executed as messaging application processing, an MS application ID 283, and a voice database 285 are stored.

[0119] The MS application ID 283 is an MS application ID associated with the own terminal 20 or the user of the own terminal 20 .

[0120] The voice database 285 is, by way of example and not limitation, a database that stores voice data acquired by the sound input unit 25 of the terminal 20 .

[0121] Note that the data stored in the voice database 285 may or may not include, for example and not by way of limitation, video data captured by the imaging unit 27 of the terminal 20 .

[0122] Furthermore, the data stored in the audio database 285 is not limited to that acquired by the terminal 20. For example and without limitation, the data may be audio (video) data acquired by a terminal or electronic device (for example and without limitation, a video conferencing device) different from the terminal 20, or may not be data received by the terminal 20.

[0123] <Display screen> In the following, as an example and not a limitation, a case will be illustrated in which the terminal 20 is a smartphone equipped with a display unit 24 having a portrait display.

[0124] For example, but not by way of limitation, a smartphone has a touch panel that functions as an input unit and is disposed opposite the display, thereby forming a touch screen. When an element such as an icon, button, item, or input area is displayed on the display, and a part of the touch panel that faces the area where the element is displayed is operated by a user, a program associated with the element or a subroutine of the program is executed.

[0125] In the following, the operation by the user will be described as a tap (tap operation) as an example and not as a limitation. A tap (tap operation) is, by way of example and not limitation, an action in which a user lightly touches the display unit 24 (touch screen) that is integrally configured with a touch panel with a finger or the tip of a pen, and then releases the touch.

[0126] The transition of the display screens described below is merely an example of the transition of the display screens for implementing the method of the present disclosure. In the transition of the display screens illustrated below, some of the display screens may be omitted, or other display screens may be added.

[0127] 2-6 is a diagram showing an example of a home screen of a messaging application displayed on the display unit 24 of the terminal 20 in this embodiment. This display screen is an example of a screen displayed on the display unit 24 of the terminal 20A, which is the terminal 20 of the user AA. The name of the messaging application is displayed in the center of the top of the screen as "Messaging App." In addition, the icon image and user name (user AA in this example) of the messaging application of the user of this terminal 20 are displayed in the top right of the screen.

[0128] Also below that is a current location display area that shows the current location in the messaging application. In this example, the word “home” is displayed in the current location display area to indicate that the current location is the home of the messaging application.

[0129] Below the current location display area is a messaging application function search area, which is an input area for searching for any function within the messaging application. In this example, the word "search" is displayed in the messaging application function search area to prompt the user to enter a search keyword.

[0130] Below the messaging application function search area, an icon image and a user name in the messaging application of the user of this terminal 20 are displayed, and below that, a plurality of function icons corresponding to various functions of the messaging application are displayed. The function icons include, by way of example and not limitation, icons corresponding to each of a plurality of functions, such as "Friends," "Businesses," and "Services."

[0131] "Friends" is one of the functions of the messaging application, and a user (one account) of one terminal 20 performs a process to add a user (another account) of another terminal 20 as a friend from his / her own terminal 20, and the server 10 executes a process to add and register the friend (friend addition process, friend registration process), thereby making it possible to send and receive messages between the terminals 20 registered as friends via the server 10. An account of the messaging application owned by an individual user is called a "general account." A history of messages sent and received between the terminals 20 of users who own general accounts is displayed as a talk room or the like on the display unit 24 of the terminal 20.

[0132] By way of example and not limitation, a "business" is a type of friend function described above, and by registering an account of a messaging application owned by a user of the business (hereinafter referred to as an "official account") as a friend, rather than a general account, messages sent from the business that owns the official account can be received and displayed on terminal 20 via server 10.

[0133] Note that, like the terminal 20 of an individual user who owns a general account, a device or terminal of an operator who owns an official account can also be made capable of transmitting and receiving messages to and from other devices via the server 10. As a non-limiting example, similar to the case where a message is sent from one terminal 20 to another terminal 20, a message sent from the terminal 20 of a user who owns a general account to a device or terminal of an operator who owns an official account is also delivered to the operator's device or terminal via the server 10. Then, the history of the message can be displayed as a talk room or the like on a display unit (not shown) of the operator's device or terminal.

[0134] FIG. 2-6 shows a state where the function icon "Service" among multiple function icons is tapped. Below the display area of ​​the function icons, icons indicating recommended services are displayed, including an icon for using a minutes service. When the icon for using the minutes service is tapped, as an example and not a limitation, a voice data list information display area WR1 for displaying a list of data stored in the voice database 285 used to create minutes data shown in FIG. 2-7 is displayed on the displayed home screen. The data displayed in the voice data list information display area WR1 is data acquired (recorded) in advance by the sound input unit 25, as an example and not a limitation, in accordance with a conference or meeting before a chat room (as an example and not a limitation, a talk room) is selected.

[0135] In this example, the voice data list information display area WR1 displays, in each row, a file name for identifying the voice data (for example, not limited to, "voice20200214"), the date and time when the voice data was created (for example, not limited to, "2020.02.14 10:54"), and the time representing the length of the voice data (for example, not limited to, "1:07:35"). In addition, at the right end of this row, a voice data confirmation function icon represented by a speaker mark is displayed for outputting the recorded voice data by the sound output unit 26 when tapped by the user and confirming the recorded contents.

[0136] It should be noted that data stored in the audio database 285 is not limited to audio data. As a non-limiting example, video data including audio data may or may not be stored. In this case, by way of example and not limitation, the audio data confirmation function icon can be a video data confirmation function icon represented by a right-sided triangular mark or the like, for outputting and previewing video to the display unit 24.

[0137] When a user taps on a line indicating the voice data for which minutes data will be created using the minutes service (in this example, the second line "voice20200228"), a minutes title input field WR2 is displayed, prompting the user to enter a name to identify the minutes data to be generated from this voice data, as shown in Figure 2-8, by way of example and not limitation.

[0138] As an example and not as a limitation, in Figure 2-7, a row showing multiple audio data in the audio data list information display area WR1 may be selected by the user (tapped, as an example and not as a limitation), and then an audio data selection confirmation button (not shown) may be tapped, thereby creating one piece of minutes data from multiple audio data, or this may not be the case.

[0139] In the example of FIG. 2-8, based on a user operation, by way of example and not limitation, the characters of the minutes title "Project X February Meeting" are input in the minutes title input field WR2. The user taps the function button with the characters "Save" which is a function button for starting to create minutes data for the input contents in the minutes title input area WR2. Then, the control unit 21 of the terminal 20 transmits the voice data selected in FIG. 2-7 (in this example, data with the file name "voice20200228") to the server 10 via the communication I / F 22.

[0140] When the communication I / F 14 receives the voice data from the terminal 20, the control unit 11 of the server 10 executes a minutes data conversion process to convert the received voice data into minutes data. Then, based on the processing result of the minutes data conversion process, the control unit 11 executes a minutes posting candidate selection process, which will be described later.

[0141] Fig. 2-9 is a diagram showing an example of a screen for selecting a posting destination (a sharing target) of the minutes data displayed on the display unit 24 of the terminal 20 after the control unit 11 of the server 10 executes the minutes posting candidate selection process. In Fig. 2-9, as an example and not a limitation, a posting destination chat room selection area WR3 is displayed on the displayed home screen.

[0142] In the destination chat room selection area WR3, the text "Please select the chat room to post to" is displayed as an example, not a limitation, to prompt the user to select the destination to post the minutes data.

[0143] Below that, a destination candidate display area is displayed, which displays destination candidates for the minutes data selected by the minutes posting candidate selection process from among groups to which the user of terminal 20A (user AA in this case) belongs as a member or among friends of the user of terminal 20A. In this example, the destination candidate display area displays the number of destination candidates as "2", and below that, it is displayed that a group with the group name "Second Development Room" and a group with the group name "Project X" have been selected as destination candidates. In addition, the number of members belonging to each group is displayed in parentheses to the right of each group name. In this example, by way of example and not limitation, the number of members constituting the group "Second Development Room" is "8 people", and the number of members constituting the group "Project X" is "5 people".

[0144] In the destination talk room selection area WR3, a destination group display area that displays groups to which the user of terminal 20A belongs as a member is displayed below the destination candidate display area, and a destination friend display area that displays friends of the user of terminal 20A is displayed below the destination group display area. The display contents of the destination group display area and the destination friend display area are not limited to those of the destination candidate display area, but are, for example, similar to those of the destination candidate display area, and therefore will not be described.

[0145] Note that the destination chat room selection area WR3 may or may not display only the destination candidate display area. In addition, the destination chat room selection area WR3 may display the destination group display area and the destination friend display area, but may not display the destination candidate display area. In this case, the minutes posting candidate selection process may not be executed.

[0146] When the user taps a group or friend in the destination chat room selection area WR3, that group or friend is selected as the destination (share target) for the minutes data. In this example, the group "Project X" is selected by the user.

[0147] Note that the most likely posting candidate (first minutes posting candidate) obtained by the minutes posting candidate selection process may or may not be automatically selected as the posting destination of the minutes data, rather than a user operation in the posting destination chat room selection area WR3. In this case, the posting destination chat room selection area WR3 does not need to be displayed.

[0148] Thereafter, a minutes content confirmation display area WR4 as shown in FIG. 2-10 is displayed on the display unit 24 for confirming the contents of the minutes data converted (generated) by the minutes data conversion process.

[0149] In the minutes content confirmation display area WR4, as a non-limiting example, contents (minutes data) in which the conversation content in the voice data is converted into text based on the voice data are displayed in chronological order.

[0150] In the minutes content confirmation display area WR4, by way of example and not limitation, following the content display format in a chat room of a messaging application, comments from user AA of the own terminal, terminal 20A, are displayed in speech bubbles on the right side, and comments from other users are displayed on the left side, each in a speech bubble as content. It should be noted that the identification of the user who has spoken in the voice data may be performed in the minutes data conversion process, for example and not by way of limitation.

[0151] Specifically, in Fig. 2-10, the minutes title "Project X February Meeting" entered in the minutes title input area WR2 is displayed in the minutes title display content TTL1 at the top of the minutes content confirmation display area WR4. The minutes title display content TTL1 is not the spoken content in the audio data, so it is displayed center-justified in the minutes content confirmation display area WR4 as an example and not a limitation.

[0152] Below the minutes title display content TTL1, text content CT1 is displayed based on the speech by user AA at the beginning of the audio data, "Let's begin the February 2020 meeting." To the left of the text content CT1, the characters "00:00:00" are displayed, indicating the time from the beginning of the audio data when user AA spoke this content.

[0153] Although this specification refers to this as "text content," it may also be referred to as a "message (message information)" in a messaging application.

[0154] Below the text content CT2, text content CT2 based on the speech of user AA "It's noisy outside, can you hear me?" uttered at "00:00:06" from the beginning of the audio data is displayed. Note that, as an example and not a limitation, a symbol "?" is added to the end of the sentence in the text content CT2, which indicates that the spoken sentence has been identified as a question.

[0155] Below the text content CT3, text content CT3 is displayed based on the speech of user BB "When was the product release?" at "00:00:16" from the beginning of the audio data. The symbol "?" at the end of the text content CT3 indicates that this sentence is a question, just like the text content CT2. Similarly, text content is displayed based on the speech content in the voice data.

[0156] The text content CT4 is text content based on the user BB's utterance, "I'll check with the Content Planning Department just to be sure." In the text content CT4, as a non-limiting example, the group "Content Planning Department" to which the user of the terminal 20A belongs as a member is highlighted with an underline, "@Content Planning Department," indicating that a mention function in a messaging application will automatically be activated based on the utterance of "Content Planning Department."

[0157] Here, the mention function is a function of a messaging application for sending a message (content) to a specified user. In addition, on the terminal of a user who receives a message via the mention function, a notification emphasizing that the message has been received may or may not be displayed.

[0158] As a non-limiting example, when the mention function in the text content CT4 is activated, a message with the content "I will check with the content planning department just to be sure" is sent to the members belonging to the group "Content Planning Department" from user BB. In other words, the members of the content planning department can easily determine that they have received an important message from user BB.

[0159] The text content CT5 is text content based on the user EE's utterance of "What do you think, DD?" In the text content CT5, as an example and not a limitation, the notation "@DD" is highlighted with an underline, indicating that the mention function will automatically work based on the utterance of "DD" for the user DD, who is a member of the group that posts the minutes data.

[0160] In this way, when a user name or group name is spoken in the voice data, the mention function can be automatically activated in accordance with the spoken name in the minutes data conversion process.

[0161] However, the mention function is not essential, and the mention function may be disabled. Also, as a non-limiting example, the mention function may or may not be applied to only users or groups who did not attend the conference or meeting where the voice data was obtained, based on a user operation on the terminal 20. Alternatively, the mention function may or may not be applied to only users or groups who attended the conference or meeting where the voice data was obtained. Alternatively, the mention function may or may not be applied to only specific users or groups set by the user.

[0162] In addition, the targets for which the mention function operates are not limited to user names or group names. As a non-limiting example, when a predefined keyword is spoken, the mention function may or may not operate on the keyword.

[0163] At the bottom of the minutes content confirmation display area WR4, a minutes posting button SBT1 containing the word “Post” is displayed, which is used to confirm the minutes data with the content displayed in the minutes content confirmation display area WR4 and post it to the chat room selected as the posting destination.

[0164] FIG. 2-11 is an example of a screen for correcting the minutes data displayed in the minutes content confirmation display area WR4 shown in FIG. 2-10. In FIG. 2-11, by way of example and not limitation, when the text content CT2 is tapped by the user, the text content CT2 is displayed inverted, and a content edit selection menu MBT1 for modifying the content related to the text content CT2 is displayed above the text content CT2.

[0165] In this example, the content edit selection menu MBT1 is configured to allow three types of edit operations, namely, "delete," "add between lines," and "comment," by way of example and not of limitation.

[0166] "Delete" is, by way of example and not limitation, an editing operation that deletes the selected text content CT2. "Add between lines" is, for example and not by way of limitation, an editing operation for inserting and displaying text content etc. below the selected text content CT2. "Comment" is, by way of example and not by way of limitation, an editing operation that adds a comment to the selected text content CT2.

[0167] The editing operations selectable in the content editing selection menu MBT1 are not limited to the above three types. For example, but not limited to, an editing operation for deleting a comment or an editing operation for adding a URI (Uniform Resource Identifier) ​​that can refer to the content according to the content of the text content may or may not be selectable.

[0168] FIG. 2-11 illustrates, by way of example and not by way of limitation, a state in which a "delete" button, which is an editing operation for deleting the text content CT2, has been tapped based on a user operation.

[0169] FIG. 2-12 is an example of a screen showing the results of editing operations "delete" and "add space between lines" performed on the content edit selection menu MBT1 in FIG. 2-11. In Figure 2-12, instead of the text content CT2, the edited text content MCT1 is displayed with a strikethrough indicating that the contents of the text content CT2 have been deleted. In addition, below the edited text content MCT1, the characters "We are talking about the planned release date", which are the text inserted by the "add interline" operation, are displayed in the edited text content MCT2. The edited text content MCT2 is not the spoken content in the audio data, and is therefore displayed center-justified in the minutes content confirmation display area WR4, by way of example and not by way of limitation.

[0170] In the "add between lines" operation, the edited text content may be displayed in the same manner as the user's spoken content by adding user information and inputting the text, or it may not be displayed in this manner.

[0171] In Figure 2-12, as an example and not a limitation, when a user taps the minutes posting button SBT1, the minutes data displayed in the minutes content confirmation display area WR4 at the time of tapping is posted to the chat room selected as the posting destination (in this example, the chat room for the group "Project X").

[0172] Figure 2-13 is a diagram showing an example of a chat room screen of a messaging application displayed on display 24 of terminal 20B, which is terminal 20 of user BB, after user AA taps the minutes posting button SBT1 and posts minutes data to the group "Project X."

[0173] The current location display area, which shows the current location in the messaging application, displays text indicating that the current location is in the chat room of the group "Project X." Below the current location display area, a content display area TR1 for displaying content (messages) to be sent (posted) to this talk room is displayed. Also, below the content display area TR1, a content input area for inputting and selecting content to be sent to this chat room is displayed.

[0174] In Fig. 2-13, in the content display area TR1, as an example and not a limitation, minutes data summary content PRC1 is displayed based on the minutes data posted by the operation in Fig. 2-12 below the text content "These are the minutes of the meeting that just finished" sent by user AA. That is, each operation shown in Fig. 2-6 to Fig. 2-12 represents an operation that was performed after user AA sent the text content "These are the minutes of the meeting that just finished" to the chat room of group "Project X".

[0175] In the minutes data summary content PRC1, by way of example and not limitation, the minutes title "Project X February Meeting" entered in the minutes title input field WR2 and the characters "Last revised 2020 / 02 / 28 / 16:30" indicating the date and time when the minutes data was last revised are displayed. In addition, in the lower right corner of the minutes data summary content PRC1, the time when the minutes data was sent (posted) to this chat room is displayed.

[0176] Below the minutes data summary content PRC1, for example and not limitation, is displayed text content sent by user EE saying, "There were some corrections, so I will fix them." Also, below that is displayed text content sent by user BB saying, "Thank you for the minutes. I will check them." In the content display area TR1, the time when the message was sent to the talk room is displayed to the side of each piece of text content.

[0177] In the content display area TR1, for example and not for limitation, the minutes data summary content PRC1 is tapped by the user. Then, for example and not for limitation, the minutes data summary content PRC1 is extended downward, and the contents of the minutes data are displayed according to the display mode of the minutes content confirmation display area WR4. This configuration is used to prevent the chat room display from being filled with the minutes data, which reduces the visibility of the chat room.

[0178] FIG. 2-14 is an example of a display screen after the minutes data summary content PRC1 is tapped by the user BB. In FIG. 2-14, in the content display area TR1, for example and not for limitation, a minutes data display area PRW1 extending downward from the position of the minutes data summary content PRC1 is displayed between the content input area.

[0179] Alternatively, when minutes data is sent to a talk room, the minutes data display area PRW1 may be displayed from the beginning instead of the minutes data summary content PRC1, or it may not be displayed.

[0180] At the top of the minutes data display area PRW1, the minutes title, which is the same as that of the minutes data summary content PRC1, and the date and time when the minutes data was last modified are displayed. At the top right of the minutes data display area PRW1, an edit function button BT1 for editing the minutes data is displayed.

[0181] By way of example and not limitation, when the edit function button BT1 is tapped by the user, the editing operation shown in Figure 2-11 is possible. If the edit function button BT1 is not tapped, by way of example and not limitation, tapping on text content will not display the content edit selection menu. This prevents the user from accidentally editing the minutes data while viewing it.

[0182] Displayed in the lower right corner of the minutes data display area PRW1 are a fold function button BT2 for shrinking the minutes data display area PRW1 (folding the display) and displaying the minutes data summary content PRC1, and a filter function button BT3 for narrowing down the content displayed in the minutes data display area PRW1, by way of example and not limitation, speaker or keyword. The folding function button BT2 and the filter function button BT3 are displayed in a floating manner, for example and not for limitation, and their display positions are constant regardless of the scroll operation on the minutes data display area PRW1.

[0183] In the minutes data display area PRW1 of FIG. 2-14, the content confirmed / modified in the minutes content confirmation display area WR4 is displayed according to the terminal 20 displaying it or the user of the terminal 20 displaying it. As an example and not by way of limitation, text content CT1 is displayed in a balloon on the left side of user BB's terminal 20B because it is based on the speech of user AA. Text content CT3 is displayed in a balloon on the right side of user BB's terminal 20B because it is based on the speech of user BB.

[0184] In the minutes data display area PRW1 in Figure 2-14, as an example and not a limitation, the edited text content MCT2 that was deleted or edited before posting displays the words "Comment deleted" indicating that a deletion or editing operation has been performed, although the original content cannot be confirmed.

[0185] Furthermore, in the minutes data display area PRW1 in Figure 2-14, edited text content MCT3 is displayed, which indicates that an editing operation was performed on the text content CT3 after it was posted to the chat room. In the edited text content MCT3, as a non-limiting example, the correction (edit) of the characters "Kyoto" to "Kobe" is displayed as a revision history. In addition, edit history content MRF1, which is the editor's comments on the edited text content MCT3, is inserted and displayed below the edited text content MCT3.

[0186] As a non-limiting example, the edit history content MRF1 is associated with and displayed with an icon of the user who performed the edit (user EE in this case). Also, as a non-limiting example, the edit history content MRF1 displays the text "Wrong location" entered by user EE and a URI related to a workshop in Kobe entered by user EE. Below the URI, as a non-limiting example, a thumbnail related to the link destination of the URI is automatically generated and displayed. As a non-limiting example, when this thumbnail is tapped by the user, the terminal 20 launches a web browser and displays information about the URI.

[0187] For editing operations performed before posting, for example and without limitation, a history may be left and displayed so that the editor and the editing contents can be identified, or it may not be. Also, for editing operations performed after posting, the editor and their comments may not be displayed. Editing operations may be displayed in the form of additions or annotations, or it may not be displayed.

[0188] For example, but not by way of limitation, when the user performs a drag operation up or down on the minutes data display area PRW1, the content in the minutes data display area PRW1 is displayed by scrolling up or down. Here, a drag operation (drag) is, by way of example and not limitation, an action in which a user touches the display unit 24 (touch screen) that is integrally configured with a touch panel with a finger or the tip of a pen, and then moves the finger or the tip of the pen.

[0189] A navigation bar is displayed on the right side of the minutes data display area PRW1, indicating the location area of ​​the content of the minutes data that is displayed in the minutes data display area PRW1. By way of example and not of limitation, when a downward scroll operation is performed on the minutes data display area PRW1, the navigation bar also moves downward.

[0190] FIG. 2-15 is an example of a display screen when the filter function button BT3 in FIG. 2-14 is tapped by the user. In Figure 2-15, at the bottom of the minutes data display area PRW1, by way of example and not limitation, when the user taps the filter function button BT3, the filter function button BT3 is extended to the left and a filter selection area FLW1 is displayed.

[0191] In the filter selection area FLW1, for example and not for limitation, user icons are displayed in a row to narrow down the speakers (users) to be displayed in the minutes data display area PRW1. When these user icons are tapped, the check mark in the upper right corner of the icon is inverted, and the user can select whether or not to display the comments (posted content) of each user in the minutes data display area PRW1.

[0192] By way of example and not limitation, in FIG. 2-15, the check marks of the icons for user CC and user EE are highlighted to indicate that they have been narrowed down to display. At this time, by way of example and not limitation, content other than the comments between user CC and user EE is not displayed in the minutes data display area PRW1. As an example and not a limitation, the text content "The workshop is in three months, so it's difficult" representing the comment by user BB that was displayed under the edit history content MRF1 in Figure 2-14, and the text content CT4 are not displayed in the minutes data display area PRW1 in Figure 2-15, and the text content CT5 is displayed filling in the gaps.

[0193] Note that titles and comments that have no speaker may or may not be displayed regardless of the filter contents. In addition, the filter target in the filter selection area is not limited to the speaker. For example and without limitation, words that frequently appear in the minutes data may be automatically extracted, or text content containing keywords designated by the user may or may not be extracted as the filter target. Furthermore, as a filter target, by way of example and not limitation, a specific section (paragraph) in the minutes data linked by a tag may or may not be extracted.

[0194] In FIG. 2-15, a link destination confirmation area LK1 is displayed below the text content CT6 based on the user CC's utterance "Please look at page 5 of the agenda" by way of example and not limitation. The link destination confirmation area LK1 reads out agenda data stored in the associated URI of the calendar information management data linked to the acquisition date and time of this voice data based on the spoken content of "agenda" in the text content CT6 by way of example and not limitation, and based on the calendar information management database 159A by way of example and not limitation. Then, based on the spoken content of "page 5", a link destination (URI) corresponding to page 5 of the agenda data is automatically generated and displayed. By way of example and not limitation, when this link is tapped by the user, the terminal 20 launches a web browser and displays the information of this URI.

[0195] The agenda data may or may not be data that has been manually or voice-entered in advance by the user.

[0196] In the edit history content MRF1, a URI is input by the user and a thumbnail of the URI is displayed. However, in the link destination confirmation area LK1, a URI is automatically generated based on the spoken content, which is different.

[0197] <Processing> FIG. 2-16 is a flowchart showing an example of the flow of processing executed by each device in this embodiment. From the left, an example of a process executed by the control unit 21 of the terminal 20A and an example of a process executed by the control unit 11 of the server 10 are shown.

[0198] First, the control unit 21 of the terminal 20A selects, by way of example and not limitation, voice data to be used for creating minutes data from the voice data stored in the voice database 285 based on user operation (A110).

[0199] Then, the control unit 21 of the terminal 20A transmits the selected voice data to the server 10 via the communication I / F 22 (A130).

[0200] When the control unit 11 of the server 10 receives the voice data via the communication I / F 14, it executes a minutes data conversion process to convert the voice data into minutes data, which is character (text) data (S120). The minutes data conversion process can be realized by a process similar to step A120 in FIG. 1-2.

[0201] In addition, in the minutes data conversion process, in addition to the voice recognition process and the speaker recognition process, a content addition process may be performed to add content corresponding to the speech content converted into text, or it may not be necessary to do so.

[0202] The content addition process may include, by way of example and not limitation, a mention function addition process in response to a name or keyword being spoken, an image addition process that adds an image related to the spoken content, and a URI addition process that adds a URI related to a specific word (as an example and not limitation, "agenda") when the specific word is spoken (as an example and not limitation, a related URI in the calendar information management database 159A).

[0203] Then, the control unit 11 of the server 10 executes a minutes posting candidate selection process for selecting candidates for posting destinations of the minutes based on the generated minutes data (S130).

[0204] In the process of selecting candidates for posting minutes, the control unit 11 of the server 10 identifies a group of speaking users who are speaking in the minutes data based on the speaker recognition process result, for example and not for limitation. Then, from among the group management data stored in the group management database 157, for example and not for limitation, selects a group having a high correlation between the speaking user group and members registered in the group member data as a candidate for posting minutes. Alternatively, when the speaking user group is two people, selects a talk room including that user as a candidate for posting minutes. In addition, the candidate for posting minutes may be the top one, or multiple candidates may be selected from the top.

[0205] In addition, the method of selecting candidates for posting minutes is not limited to using a group of speaking users obtained from voice data. As a non-limiting example, the group of speaking users may or may not be users of terminals 20 that are within the reception range of a beacon signal transmitted from an electronic device such as a beacon device (proximity beacon device) installed at the venue of a conference (meeting).

[0206] Also, in this case, as a non-limiting example, when a beacon signal is transmitted from terminal 20B and terminal 20A is present within the reception range of the beacon signal, user BB may or may not be one of the users of the speaking user group. That is, as a non-limiting example, beacon signals may be transmitted from terminals 20 to each other, and users of terminals 20 present within the reception range of the beacon signal may or may not be the speaking user group. More specifically, the position calculation information detection unit 29B of terminal 20A may grasp nearby terminal groups using a UWB positioning unit, and the users of the terminal groups may be the speaking user group.

[0207] As another method for identifying the group of speaking users, for example and without limitation, a facial image including each user may be acquired by the imaging unit 27 of the terminal 20A, and the group of users shown in the facial image may be identified by person recognition, or it may not be possible to do so.

[0208] Also, as an example and not as a limitation, the date and time of the conference (meeting) may be obtained from the timestamp of the voice data, and based on the calendar information management database 159A, the group of speaking users may be identified from the invitee data of the calendar information management data that matches the date and time of the conference from among the calendar information of user AA, or this may not be done.

[0209] The speech user group may be estimated by combining the above-mentioned methods for identifying the speech user group, but this is not essential.

[0210] Thereafter, the control unit 11 of the server 10 transmits the obtained minutes posting candidates to the terminal 20A via the communication I / F 14.

[0211] The minutes posting candidate selection process may be executed by the terminal 20A instead of the server 10.

[0212] When the control unit 21 of the terminal 20A receives the minutes posting candidates from the server 10 via the communication I / F 22, the control unit 21 selects a minutes posting destination, which is a chat room for adding (inputting) minutes data, from the minutes posting candidates based on a selection made by a user operation, as a non-limiting example. Then, the control unit 21 of the terminal 20A transmits the minutes posting destination to the server 10 via the communication I / F 22 (A150).

[0213] Note that a chat room other than the candidate for posting the minutes may or may not be selected as the destination for posting the minutes. Also, when there is only one minutes posting candidate, that minutes posting candidate may be automatically set as the minutes posting destination, and this step may or may not be omitted.

[0214] The control unit 11 of the server 10 receives the minutes posting destination via the communication I / F 14. Then, the control unit 11 of the server 10 transmits the minutes data generated in step S120 as minutes draft data indicating a draft of the minutes to the terminal 20A via the communication I / F 14 (S140).

[0215] When the control unit 21 of the terminal 20A receives the minutes draft data from the server 10 via the communication I / F 22, it displays it on the display unit 24. Then, the control unit 21 of the terminal 20A accepts an operation to modify the minutes draft data based on an operation input by the user to the input / output unit 23 (A170).

[0216] The process of correcting the draft minutes data may include, by way of example and not limitation, editing operations such as "delete," "add interline space," and "comment" in the content editing selection menu described above.

[0217] Then, the control unit 21 of the terminal 20A transmits the minutes draft data amendment information, which is the content of the editing operation, to the server 10 via the communication I / F 22.

[0218] When the control unit 11 of the server 10 receives the minutes draft data amendment information via the communication I / F 14, the control unit 11 executes minutes draft data amendment processing for updating the minutes data based on the minutes draft data amendment information (S150).

[0219] Then, the control unit 11 of the server 10 adds (inputs) the corrected minutes data to the chat room specified as the destination for posting the minutes (S160).

[0220] Thereafter, the control unit 11 of the server 10 transmits the minutes data added (input) to the chat room to the terminals 20 of the users who belong to the chat room including the terminal 20A via the communication I / F 14 (S170). Then, the control unit 11 of the server 10 ends the process.

[0221] When the minutes data is received by the communication I / F 22, the control unit 21 of the terminal 20A causes the display unit 24 to display the minutes data (A180). Then, the control unit 21 of the terminal 20A ends the process.

[0222] When the minutes data is to be revised after it has been added to the chat room, the terminal 20 to which the minutes data has been transmitted may regard the minutes data as draft minutes data and execute steps A170 to A180 and S150 to S170.

[0223] Also, the step of S160 may not be executed by the server 10, and the control unit 21 of the terminal 20 may add (input) the minutes data to the chat room after receiving the minutes data, or it may not be necessary to do so.

[0224] <Effects of the second embodiment> The second embodiment illustrates a configuration in which a talk room (not limited to this, but an example of a chat room) is selected from candidates for posting minutes based on an input by a user AA (not limited to this, but an example of a user of a terminal) of the terminal 20A. As an example of an effect of the embodiment obtained by such a configuration, since chat rooms can be selected from candidates for posting minutes, the user can select a chat room more appropriately and easily.

[0225] Furthermore, the second embodiment illustrates a configuration in which a talk room is selected based on information (not limited to this, but an example of the first information) acquired by the terminal 20A. As an example of an effect of the embodiment obtained by such a configuration, candidate chat rooms can be selected based on the first information, which makes it possible to more appropriately select chat rooms.

[0226] In this case, the first information may include voice data (not limiting, an example of sound information) acquired by the sound input unit 25 of the terminal 20 (not limiting, an example of a first acquisition unit). As an example of the effect of the embodiment obtained by such a configuration, the sound information on which the character information is based can be used as a criterion for selecting a chat room, making it possible to select a more appropriate chat room.

[0227] In this case, the first information may include information transmitted from the electronic device. As an example of the effect of the embodiment obtained by such a configuration, even when it is difficult to identify the speaking user from the sound information, the chat room to which the input is to be made can be appropriately selected.

[0228] In this case, the electronic device may be a beacon device that transmits a beacon signal. As an example of the effect of the embodiment obtained by such a configuration, a chat room can be easily and appropriately selected based on information transmitted from an electronic device that has been installed in advance.

[0229] Also in this case, the electronic device may be a terminal 20 other than the terminal 20A, for example and without limitation, the terminal 20B (an example of a first terminal, and without limitation). As an example of an effect of the embodiment obtained by such a configuration, it is possible to obtain the first information without installing the electronic device in advance, which results in improved convenience for the user.

[0230] In this case, the first information may include image information of an image captured by the imaging unit 27 of the terminal 20 (for example, but not limited to, a face image). As an example of the effect of the embodiment obtained by such a configuration, even when it is difficult to identify the speaking user from the sound information, the chat room to which the input is to be made can be selected by using the image information.

[0231] The first information may also include calendar information of user AA (not limited to this, an example of schedule information of the user of the terminal). As an example of an effect of an embodiment obtained by such a configuration, it is possible to use schedule information to select an appropriate chat room in which to post, and even when it is difficult to identify the speaking user from the sound information, it is possible to more accurately select the chat room to which to post.

[0232] In addition, the second embodiment illustrates a configuration in which first character information (as an example, not limiting, text content CT1) based on first sound information acquired by the sound input unit 25 (as an example, not limiting, an utterance, "Let's start the meeting for February 2020") is input into a selected chat room (as an example, not limiting, a talk room for group "Project X") in association with user AA, and second character information (as an example, not limiting, text content CT3) based on second sound information acquired by the sound input unit 25 (as an example, not limiting, an utterance, "When was the product release?") is input into the chat room in association with user BB, and the chat room including the first character information and the second character information is displayed on the display unit (as an example, not limiting, display unit 24). As an example of the effect of the embodiment obtained by such a configuration, the user can easily distinguish that the first character information based on the first phonetic information and the second character information based on the second phonetic information are associated with different users, thereby improving the convenience for the user.

[0233] Furthermore, the second embodiment illustrates a configuration in which voice data (sound information) is acquired by sound input unit 25 before a chat room is selected, and minutes data (text information) is input to the chat room based on the selection of the chat room. As an example of the effect of the embodiment obtained by such a configuration, character information based on sound information before selecting a chat room can be input to the chat room later. In other words, since there is no need to select a chat room when acquiring sound information, sound information can be acquired quickly without complicated operations. Therefore, the user can easily decide to acquire sound information without being aware of the existence of the chat room, and the user's convenience can be improved.

[0234] In addition, the second embodiment shows a configuration in which minutes data (text information) is entered into a first chat room (as an example, not limiting, a group "Project X Talk Room") including user AA (not limiting, an example of a user of the terminal), user BB (not limiting, an example of a user of the first terminal), and user CC (not limiting, an example of a user of the second terminal), and is transmitted to at least terminal 20C (not limiting, an example of the second terminal). As an example of the effect that can be obtained by such a configuration, text information can be entered into a chat room including a user of the terminal, a user of the first terminal, and a user of the second terminal, and then transmitted to at least the second terminal, making it viewable by at least the user of the second terminal.

[0235] <Second Modification Example (1)> In the second embodiment, the server 10 executes the minutes data conversion process based on the voice data to create the minutes data, but the present invention is not limited to this. As a non-limiting example, the process may or may not be realized by the above method (2).

[0236] In this case, as an example and not a limitation, the control unit 21 of the terminal 20 executes the minutes data conversion process when it executes step A110 in FIG. 2-16. Each process can be executed similarly to step S120 in FIG. 2-16, so detailed explanations will be omitted. Then, the control unit 21 of the terminal 20 transmits the minutes data generated by the minutes data conversion process to the server 10 via the communication I / F 22. When the control unit 11 of the server 10 receives the minutes data via the communication I / F 14, it executes steps S130 and thereafter in FIG. 2-16.

[0237] As an example of the effect of this modified example obtained by such a configuration, when the data volume of the sound information is large, it becomes possible for the terminal to convert the sound information into text information, which is easier to compress the amount of information, before transmitting it to the server, thereby reducing the amount of communication.

[0238] 2-16 may or may not be performed by the terminal 20. In this case, it can be said that the first information is generated by the terminal.

[0239] Also, the minutes data conversion process may be executed by the terminal 20, and the minutes data may be added to the chat room by the server 10, but this is not essential.

[0240] <Third Example> The third embodiment is an embodiment using the technique (1) regarding the timing of (B) above. In the third embodiment, in the minutes-of-meeting service, when the control unit 21 of the terminal 20 acquires a portion of the audio data of a conference or meeting by the audio input unit 25, it transmits the acquired data to the server 10. When the control unit 11 of the server 10 creates partial minutes-of-meeting data based on the received audio data, it transmits the partial minutes-of-meeting data to the terminal 20. The above process is repeated until the end of the conference (meeting). In the following, the method according to the third embodiment is referred to as the "real-time mode." The contents described in the third embodiment are applicable to any of the other embodiments and other modified examples.

[0241] <Display screen> The following shows an example of a display screen when the chat room of the group "Project X" is selected based on user input as the chat room to which the minutes data is added (input), as an example and not a limitation.

[0242] Figure 3-1 is an example of a display screen of the chat room for the group “Project X” when a user taps (selects) “Friends” in Figure 2-6 and then taps (selects) the group “Project X” from the user accounts or groups registered as “Friends.”

[0243] In FIG. 3-1, the current location display area displays text indicating that the current location is the chat room of group "Project X." Additionally, the content display area TR1 displays the text content "It's time to start the meeting," input (sent) by user AA, and the text content "I will be a little late due to a train delay," input (sent) by user DD.

[0244] In the content input area, by way of example and not limitation, when a user taps a talk room function call icon represented by the character "+", a talk room function selection area WR5 is displayed which calls up various functions of the messaging application associated with this talk room. In the chat room function selection area WR5, a plurality of function icons corresponding to various functions of the messaging application related to the chat room are displayed, including, by way of example and not limitation, icons corresponding to a plurality of functions such as "Files," "Contacts," and "Location Information."

[0245] In this example, the function icon for "minutes" among multiple function icons is shown to be in a tapped state. Then, minutes data is added to this talk room in real time based on the voice data acquired by the sound input unit 25. That is, the timing when the function icon for "minutes" is tapped is before the start of the conference or meeting.

[0246] In addition, when the "Minutes" function icon is tapped, based on user operation on a user selection screen not shown, the comments of only specific users among the members belonging to this chat room may or may not be recorded as minutes of meeting data thereafter.

[0247] Figure 3-2 is an example of the chat room screen when the “Minutes” function icon is tapped in Figure 3-1. In Figure 3-2, at the top of the content display area TR1, an audio data acquisition status confirmation area STW1 is displayed, which indicates that audio data is being acquired by the audio input unit 25, by way of example and not by way of limitation, based on the minutes function of the messaging application.

[0248] In the voice data acquisition status confirmation area STW1, for example and not for limitation, the words "Recording minutes" are displayed, which indicates that the terminal 20A is acquiring voice data for generating the minutes data. To the right of the words, the elapsed time since the voice data acquisition started (in this example, "00:03") is displayed.

[0249] To the right of the elapsed time are displayed side by side a recording end button BT4 for stopping the acquisition of audio data and ending the addition of the minutes data to this talk room, a capture button BT5 for capturing an image using the imaging unit 27 and adding it to this talk room, and the edit function button BT1 mentioned above.

[0250] Also, in the lower right corner of the content display area TR1, a filter function button BT3, a content highlight function button BT6 indicated by a speech bubble icon, and a tag function button BT7 indicated by a pound sign icon are displayed vertically aligned. The filter function button BT3, the content highlight function button BT6, and the tag function button BT7 are displayed in a floating manner, for example and not by way of limitation, and their display positions remain constant regardless of the scroll operation on the content display area TR1.

[0251] By way of example and not limitation, when the tag function button BT7 is tapped by the user, the tag function button BT7 extends to the left, and a tag input area and a tag reflection function button indicated with the word "Settings" for reflecting the input tag are displayed, as shown in FIG. 3-2. By way of example and not limitation, in the example of FIG. 3-2, the word "Agenda 1" is input in the tag input area.

[0252] When the user taps the tag reflection function button, the tag "Agenda 1" is added to the minutes data converted from the voice data thereafter, by way of example and not limitation.

[0253] Based on this tag, by way of example and not limitation, it becomes possible to quickly extract and display a portion of the minutes data to which any tag has been added using the filter function button BT3.

[0254] In this example, by way of example and not limitation, tags may or may not be added automatically in response to the user's utterance, such as "Now, let's talk about agenda 1."

[0255] In addition, in the content display area TR1, by way of example and not limitation, text content CT1 is generated from partial voice data separated by speech segments, and is input into the talk room in real time and displayed. In this embodiment, the time when the content was generated and added to the talk room is displayed beside the content (message), not as a relative time from when the voice data began to be acquired, but as a way of example and not limitation. By way of example and not limitation, text content CT1 represents being added at the time "13:42".

[0256] It should be noted that the relative time from when the voice data started to be acquired may be displayed instead of the time, but this is not essential.

[0257] FIG. 3-3 is an example of a display screen when the acquisition of voice data continues after the tag reflection function button in FIG. 3-2 is tapped.

[0258] In FIG. 3-3, as a result of the tag being input, tag content TG1 is displayed below text content CT1, with the characters "Regarding Agenda 1" indicating the content of the input tag. The contents of the tag content TG1 are not the contents of speech in the voice data, and therefore, by way of example and not of limitation, it is displayed center-justified in the content display area TR1.

[0259] The color of the bands painted on both ends of the tag content TG1 corresponds, by way of example and not limitation, to the color of a band that is associated with and displayed on the left edge of the content display area TR1 for subsequently added content.

[0260] By way of example and not limitation, the text content CT3 displays a strip to the left of it that is associated with the tag "Agenda 1."

[0261] Also, below the text content based on the user EE's utterance "Will you make it in time for the workshop in Kyoto?", image content PC1 for displaying an image obtained as a result of an image search based on the utterance content, as an example and not a limitation, is automatically added / attached (input). In FIG. 3-3, an image of a temple is added to the image content PC1 and displayed as an example of the result of an image search based on the place name "Kyoto" in the utterance content, as an example and not a limitation.

[0262] It should be noted that the images to be added or attached are not limited to images based on place names. In addition, as a non-limiting example, an image such as an employee ID card image of user AA may or may not be added based on the content of the utterance "Mr. AA." Any other image may or may not be added as long as it is related to the content of the conversation.

[0263] FIG. 3-4 is an example of a display screen when the acquisition of voice data continues following FIG. 3-3.

[0264] In Fig. 3-4, as an example and not a limitation, when text content CT5 is added to a talk room, in response to the mention function of text content CT5, user DD sends a message MSG1 to the talk room with the text "I'm on the move, I'll check the contents later." Then, as an example and not a limitation, a message arrival notification area NTW1 indicating that a message has been received from user DD is displayed on the display unit 24 of terminal 20A.

[0265] Since the message MSG1 is not minutes data based on audio data, by way of example and not limitation, no band associated with a tag is displayed to the left of the message MSG1 in the content display area TR1.

[0266] It should be noted that a tag may also be associated with a message sent (input) by a user, and a band may be displayed to the left of the tag, but this is not essential.

[0267] In addition, in the content display area TR1, a link destination confirmation area LK2 is displayed below the text content CT6, by way of example and not by way of limitation. The content displayed in the lower part of the link destination confirmation area LK2 is the same as that in the link destination confirmation area LK1, so a description thereof will be omitted. In the upper part of the link destination confirmation area LK2, by way of example and not by way of limitation, a thumbnail relating to the link destination of the URI is automatically generated and displayed. This makes it easy to refer to content that may be necessary during a meeting without having to manually follow links.

[0268] FIG. 3-5 is an example of a display screen when the acquisition of voice data continues after a period of time has elapsed since FIG. 3-4.

[0269] As a non-limiting example, when the user taps the image capture button BT5, the control unit 21 of the terminal 20A executes the image capture function of the messaging application and captures an image using the image capture unit 27. The captured image is embedded in the minutes data at the time of capture and displayed in the content display area TR1.

[0270] In FIG. 3-5, as an example and not a limitation, an image of a whiteboard captured by tapping the image capture button BT5 is embedded in the minutes data and displayed in the image content PC2.

[0271] In this case, by way of example and not limitation, after capturing the image, the image may be trimmed to fit the size of the image content PC2, and may or may not be displayed on the image content PC2.

[0272] Also, when the image capture button BT5 is tapped, a moving image may or may not be acquired by the image capture unit 27. In this case, a moving image is embedded in the image content PC2, and as a non-limiting example, while the image content PC2 is displayed in the content display area TR1, the moving image is displayed in chronological order. Then, as a non-limiting example, when the image content PC2 is scrolled out of view, when the image content PC2 is scrolled back into view in the content display area TR1, the moving image is played back from the beginning in the image content PC2.

[0273] Furthermore, the timing at which the image content PC2 is displayed does not depend on the tapping of the image capture button BT5. In addition, the image capture unit 27 may continue to capture moving images, and when the difference between frames is relatively large (when the difference between frames is equal to or exceeds a threshold value), the frame image may or may not be automatically displayed as the image content PC2.

[0274] FIG. 3-6 is an example of a display screen when the acquisition of voice data continues following FIG. 3-5.

[0275] By way of example and not limitation, text content CT7 based on user EE's utterance "We're running out of time, let's move on to the next topic" is displayed below image content PC2.

[0276] At this time, when the user of the terminal 20A taps the tag function button BT7 in response to the contents of the user EE's speech, the tag function button BT7 is extended to the left, and a tag input area and a tag reflection function button are displayed. As a non-limiting example, in FIG. 3-6, the characters "agenda 2" are input in the tag input area.

[0277] When the user taps the tag reflection function button, the tag "Agenda 2" is added to the minutes data converted from the audio data thereafter, by way of example and not limitation.

[0278] FIG. 3-7 is an example of a display screen when the acquisition of voice data continues after the tag reflection function button in FIG. 3-6 is tapped.

[0279] In Figure 3-7, as a result of the tag being entered, tag content TG2 is displayed below text content CT7, with the text "About Agenda 2" indicating the content of the entered tag. The painted bands on both sides of tag content TG2 are the same as those in tag content TG1, so a description of them will be omitted.

[0280] As a non-limiting example, a band associated with the tag "Agenda 2" is displayed on the left side of text content CT8 based on user AA's utterance "Now, regarding the contents of Agenda 2." That is, by paying attention to the color of the band, the user can easily determine which portion to refer to when reviewing the minutes data later.

[0281] FIG. 3-8 is an example of a display screen when a further time has elapsed since FIG. 3-7 and the acquisition of voice data continues.

[0282] In FIG. 3-8, by way of example and not limitation, a content highlighting selection area MBT2 is displayed in response to a user tapping on the content highlighting function button BT6.

[0283] In the content highlighting selection area MBT2, collaborative function selection buttons are arranged for changing the display mode of text content based on user operations. By tapping these buttons and then tapping the text content, the display mode of the text content can be changed meaningfully.

[0284] As a non-limiting example, when a selection button indicating "Key Points" in the minutes is tapped and applied to the text content, the edited text content MCT4 is obtained. In the edited text content MCT4, the shape of the speech bubble is changed to a rectangle with the diagonal corners cut off, and a finger mark is added. This allows the user to easily recognize that this content is being treated as a key point in the minutes.

[0285] Similarly, by way of example and not limitation, the content surrounding "approval" in the minutes will become the tortoise shell shaped speech bubbles shown in the edited text content MCT5 through MCT7.

[0286] In addition, in the content highlighting selection area MBT2, the selection button indicating "Revert" may be tapped and then the highlighted content may be subsequently tapped to restore the original bubble-type text content, or it may not be necessary to do so. This allows, for example and not limitation, for content to be restored to its original display state even if it is mistakenly highlighted.

[0287] In Figure 3-8, the user taps the selection button indicating the "Conclusion" of the minutes in the content highlighting selection area MBT2. Then, the user taps the text content CT9 based on the user AA's utterance "We will postpone the release for about a year."

[0288] FIG. 3-9 is an example of a screen after the text content CT9 in FIG. 3-8 is tapped.

[0289] In FIG. 3-9, edited text content MCT8 is displayed in the content display area TR1, and is highlighted with a double-lined balloon indicating "conclusion" for text content CT9.

[0290] Note that the highlighting function for changing the display mode of the text content may, for example and without limitation, be automatically applied based on the results of text analysis (semantic analysis) of the minutes data, or may not be applied in this manner.

[0291] In addition, as content to be coordinated, in addition to "points" and "conclusions," for example and without limitation, "homework" meaning a task for each user may or may not be set. For example and without limitation, when content is highlighted as "homework," the contents of the content may or may not be automatically added to the user's TODO list.

[0292] In FIG. 3-9, below the edited text content MCT8, text content CT10 based on the user AA's utterance "This concludes the conference" is displayed.

[0293] As a non-limiting example, the user of the terminal 20A taps the recording end button BT4 to stop acquiring voice data and adding the minutes data based on the speech of the text content CT10. Then, a minutes end display message EXM1 with the text "Minutes Ended" is added to the talk room. This message notifies the members of the talk room that adding the minutes data has ended.

[0294] When the acquisition of audio data and the addition of minutes data are stopped, by way of example and not limitation, function buttons related to editing and displaying minutes data, such as the edit function button BT1 and the filter function button BT3, will no longer be displayed in the content display area TR1.

[0295] As an example and not a limitation, when a user scrolls up on the content display area TR1 and the contents of the minutes data are displayed, the edit function button BT1 and the filter function button BT3 may be redisplayed in the content display area TR1 so that editing operations can be performed at any time after the minutes data has been added, or this may not be the case.

[0296] Furthermore, when the contents of the minutes data are displayed, a folding function button BT2 may be displayed in the contents display area TR1 to enable folding of the minutes data portion, but this is not essential.

[0297] <Processing> FIG. 3-10 is a flowchart showing an example of the flow of processing executed by each device in this embodiment. From the left, an example of a process executed by the control unit 21 of the terminal 20A and an example of a process executed by the control unit 11 of the server 10 are shown.

[0298] This flowchart is a flowchart obtained by rewriting the flowchart in FIG. 2-16 for processing in the real-time mode in this embodiment.

[0299] First, the control unit 21 of the terminal 20A selects a destination to post the minutes, and transmits the minutes to the server 10 via the communication I / F 22 (A150).

[0300] Note that, before step A150, for example and not limitation, the server 10 may perform the minutes posting candidate selection process, and the terminal 20A may or may not receive the minutes posting candidates. Alternatively, the terminal 20A may or may not perform the minutes posting candidate selection process. In these cases, the minutes posting candidate selection process uses a method other than the method based on voice data.

[0301] Next, the control unit 21 of the terminal 20A determines whether or not to end the acquisition of the voice data based on the user input to the input / output unit 23 (A190).

[0302] When the user selects to end the acquisition of the voice data (A190: YES), the control unit 21 of the terminal 20A transmits minutes end information for ending the acquisition of the voice data and the addition of the minutes data to the server 10 via the communication I / F 22. Then, the control unit 21 of the terminal 20A ends the process.

[0303] If the user does not select to end the acquisition of voice data (A190: NO), the control unit 21 of the terminal 20A accepts input to the sound input unit 25, and acquires voice data including the contents of the user's dialogue, for example and not for limitation. Then, when a silent period exceeding a predetermined time occurs in the voice data, the control unit 21 of the terminal 20A uses this as a delimiter to convert the voice data into partial voice data and stores it in the storage unit 28 (A210). In this case, the partial voice data is a section of the voice data delimited by a silent period.

[0304] Thereafter, the control unit 21 of the terminal 20A transmits the acquired partial voice data to the server 10 via the communication I / F 22 (A220).

[0305] The control unit 11 of the server 10 receives the minutes posting destination from the terminal 20A via the communication I / F 14. After that, the control unit 11 of the server 10 determines whether or not minutes end information has been received from the terminal 20A via the communication I / F 14 (S180).

[0306] When the minutes end information is received from the terminal 20A (S180: YES), the control unit 11 of the server 10 ends the process.

[0307] When the minutes end information is not received from the terminal 20A (S180: NO), the control unit 11 of the server 10 receives the partial voice data from the terminal 20A via the communication I / F 14. Then, the control unit 11 of the server 10 executes a minutes data conversion process on the received partial voice data to create (convert) partial minutes data (S190).

[0308] Note that the minutes data conversion process can be executed similarly to step S120 in FIG. 2-16, for example and not for limitation, and therefore a detailed description thereof will be omitted.

[0309] When the partial minutes data is created, the control unit 11 of the server 10 adds (inputs) the partial minutes data to the chat room specified as the destination for posting the minutes (S200).

[0310] Thereafter, the control unit 11 of the server 10 transmits the partial minutes data added (input) to the chat room to the terminals of the users who belong to the chat room, including the terminal 20A, via the communication I / F 14 (S210). Then, the control unit 11 of the server 10 returns the process to step S180.

[0311] When the partial minutes data is received by the communication I / F 22, the control unit 21 of the terminal 20A causes the display unit 24 to add and display the partial minutes data (A230). Then, the control unit 21 of the terminal 20A returns the process to step A190.

[0312] Incidentally, the step of S200 may not be executed, and the control unit 21 of the terminal 20 may add the partial minutes data to the chat room after receiving the partial minutes data, or it may not be necessary to do so.

[0313] <Effects of the third embodiment> In the third embodiment, based on the first sound information (as an example, not limiting, partial voice data including user AA's utterance "Let's start the meeting for February 2020") acquired by the sound input unit 25 (an example of an acquisition unit, not limiting), the terminal 20 inputs first character information (as an example, not limiting, partial minutes data including the characters "Let's start the meeting for February 2020" linked to user AA) into a chat room, and displays the chat room including the first character information (as an example, not limiting, text content CT1) on the display unit 24. Then, based on the second sound information (as an example, not limited to, partial voice data including user BB's speech "When was the product released?") acquired by the sound input unit 25, the terminal 20 inputs second character information (as an example, not limited to, partial minutes data including the characters "When was the product released?" linked to user BB) into the chat room, and displays a chat room including the second character information (as an example, not limited to, text content CT3) on the display unit 24. As an example of the effect of the embodiment obtained by such a configuration, when the acquisition unit acquires the first sound information, the first character information is displayed on the display unit of the terminal, and when the acquisition unit acquires the second sound information, the second character information is displayed on the display unit of the terminal, so that the time from when the acquisition unit acquires the sound information to when the character information based on the sound information is displayed can be shortened. As a result, the user can refer to the necessary information in real time, and the user's convenience can be improved.

[0314] Furthermore, the third embodiment illustrates a configuration in which terminal 20 inputs image information (as an example, not limiting, image content PC2) including at least a partial area of ​​a captured image (as an example, not limiting, an image acquired by tapping the capture button BT5) into a chat room, and displays a chat room (as an example, not limiting, content display area TR1) including the image information on display unit 24. As an example of the effect of the embodiment obtained by such a configuration, even if it is difficult to express information using text information, by using image information in combination, information can be shared more accurately in a chat room.

[0315] Furthermore, the third embodiment illustrates a configuration in which character information (as an example, not a limitation, text content CT3) is displayed in a first display area of ​​the chat room (as an example, not a limitation, content display area TR1), and image information is displayed in a second display area of ​​the chat room (as an example, not a limitation, below the text content in content display area TR1). As an example of the effect of this embodiment obtained by such a configuration, image information and text information can be displayed in a chat room without interfering with each other.

[0316] <Third Modification (1)> In the third embodiment, the server 10 executes a minutes data conversion process based on the voice data to create minutes data, but is not limited to this. As a non-limiting example, the process may or may not be realized by the above method (2).

[0317] In this case, as an example and not a limitation, when the control unit 21 of the terminal 20 executes step A210 in Fig. 3-10, it executes the partial minutes data conversion process. Each process can be executed similarly to step S190 in Fig. 3-10, so details are omitted. Then, the control unit 21 of the terminal 20 transmits the partial minutes data generated by the partial minutes data conversion process to the server 10 via the communication I / F 22. When the control unit 11 of the server 10 receives the partial minutes data via the communication I / F 14, it executes steps S200 and after in FIG.

[0318] As an example of the effect of this modified example obtained by such a configuration, when the data volume of the partial sound information is large, it becomes possible for the terminal to convert the partial sound information into character information, which is easier to compress the amount of information, before transmitting it to the server, thereby reducing the amount of communication. In addition, when determining the division of the partial voice data, it is possible to determine the division of the partial voice data by detecting the division of phrases from the character information, for example and not by way of limitation, and the partial voice data can be obtained more appropriately. As a result, the accuracy of the partial minutes data conversion process is improved.

[0319] Incidentally, step S200 in FIG. 3-10 may also be performed by the terminal 20, but this is not essential.

[0320] <Fourth Example> The fourth embodiment is an embodiment using the technique (1) regarding the timing of (C) above. In the fourth embodiment, in the minutes-of-meeting service, when a chat room to which minutes-of-meeting data is to be added is selected, the control unit 21 of the terminal 20 displays a selection screen for selecting whether to add the minutes-of-meeting data to the chat room sequentially (real-time input) or to add the audio data all at once after the meeting is ended. The contents described in the fourth embodiment are applicable to any of the other embodiments and other modified examples.

[0321] <Display screen> FIG. 4-1 is an example of a chat room screen when the user taps the “minutes” function icon in FIG. 3-1.

[0322] In FIG. 4-1, the minutes mode selection area WR6 is displayed superimposed on the content display area TR1. In the minutes mode selection area WR6, by way of example and not limitation, a real-time mode selection button identified by the words "Add Real-time" and a batch addition mode selection button identified by the words "Add Later" are displayed.

[0323] When the real-time mode selection button is tapped by the user, a format in which minutes data is added sequentially during the conference according to the real-time mode format is selected, for example and not by way of limitation. Furthermore, when the batch addition mode selection button is tapped by the user, by way of example and not limitation, a format in which minutes data is added in bulk after the end of the conference in accordance with the batch addition mode format is selected.

[0324] Figure 4-2 is an example of the chat room screen when the batch addition mode selection button is tapped in Figure 4-1.

[0325] In FIG. 4-2, by way of example and not limitation, a voice data acquisition status confirmation area STW1 is displayed in an intrusive manner in the content display area TR1 depending on the time when the batch addition mode selection button is tapped. That is, the message MSG1 indicates that it is a message that was input (sent) to this talk room after voice data acquisition started.

[0326] FIG. 4-3 is an example of the talk room screen when, after the conference ends, the user taps the End Recording button BT4 and then taps the minutes data summary content PRC1 that was added to the talk room.

[0327] In FIG. 4-3, when the minutes data summary content PRC1 is tapped, the minutes data display area PRW1 is displayed. In the minutes data display area PRW1, by way of example and not limitation, a message MSG1 received while acquiring the audio data is inserted and displayed immediately below the text content CT5. The display time of the message MSG1 is automatically replaced from the time shown in Figure 4-2 with "00:03:30", which indicates the relative time from the beginning of the audio data.

[0328] As shown in FIG. 4-4, the message MSG1 received during acquisition of the voice data may or may not be displayed without being inserted into the minutes data display area PRW1. As a non-limiting example, in the display example of Figure 4-4, text content based on the results of the acquired audio data is displayed immediately below the text content CT5 in the minutes of meeting data display area PRW1, and the message MSG1 is displayed in the content display area TR1 outside the minutes of meeting data display area PRW1 while maintaining its display time.

[0329] Furthermore, when the real-time mode selection button is tapped in FIG. 4-1, the display screen will be similar to that shown in FIG. 3-2 and subsequent figures, by way of example and not limitation.

[0330] <Processing> FIG. 4-5 is a flowchart showing an example of the flow of processes executed by each device in this embodiment when the batch addition mode selection button is tapped. From the left, an example of a process executed by the control unit 21 of the terminal 20A and an example of a process executed by the control unit 11 of the server 10 are shown.

[0331] This flowchart is a flowchart in which, in the processing of the terminal 20A in the flowchart of FIG. 2-16, step A150 is moved forward, step A110 is replaced with steps A150 and A100, and step S130 is deleted in the processing of the server 10.

[0332] First, the control unit 21 of the terminal 20A selects a destination to post the minutes, and transmits the minutes to the server 10 (A150).

[0333] Note that, before step A150, for example and not limitation, the server 10 may perform the minutes posting candidate selection process, and the terminal 20A may or may not receive the minutes posting candidates. Alternatively, the terminal 20A may or may not perform the minutes posting candidate selection process. In these cases, the minutes posting candidate selection process uses a method other than the method based on voice data.

[0334] Next, the control unit 21 of the terminal 20A acquires voice data by the sound input unit 25, for example and not for limitation (A100). Then, the control unit 21 of the terminal 20A executes each of steps A130 to A180, and then ends the process.

[0335] After executing each step of S120 to S170, the control unit 11 of the server 10 ends the process.

[0336] It should be noted that when the real-time mode selection button is tapped, the same processing as that shown in FIG. 3-10 can be performed, for example and not limitation, so detailed description thereof will be omitted here.

[0337] <Effects of the Fourth Embodiment> The fourth embodiment illustrates a configuration in which a talk room (not limiting, but an example of a chat room) is selected based on, by way of example and not limitation, a tap operation on a real-time mode selection button or a batch addition mode selection button (not limiting, but an example of input by a user of a terminal). As an example of an effect of the embodiment obtained by such a configuration, it is possible for a user to input text information in a chat room specified by the user in a manner intended by the user, thereby improving user convenience.

[0338] <Fourth modified example (1)> In the fourth embodiment, the server 10 executes a conversion process for converting minutes data (partial minutes data) based on the voice data to generate minutes data (partial minutes data), but is not limited to this. As an example and not a limitation, the process may or may not be realized by the above method (2).

[0339] In this case, when the real-time mode selection button is tapped, the same processing as in the third modified example (1) may be performed, and when the batch addition mode selection button is tapped, the same processing as in the second modified example (1) may be performed.

[0340] <Fourth Modification (2)> In the fourth embodiment, the real-time mode and the batch addition mode are applied collectively to all user utterances, but the present invention is not limited to this.

[0341] By way of example and not limitation, the real-time mode may be applied to speech from a particular user (by way of example and not limitation, user CC) and the batch append mode may or may not be applied to speech from other users.

[0342] In this case, while the conference or meeting is in progress, the speech content of user CC is converted into minutes data and shared in the chat room. Then, after the conference or meeting ends, the speech content of other users is converted into minutes data and added to the chat room. Therefore, as a non-limiting example, if user CC is a key person in a conference, the speech content of the key person is shared in the chat room while the conference is in progress, making it easier to progress with the conference. After the conference ends, the speeches of other users are also added to the chat room, making it easy to review the overall flow of the conference.

[0343] The minutes data in the real-time mode and the minutes data in the batch addition mode may be merged into one another after the end of the conference, or may be stored independently.

[0344] <Fourth modified example (3)> In the fourth embodiment, the real-time mode and the batch addition mode are applied collectively to the entire time during the conference (meeting), but the present invention is not limited to this.

[0345] For example and not by way of limitation, minutes data may be created and added in real-time mode only during the period when important points of the meeting are being discussed, and minutes data may be created and added in batch addition mode during other periods, or it may not be necessary to do so.

[0346] In this case, the contents of discussions on important matters are shared in the chat room in real time, allowing users to review them on the spot and preventing misunderstandings about important matters.

[0347] <Fifth Example> In the above embodiment, the minutes data is created based on the audio data stored in the audio database 285. In contrast, in the present embodiment, video (moving image) data is used to create the minutes data based on the audio data (audio track) included in the video data. The contents described in the fifth embodiment are applicable to any of the other embodiments and other modified examples.

[0348] <Display screen example> FIG. 5-1 is a screen diagram showing an example of a display mode of the minutes data display area PRW1 when minutes data is created using video data. FIG. 5-1 shows a display screen on the terminal 20A of user AA by way of example and not by way of limitation.

[0349] Figure 5-1 is, by way of example and not limitation, a screen corresponding to the display screen of Figure 2-14, and, by way of example and not limitation, is an example of a screen after user AA taps on the minutes data summary content PRC1 displayed on the display unit 24 of terminal 20A.

[0350] In Fig. 5-1, a video data display area PP1 is displayed in the lower left corner of the minutes data display area PRW1, where the video data used to create the minutes data is displayed in a picture-in-picture format. That is, the display position of the video data display area PP1 is fixed even if a scroll operation is performed on the minutes data display area PRW1.

[0351] As a non-limiting example, the video data display area PP1 displays an image of the whiteboard at a meeting at the time when text content CT5, which is displayed at the bottom of the minutes data display area PRW1, is spoken from the video data.

[0352] As a non-limiting example, the lower part of the video data display area PP1 displays the elapsed time from the beginning of the video data being played in the video data display area PP1. That is, FIG. 5-1 shows that the video data display area PP1 displays the video corresponding to the speech time of the text content CT5 (the time determined by the elapsed time from the start of the conference).

[0353] As a non-limiting example, when a user performs an upward scroll operation on the minutes data display area PRW1, the content (text content, etc.) displayed in the minutes data display area PRW1 moves back toward the start time of the conference. At this time, the video displayed in the video data display area PP1 also moves back in time to match the content displayed in the minutes data display area PRW1. Conversely, when the user performs a downward scroll operation on the minutes data display area PRW1, the content displayed in the minutes data display area PRW1 advances toward the end time of the conference. In this case, the video displayed in the video data display area PP1 is also fast-forwarded in accordance with the content displayed in the minutes data display area PRW1.

[0354] Note that the video may or may not be fast-forwarded or rewound based on the user's utterance of "fast-forward" or "rewind" instead of the user's scrolling operation. Also, when a tap operation is performed on the minutes data display area PRW1, the video may or may not be paused and displayed.

[0355] That is, the video displayed in the video data display area PP1 is synchronized on the time axis with the content displayed in the minutes data display area PRW1.

[0356] In addition, when the minutes data is created using the video data, it is not essential that the video data display area PP1 is displayed in the minutes data display area PRW1. As a non-limiting example, the video data display area PP1 may or may not be overlaid on the minutes data display area PRW1 based on a user's selection operation.

[0357] Also, as an example and not by way of limitation, in Figure 2-7, a row showing multiple audio data and video data in the audio data list information display area WR1 may be selected (tapped) by the user, and then an audio data selection confirmation button (not shown) may be tapped to create one piece of minutes data from the multiple audio data and video data, or this may not be the case.

[0358] <Processing> The processing in this embodiment can be realized in the same manner as in FIG. 2-16 by regarding the audio track accompanying the video data as audio data, as an example and not a limitation, so a detailed description will be omitted here.

[0359] In this embodiment, the video (moving image) displayed in the video data display area PP1 is controlled to be played back in synchronization with the content displayed in the minutes data display area PRW1 on the time axis. In other words, the playback position of the moving image is changed according to the position of the content displayed in the minutes data display area PRW1.

[0360] As a method for realizing this, for example and not by way of limitation, in the minutes data conversion process, chapter marks (pointers indicating the division of each scene) are inserted into the video data at the speech start points of the audio tracks used to generate each content, linked to the content. Then, among the contents displayed in the minutes data display area PRW1, the video data can be played back with the chapter mark linked to the content that is the reference for the video displayed in the video data display area PP1 as the playback start point.

[0361] In addition to the above, as a non-limiting example, this can also be achieved by setting the speech start time of the audio track used to generate the first (leading) content to be displayed in the minutes data display area PRW1 in the minutes data conversion process as the playback reference point (playback start time) of the video displayed in the video data display area PP1. Once the playback reference point (playback start time) is determined, the video can be played back from the point (timing) corresponding to the speech time of the text content, thereby synchronizing the video (moving image) displayed in the video data display area PP1 with the content displayed in the minutes data display area PRW1 on the time axis.

[0362] As a non-limiting example, information on the time when the creation of the minutes started (minutes creation start time) and information on the time when the recording of the video started (recording start time) are stored in the storage unit 28. Then, the playback reference point may or may not be determined based on the difference between the minutes creation start time and the recording start time.

[0363] <Effects of the Fifth Example> The fifth embodiment illustrates a configuration in which a moving image (not limiting, an example of image information) of a whiteboard including at least a portion of video data (not limiting, an example of a captured image) is input into a talk room (not limiting, an example of a chat room), and the terminal 20 displays a content display area TR1 including this moving image on the display unit 24. As an example of the effect of the embodiment obtained by such a configuration, even if it is difficult to express information using text information, by using image information in combination, information can be shared more accurately in a chat room.

[0364] In addition, the fifth embodiment shows a configuration in which content such as text content (not limiting, an example of character information) is displayed in the minutes data display area PRW1 (not limiting, an example of the first display area) of the talk room, and image information such as moving images is displayed in the video data display area PP1 (not limiting, an example of the second display area) of the talk room. As an example of the effect of this embodiment obtained by such a configuration, image information and text information can be displayed in a chat room without interfering with each other.

[0365] Furthermore, the fifth embodiment illustrates a configuration in which terminal 20 uses control unit 21 to perform scrolling control of content in the talk room (not limiting, but an example of control for changing the position of the first display area) based on input by the user of terminal 20, and the position of video data display area PP1 in the talk room (not limiting, but an example of the position of the second display area in the chat room) does not change based on input by the user of terminal 20. As an example of an effect of an embodiment obtained by such a configuration, since the position of the second display area is fixed regardless of the control that changes the position of the first display area, the viewability of the second display area can be improved.

[0366] Furthermore, in the fifth embodiment, the image information is information about a moving image, and the terminal 20 is configured to use the control unit 21 to control rewinding or fast-forwarding of the moving image (not limiting, but an example of control for playing a moving image) based on operations such as scrolling up or down by the user of the terminal 20 (not limiting, but an example of input by the user of the terminal). As an example of an effect of the embodiment obtained by such a configuration, since control of playing back moving images can be executed based on user input, convenience for the user can be improved.

[0367] Furthermore, in the fifth embodiment, the control unit 21 controls the terminal 20 to display a portion of the content including the text content (not limited to this, an example of third character information) in the minutes data display area PRW1 (not limited to this, an example of the first display area) based on an input by the user of the terminal 20. The terminal 20 also shows a configuration in which the control unit 21 controls the playback of a moving image based on the content displayed in the minutes data display area PRW1. As an example of an effect of the embodiment obtained by such a configuration, control is performed to display third character information, which is a part of the character information, in the first display area based on an input by a user of the terminal. Then, control to play a moving image based on the third character information displayed in the first display area can be executed, thereby improving user convenience.

[0368] In this case, by way of example and not limitation, the playback position of the video may be changed based on a portion of the content (third character information) displayed in the minutes data display area PRW1. As an example of an effect of an embodiment obtained by such a configuration, the moving image displayed in the second display area can be synchronized on the time axis with the content displayed in the first display area, thereby enabling the playback of moving images in an intuitive and easy-to-understand manner for the user.

[0369] In addition, the fifth embodiment shows a configuration in which the terminal 20 performs scrolling control of content in a talk room (not limited to this, an example of control for changing the position of the first display area) using the control unit 21 based on input by a user of the terminal 20, and the control unit 21 controls the playback of moving images corresponding to the content displayed on the display unit 24 based on the content scrolling control. As an example of an effect of an embodiment obtained by such a configuration, it is possible to execute control to play moving images corresponding to text information displayed on the display unit of the terminal based on control to change the position of the first display area, thereby improving user convenience.

[0370] Moreover, the fifth embodiment shows a configuration in which the input by the user of the terminal 20 includes an input by a scroll operation. As an example of an effect of an embodiment obtained by such a configuration, a user of a terminal can play moving images by performing a simple and intuitive operation of scrolling, thereby improving user convenience.

[0371] <Fifth Modification (1)> In the fifth embodiment, an example in the batch addition mode has been described, but the present invention is not limited to this. As an example and not a limitation, the minutes data may or may not be created based on the video data in the real-time mode.

[0372] In this case, as a non-limiting example, video data may be acquired using the imaging unit 27, and partial audio data may be created from the audio track of the video data.

[0373] As for the display screen, a video data display area PP1 is displayed on the lower left side of the content display area TR1, and the display position of the video data display area PP1 is fixed even if a scrolling operation is performed on the content display area TR1.

[0374] At this time, when the user performs an upward scroll operation on the content display area TR1, the video displayed in the video data display area PP1 is rewound to match the content displayed in the content display area TR1. Conversely, when the user performs a downward scroll operation on the content display area TR1, the video displayed in the video data display area PP1 is displayed at fast forward in accordance with the content displayed in the content display area TR1.

[0375] <Fifth Modification (2)> In the fifth embodiment, when creating minutes data using video (moving image) data, the video is displayed in the video data display area PP1, but this is not limiting. As a non-limiting example, when the edit function button BT1 is tapped and an operation to insert a video (moving image) is selected, the video may or may not be displayed in the video data display area PP1.

[0376] Even in this case, the video displayed in the video data display area PP1 may, by way of example and not limitation, be subjected to display operations such as fast forwarding and rewinding by a scrolling operation on the content display area TR1, or it may not be subjected to such operations.

[0377] <Sixth Example> The sixth embodiment is an embodiment related to a display mode of contents (messages) displayed in a chat room (talk room). In the sixth embodiment, the manner in which the same content is displayed in a chat room is changed depending on the type of messenger (messaging service, messaging application). The contents described in the sixth embodiment are applicable to any of the other embodiments and other modified examples.

[0378] FIG. 6-1 is a diagram showing an example of a display format table TBL1 that defines a display mode for each type of messenger. This display format table TBL1 is stored in the storage unit 15 of the server 10, for example and not for limitation.

[0379] In the display format table TBL1, for each type of messenger, by way of example and not limitation, a line break position, a speech bubble frequency, and an emoji display are defined in association with each other.

[0380] The line break position is the number of characters that specifies the length of one line in the content. If the number of characters specified in the line break position is exceeded, then, by way of example and not limitation, a line break will be automatically inserted and displayed.

[0381] The callout frequency defines the number of lines that may be contained within a single piece of text content, and if the number of lines defined by the callout frequency is exceeded, then, by way of example and not limitation, the text content may be automatically split and displayed as multiple pieces of content.

[0382] Emoji notation is a provision related to the display of emojis, and by way of example, and not limitation, emojis contained in content may be displayed as pictures (defined as "emojis"), converted into text according to the content of the emoji and displayed (defined as "text conversion"), or deleted and displayed (defined as "deleted").

[0383] Here, emoji refers, by way of example and not by way of limitation, to things or events that are defined as Emoji in the Unicode 6 format or later and that use pictures like letters to symbolically indicate things or events.

[0384] As an example and not by way of limitation, the display format table TBL1 specifies that in "Messenger A," a line break is inserted every 28 characters of text in the content, and if the text exceeds 10 lines, it is split into a different content, and emoticons are displayed as emoticons.

[0385] In addition, by way of example and not limitation, "Messenger B" specifies that a line break should be inserted every 16 characters of text in the content, and if the text exceeds four lines, it should be split into separate content, and emojis should be displayed as emojis.

[0386] When displaying a text consisting of the same length of characters, "Messenger B" contains fewer characters in a single piece of content than "Messenger A," and also contains fewer characters per line. In other words, it is presumed that the display area of ​​"Messenger B" is narrower than the display area of ​​"Messenger A."

[0387] As a non-limiting example, "Messenger C" specifies that a line break should be inserted every 30 characters of text in the content, and if the text exceeds 12 lines, it should be split into separate content, and emojis should be converted to text and then displayed.

[0388] In this case, since "Messenger C" contains a large number of characters in a single piece of content and the information in the content is replaced with characters, it is inferred to be a messenger specialized in displaying text.

[0389] As a non-limiting example, "Messenger D" specifies that a line break is inserted every 140 characters of text in the content, each piece of content contains only one sentence, and emojis are deleted (omitted) before display.

[0390] From this, it can be inferred that "Messenger D" is a messenger specialized for exchanging short messages.

[0391] The control unit 11 of the server 10 formats the minutes data into a format defined in the display format table TBL1, for example and not by way of limitation, based on the messenger used, and adds (inputs) the data to the chat room.

[0392] After receiving the minutes data from the server 10, the display mode may or may not be changed in the terminal 20A according to the display format table TBL1.

[0393] <Effects of the Sixth Embodiment> The sixth embodiment illustrates a configuration in which the text content of the minutes data (not limiting, but an example of character information) is input into a chat room in a manner based on the type of messenger A (not limiting, but an example of a chat service). As an example of the effect of the embodiment obtained by such a configuration, the display mode of the text information can be changed based on the characteristics of the chat service, thereby improving the user's viewability.

[0394] Moreover, the sixth embodiment shows a configuration in which text information is entered into a chat room with line breaks inserted at line break positions based on the type of chat service. As an example of the effect of the embodiment obtained by such a configuration, the number of characters in one line can be changed according to the characteristics of the chat service, thereby improving convenience for the user.

[0395] <Sixth Modification (1)> In the sixth embodiment, the control unit 11 of the server 10 formats the minutes data in accordance with the display format table TBL1, but the present invention is not limited to this. By way of example and not limitation, shaping of the minutes data according to the type of messenger may or may not be performed at the terminal 20 .

[0396] In this case, by way of example and not limitation, a display format table TBL1 is stored in the storage unit 28 of the terminal 20. Then, when the control unit 21 of the terminal 20 receives the minutes data from the server 10, the control unit 21 forms each content in accordance with the display format table TBL1 depending on the type of messenger, and displays the contents on the display unit 24.

[0397] <Sixth Modification (2)> In the sixth embodiment, the display format table TBL1 defines the display mode for each type of messenger, but the present invention is not limited to this.

[0398] For example and not limitation, the display manner may or may not be determined depending on the size of the display area of ​​the display unit 24 of the terminal 20.

[0399] In this case, as an example and not a limitation, when the display unit 24 has a display screen larger than 1334 x 750 pixels, the content can be formed in the display mode of Messenger A, and when the display screen is 1334 x 750 pixels or less, the content can be formed in the display mode of Messenger B.

[0400] <Seventh Example> The seventh embodiment is an embodiment in which a viewing restriction is set on the minutes data displayed in a chat room (talk room) depending on the attendance status of a conference or meeting. The seventh embodiment is an embodiment relating to the timing of (A) above. The contents described in the seventh embodiment are applicable to any of the other embodiments and other modified examples.

[0401] <Data Structure> FIG. 7-1 shows an example of the data structure of the calendar information management database 159B, which is an example of the calendar information management database 159 in this embodiment. Calendar information management database 159B stores calendar information management data as management data for each event registered in the calendar of the schedule management application.

[0402] Each calendar information management data may store, by way of example and not limitation, an event ID, an MS application ID, an event name, an event start date and time, an event end date and time, an event location, an associated URI, invitee data, and attendance management data.

[0403] The event ID, MS application ID, event name, event start date and time, event end date and time, event location, related URIs, and invitee data are, by way of example and not limitation, similar to those in calendar information management database 159A, and therefore will not be described again.

[0404] The attendance management data stores information about users invited to an event identified by the event ID, such as, but not limited to, an MS application ID, an attendance time, and a departure time, in association with each other.

[0405] The MS application ID is, by way of example and not limitation, the MS application ID of the user stored in the invitee data.

[0406] The attendance time is, by way of example and not limitation, the time when the user's terminal 20 identified by the MS application ID arrives within the reception range of a beacon signal emitted from a beacon device installed at the event venue identified by the event ID.

[0407] The time of leaving is, by way of example and not limitation, the time when the user's terminal 20 identified by the MS application ID leaves the range of reception of a beacon signal transmitted from a beacon device installed at the event venue identified by this event ID.

[0408] That is, the arrival time and departure time are stored in the attendance management data after the start date and time of the event identified by this event ID.

[0409] In addition, if a terminal repeatedly approaches and leaves a beacon device after the event start date and time, the attendance management data may or may not store multiple pairs of attendance time and departure time. Also, if a terminal does not approach a beacon device, the attendance time and departure time may or may not be set to a "NULL" value, which indicates that the user was absent, as an example and not a limitation.

[0410] Also, the method of acquiring the arrival time and the departure time is not limited to the method using a beacon device. As a non-limiting example, the location information is acquired in the location calculation information detection unit 29B of each terminal 20 based on location information that can be identified by the latitude and longitude of the event venue. When the terminal 20 moves to a location that matches the location information of the event venue within a predetermined range, the time may or may not be set as the arrival time of the user of the terminal 20. When the terminal 20 leaves a location that matches the location information of the event venue within a predetermined range, the time may or may not be set as the departure time of the user of the terminal 20.

[0411] Also, by way of example and not limitation, a camera installed at the event venue or the imaging unit 27 of the terminal 20 may be used to capture images during the event, and facial recognition may be performed on the images to obtain the user's attendance status, or it may not be necessary to do so.

[0412] Alternatively, as an example and not by way of limitation, sound (including voice) may be acquired during the event using a microphone installed at the event venue or the sound input unit 25 of the terminal 20, and sound recognition (including voice recognition) may be performed on the sound to acquire the user's attendance status, or it may not be necessary to do so.

[0413] Alternatively, by way of example and not limitation, user attendance may or may not be obtained based on user input to terminal 20, such as stamping arrival and departure times.

[0414] When performing the minutes data conversion process, the mention function may or may not be activated based on the arrival and departure times to only users or groups who were not present at the conference or meeting for which the audio data was obtained.

[0415] <Display screen> FIG. 7-2 is an example, by way of example and not limitation, of a display screen of a chat room of a messaging application when the user taps the minutes posting button SBT1 in FIG. 2-12.

[0416] In FIG. 7-2, in the content display region TR1, as a non-limiting example, minutes data summary content PRC2 is displayed below the text content "These are the minutes of the meeting just now" sent by user AA.

[0417] The minutes data summary content PRC2 displays, by way of example and not limitation, the minutes title and the date and time when the minutes data was last modified, similarly to the minutes data summary content PRC1. Also, the minutes data summary content PRC2 displays an attendance confirmation function button BT8 for confirming the attendance status of each user at the time when the voice data used to create this minutes data was acquired.

[0418] FIG. 7-3 is an example of a display screen when the attendance confirmation function button BT8 is tapped by the user. By way of example and not limitation, when the attendance confirmation function button BT8 is tapped by the user, an attendance confirmation status display area WR7 is displayed in the content display area TR1.

[0419] At the left end of the attendance confirmation status display area WR7, icons representing users of the MS application IDs stored in the attendance management data are displayed in a row, by way of example and not limitation. To the right of each user's icon, a graph is displayed for each row, by way of example and not limitation, showing whether the user was present or not during the time period from the start to the end of acquisition of the voice data that is the source of the minutes data. These graphs are drawn based on, by way of example and not by way of limitation, the attendance times and departure times of each user stored in the attendance management data.

[0420] In the attendance confirmation status display area WR7, it is displayed, by way of example and not limitation, that users AA, BB, and CC attended the conference (event) from start to finish, that user DD attended from 20 minutes after the start, and that user EE left one hour after the start.

[0421] FIG. 7-4 is an example of a display screen of the talk room when the minutes data is displayed on the terminal 20D of the user DD.

[0422] In Fig. 7-4, the minutes data display area PRW2 is displayed in the content display area TR1 based on the user tapping the minutes data summary content PRC2. In addition to the display elements of the minutes data display area PRW1, an attendance confirmation function button BT8 is displayed in the upper part of the minutes data display area PRW2.

[0423] Based on the fact that user DD was absent from the start of voice data acquisition until 20 minutes, the minutes data display area PRW2 displays the text "Viewing is locked from this point onwards" and a lock status icon IC1 with a locked padlock, indicating that viewing of the content portion from 20 minutes prior to the start of voice data acquisition is restricted. In other words, the terminal 20D cannot display the content of the minutes data for the time period during which user DD was absent.

[0424] As an example and not as a limitation, when the lock status icon IC1 is tapped by the user, a selection screen for deciding whether or not to lift the viewing restriction is displayed on the display unit 24 of the terminal 20A of the user AA who posted the minutes data. When the viewing restriction is selected to be lifted based on a user operation, the content portion of the minutes data display area PRW2 from 20 minutes before becomes viewable on the terminal 20D, and the content is displayed.

[0425] In addition, when a user taps the lock status icon IC1, a selection screen for whether or not to lift the viewing restriction may be displayed to all users in the chat room to which the minutes data has been posted, or it may not be necessary to do so.

[0426] Similar to the display screen example above, on the chat room display screen when displaying minutes data on user EE's terminal 20D, viewing restrictions are placed on the content portion of the minutes data from 60 minutes onwards, based on the fact that user EE has been absent for 60 minutes or more since the start of audio data acquisition.

[0427] <Processing> FIG. 7-5 is a flowchart showing an example of the flow of processes executed by each device in this embodiment. From the left, an example of a process executed by the control unit 21 of the terminal 20D, a process executed by the control unit 21 of the terminal 20A, and a process executed by the control unit 11 of the server 10 are shown.

[0428] First, the control unit 21 of the terminal 20A and the control unit 11 of the server 10 execute a minutes posting process. The minutes posting process can be realized by the same process as that shown in FIG. 2-16, for example and not by way of limitation, and therefore a detailed description thereof will be omitted.

[0429] Next, the control unit 11 of the server 10 generates minutes restriction data for each target user (who has a time period during which they were absent) in order to make the content for the time period during which each user was absent invisible (S300).

[0430] Then, the control unit 11 of the server 10 transmits the minutes restriction data to the target terminal 20 via the communication I / F 14 (S310).

[0431] When the control unit 21 of the terminal 20D receives the minutes restriction data from the server 10 via the communication I / F 22, it imposes a viewing restriction on the minutes data and causes the display unit 24 to display it (D100).

[0432] When it is selected to lift the viewing restriction on the minutes data based on a user operation on the input / output unit 23, the control unit 21 of the terminal 20D transmits viewing restriction lift request information to the server 10 via the communication I / F 22 (D110). Note that when it is selected not to lift the viewing restriction on the minutes data, the control unit 21 of the terminal 20D ends the process.

[0433] When viewing restriction removal request information is received from the terminal 20D via the communication I / F 14, the control unit 11 of the server 10 transmits viewing restriction removal confirmation information to the terminal 20A via the communication I / F 14 (S320). When viewing restriction removal request information is not received, the control unit 11 of the server 10 ends the process.

[0434] In addition, the control unit 11 of the server 10 may, or may not, send the confirmation information for removing the viewing restriction to all other user terminals (other than the one that received the request information for removing the viewing restriction) that belong to the talk room to which the minutes data is posted.

[0435] When the control unit 21 of the terminal 20A receives the viewing restriction lift confirmation information from the server 10 via the communication I / F 22, it causes the display unit 24 to display a selection screen for whether or not to lift the viewing restriction (A250). Note that when the viewing restriction lift confirmation information is not received, the control unit 21 of the terminal 20A ends the process.

[0436] When it is selected to release the viewing restriction based on a user operation on the input / output unit 23, the control unit 21 of the terminal 20A transmits the viewing permission information to the server 10 via the communication I / F 22 (A260) and ends the process. Note that when it is selected not to release the viewing restriction, the control unit 21 of the terminal 20A ends the process.

[0437] When viewing restriction release confirmation information is transmitted to a terminal other than the terminal 20A, processing similar to steps A250 and A260 may be executed.

[0438] When the viewing permission information is received from the terminal 20A via the communication I / F 14, the control unit 11 of the server 10 executes a viewing restriction release process to enable the minutes data to be confirmed on the terminal that sent the viewing restriction release request information (S330). When the viewing permission information is not received, the control unit 11 of the server 10 ends the process.

[0439] Then, the control unit 11 of the server 10 transmits the minutes data to the terminal 20D via the communication I / F 14 (S340), and ends the process.

[0440] In addition, when the viewing restriction release confirmation information is transmitted to a plurality of terminals, the viewing restriction release process may be executed, for example and not limited to, in the following cases. 1) When receiving viewing permission information from all devices that have sent viewing restriction removal confirmation information 2) When viewing permission information is received from the majority of devices that sent viewing restriction removal confirmation information 3) When receiving viewing permission information from the device that posted the minutes data 4) When viewing permission information is received from the device of the leader of that chat room.

[0441] When the control unit 21 of the terminal 20D receives the minutes data from the server 10 via the communication I / F 22, it updates and displays the data on the display unit 24 (D120). Then, the control unit 21 of the terminal 20D ends the process.

[0442] <Effects of the Seventh Embodiment> The seventh embodiment illustrates a configuration in which minutes data (not limiting, an example of text information) is entered into a talk room (not limiting, an example of a first chat room) of group "X" including user AA (not limiting, an example of a user of a terminal), user BB (not limiting, an example of a user of a first terminal), and user DD (not limiting, an example of a user of a second terminal), and is transmitted to at least terminal 20D (not limiting, an example of a second terminal). As an example of the effect of the embodiment obtained by such a configuration, the user of the terminal, the user of the first terminal, and the user of the second terminal can share information through the first chat room, thereby improving the convenience of the users.

[0443] Furthermore, the seventh embodiment illustrates a configuration in which some of the contents (not limited to, an example of the fourth character information which is some of the character information) of the minutes data transmitted to terminal 20D (second terminal) are not displayed on display unit 24 of terminal 20D. As an example of the effect of the embodiment obtained by such a configuration, since a part of the text information is not displayed on the display unit of the second terminal, it is possible to restrict the information shared with the user of the second terminal.

[0444] In addition, the seventh embodiment shows a configuration in which some of the contents of the minutes data (not limited to this, but an example of the fourth character information) are displayed on the display unit of the second terminal with permission based on user AA (not limited to this, but an example of a user of the terminal) and user BB (not limited to this, but an example of a user of the first terminal). As an example of the effect of the embodiment obtained by such a configuration, if there is a permission based on the user of the terminal and the user of the first terminal, the user of the second terminal can view the information that was restricted. Therefore, the range of information sharing can be flexibly set.

[0445] In addition, the seventh embodiment shows a configuration in which the content (not limiting, an example of the fourth character information) of the minutes data up to a first predetermined time after the start (for example, not limiting, 20 minutes after the start) is content (not limiting, an example of the fifth character information) based on audio data (not limiting, an example of the third sound information) up to a first predetermined time after the start acquired by the sound input unit 25 before information (for example, not limiting, information on the approach of a beacon device) regarding terminal 20D (for example, not limiting, an example of a second terminal) is acquired by terminal 20 or server 10 communicating with terminal 20. As an example of an effect of the embodiment obtained by such a configuration, it is possible to determine the fourth character information in conjunction with information about the second terminal. Also, the fifth character information is character information until the information about the second terminal is acquired. As a result, it is possible to prevent the character information before the information about the second terminal is acquired from being displayed on the second terminal.

[0446] In addition, the seventh embodiment shows a configuration in which the content (not limiting, an example of the fourth character information) from the second predetermined time after the start of the minutes data (as an example, not limiting, one hour after the start) is based on audio data (not limiting, an example of the fourth sound information) from the second predetermined time after the start of the minutes data acquired by the sound input unit 25 after information (not limiting, as an example, not limiting, information about the departure of the beacon device) is acquired by the terminal 20 or the server 10 communicating with the terminal 20. As an example of an effect of the embodiment obtained by such a configuration, the sixth character information is character information after the information about the second terminal is acquired, and as a result, the character information after the information about the second terminal is acquired can be prevented from being displayed on the second terminal.

[0447] Moreover, the seventh embodiment shows a configuration in which the information on the second terminal includes information on the position of the second terminal. As an example of an effect of an embodiment obtained by such a configuration, based on information regarding the position of the second terminal, it is possible to easily and appropriately prevent character information before information regarding the second terminal is acquired from being displayed on the second terminal, or prevent character information after information regarding the second terminal is acquired from being displayed on the second terminal.

[0448] <Seventh Modification (1)> In the seventh embodiment, a case where a viewing restriction is set on the minutes data displayed in the chat room depending on the user's attendance status has been described, but this is not limited to this. As a non-limiting example, multiple chat rooms (talk rooms) may be prepared, and depending on the user's attendance status, the minutes data may be divided and added (input) to each chat room, or it may not be.

[0449] FIG. 7-6 is an example of a display screen of a talk room in this modified example. In Figure 7-6, the current location display area has been added with the display of "(4 / 5)," indicating that the chat room is made up of four of the five members of the group "Project X." Below that, icons of the users who belong to this chat room, User AA, User BB, User CC, and User EE, are displayed.

[0450] Below that, a content display area TR2 is displayed for displaying content (messages) to be sent (posted) to this talk room.

[0451] Based on the fact that user DD was absent from the start of the meeting until 20 minutes, the minutes data is input into the content display area TR2 according to the display format of the third embodiment, by way of example and not limitation. That is, the content added (input) to the content display area TR2 is the minutes data from the start of the acquisition of the audio data until 20 minutes later, and is the content that can be viewed by members of the group "Project X" other than user DD.

[0452] At the bottom of the content display area TR2, a talk room transition function button IC2 is displayed, which displays the icons of all members of the group "Project X." By way of example and not limitation, when the user taps the talk room transition function button IC2, a talk room screen that can be viewed by all members of the group "Project X" is displayed.

[0453] FIG. 7-7 is an example of a talk room display screen when the talk room transition function button IC2 in FIG. 7-6 is tapped by the user.

[0454] In Figure 7-7, the current location display area has been added with the display of "(5 / 5)," indicating that this is a chat room made up of five users out of the five members of the group "Project X." Below that, icons of all the users who belong to this chat room are displayed.

[0455] Below that, a content display area TR1 is displayed for displaying content (messages) to be sent (posted) to this talk room (that is, the talk room of the group "Project X").

[0456] In the content display area TR1, minutes data is entered according to the display format of the third embodiment, as an example and not a limitation, based on the fact that user DD attended the meeting 20 minutes after it started and user EE left the meeting one hour after it started.

[0457] The content displayed in the content display area TR1 is content that can be viewed by all members of the group "Project X." Of the minutes data, the minutes data from 20 minutes after the start of audio data acquisition to one hour after the start of audio data acquisition is added and displayed in the content display area TR1.

[0458] At the top of the content display area TR1, a talk room transition function button IC3 is displayed, which shows the icons of the members of group "Project X" other than user DD. At the bottom of the content display area TR1, a talk room transition function button IC4 is displayed, which shows the icons of the members of group "Project X" other than user EE.

[0459] As a non-limiting example, when the user taps the talk room transition function button IC3, a talk room (i.e., content display area TR2) that can be viewed by members of the group "Project X" other than user DD is displayed. By way of example and not limitation, user DD can view the content in content display area TR1, but even if he taps the talk room transition function button IC3, the talk room does not transition and he cannot view the content entered in content display area TR2. That is, user DD cannot view the minutes data from the start of the meeting up to 20 minutes after the start, which he did not attend.

[0460] In addition, when the user taps the talk room transition function button IC4, talk rooms that can be viewed by members of the group "Project X" other than user EE are displayed. In this case, even if user EE taps the talk room transition function button IC4, the talk room does not transition and the content entered in the transitioned talk room cannot be viewed. In other words, user EE cannot view the minutes data of a meeting that has been held for more than one hour since the start, even if he or she is not present at the meeting.

[0461] It is not essential to display the talk room transition function button in the content display area TR1. When the talk room transition function button is not displayed, transition between talk rooms may or may not be performed based on a user operation, for example and not by way of limitation.

[0462] Also, as a non-limiting example, a chat room may be automatically generated with members being users who are present from the start to the end of the meeting (event) (in this case, users AA, BB, and CC), and the entire minutes data may be added (input) to this chat room, or it may not be necessary to do so.

[0463] This modified example shows a configuration in which content (not limiting, an example of the fourth character information) of the minutes data up to the third predetermined time after the start (for example, not limiting, 20 minutes after the start) is input into a second talk room (for example, not limiting, content display area TR2) that includes user AA (for example, not limiting, an example of a user of the terminal) and user BB (for example, not limiting, an example of a user of the first terminal) but does not include user DD (for example, not limiting, an example of a user of the second terminal), and character information that does not include the fourth character information is input into the first talk room (for example, not limiting, content display area TR1). As an example of an effect of the modified example obtained by such a configuration, since the fourth character information is input to the second talk room not including the user of the second terminal, and the character information not including the fourth character information is input to the first talk room, the character information input to each talk room is different, so that the restriction on viewing the character information can be implemented more naturally without the user of the second terminal being aware of it.

[0464] Also, in this modified example, a transition from the second talk room (second chat room) to the first talk room (first chat room) and a transition from the first talk room to the second talk room are performed by terminal 20A (not limited to, but an example of a terminal) and terminal 20B (not limited to, but an example of a first terminal). On the other hand, a transition from the first talk room to the second talk room is not performed by terminal 20 other than these terminals 20, such as terminal 20D or terminal 20E (not limited to, but an example of a second terminal). As an example of the effect obtained by such a configuration, it is possible to appropriately restrict the user of the second terminal from viewing text information.

[0465] <Seventh Modification (2)> In the seventh embodiment, a viewing restriction is imposed on the minutes data received by the terminal 20, but this is not limiting. As a non-limiting example, the server 10 may or may not restrict viewing of the minutes data, and the control unit 11 of the server 10 may or may not transmit a part of the minutes data to each terminal 20.

[0466] 7-8 are flowcharts showing an example of the flow of processes executed by each device in this embodiment. From the left, an example of a process executed by the control unit 21 of the terminal 20D, a process executed by the control unit 21 of the terminal 20A, and a process executed by the control unit 11 of the server 10 are shown.

[0467] First, the control unit 21 of the terminal 20A and the control unit 11 of the server 10 execute a minutes posting process. The minutes posting process can be realized by the same process as that shown in FIG. 2-16, for example, and not by way of limitation, so a detailed description will be omitted. However, in step S170 of FIG. 2-16, the minutes data is not sent to the terminals of users who did not participate in the entire event (conference) from which the voice data was acquired.

[0468] Next, the control unit 11 of the server 10 generates partially restricted minutes data for each target user (for which there was a time period during which each user was absent), which is minutes data that does not include content from the time period during which each user was absent (S350).

[0469] Then, the control unit 11 of the server 10 transmits the partially restricted minutes data to the target terminal 20 via the communication I / F 14 (S360).

[0470] When the control unit 21 of the terminal 20D receives the partially restricted minutes data from the server 10 via the communication I / F 22, it causes the display unit 24 to display it (D150).

[0471] The control unit 11 of the server 10 executes the viewing restriction removal process (S330), transmits the minutes data to the terminal 20D via the communication I / F 14 (S370), and ends the process.

[0472] When the control unit 21 of the terminal 20D receives the minutes data from the server 10 via the communication I / F 22, it updates the minutes data including the entire text and displays it on the display unit 24 (D120).

[0473] As an example of the effect of this modified example obtained by such a configuration, it is no longer necessary to transmit the entire minutes data to the terminal 20 to which viewing is restricted, so that the amount of communication can be reduced. [Explanation of symbols]

[0474] 1. Communication Systems 10 Server 20 Terminals

Claims

[Claim 1] A program executed by a terminal of a user to display a chat room including a user and a user of a first terminal different from the user, Acquiring sound information by an acquisition unit of the terminal; Character information based on the sound information acquired by the acquisition unit is input into the selected chat room, and the chat room including the character information is displayed on a display unit of the terminal.

Images