Program, information processing device, production method, and information processing method

Abstract

This program causes a computer including a processor and a storage unit to process information on a dialog between a plurality of users, the processor executing: a moving image acquisition step of acquiring moving image data related to a dialog; an object identification step of identifying object information related to an object included in the moving image data on the basis of the moving image data acquired in the moving image acquisition step; a dialog acquisition step of acquiring dialog data related to the dialog; and an object storage step of storing the object information identified in the object identification step in association with the dialog data acquired in the dialog acquisition step and related to the object information.

Description

Program, information processing device, manufacturing method, and information processing method

[0001] The present disclosure relates to a program, an information processing device, a manufacturing method, and an information processing method.

[0002] 2. Description of the Related Art Online communication services that are conducted between multiple users are known. Patent Document 1 discloses a technology for creating minutes of a meeting using a simple method.

[0003] Japanese Patent Application Laid-Open No. 2001-331614

[0004] There is a problem in that object information such as slides displayed to conference participants included in a conference video, character strings included in a document, people and objects such as conference participants, etc. cannot be stored in association with conference audio. Therefore, the present disclosure has been made to solve the above problem, and its purpose is to provide a technology for storing object information such as slides displayed to conference participants included in a conference video, character strings included in a document, superimposed captions and subtitles, people and objects such as conference participants, in association with conference audio.

[0005] A program that causes a computer having a processor and a memory unit to process information regarding a dialogue between multiple users, wherein the processor executes a video acquisition step of acquiring video data regarding the dialogue, an object identification step of identifying object information regarding objects included in the video data based on the video data acquired in the video acquisition step, a dialogue acquisition step of acquiring dialogue data regarding the dialogue, and an object storage step of storing the object information identified in the object identification step in association with the dialogue data acquired in the dialogue acquisition step regarding the object information.

[0006] According to the present disclosure, object information such as slides displayed to conference participants included in the conference video, character strings included in documents, superimposed captions and subtitles, people and objects such as conference participants, etc. can be stored in association with the conference audio.

[0007] 1 is a block diagram showing the functional configuration of the system 1. FIG. 1 is a block diagram showing the functional configuration of the server 10. FIG. 1 is a block diagram showing the functional configuration of the first user terminal 20. FIG. 1 is a block diagram showing the functional configuration of the second user terminal 30. FIG. 2 is a diagram showing the data structure of a user table 1012. FIG. 3 is a diagram showing the data structure of a group table 1013. FIG. 4 is a diagram showing the data structure of a dialogue table 1014. FIG. 5 is a diagram showing the data structure of a label table 1015. FIG. 6 is a diagram showing the data structure of a speech segment table 1016. FIG. 7 is a diagram showing the data structure of an object table 1021. FIG. 8 is a diagram showing the data structure of a customer table 5012. FIG. 9 is a flowchart showing the operation of an incoming call processing. FIG. 10 is a flowchart showing the operation of an object analysis processing. FIG. 11 is a flowchart showing the operation of a search processing. FIG. 12 is an example screen showing the operation of the search processing. FIG. 13 is a flowchart showing the operation of a presentation processing (first embodiment). FIG. 14 is a flowchart showing the operation of a presentation processing (second embodiment). FIG. 15 is a block diagram showing the basic hardware configuration of a computer 90.

[0008] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In all drawings describing the embodiments, common components are designated by the same reference numerals, and repeated description will be omitted. Note that the following embodiments do not unduly limit the content of the present disclosure described in the claims. Furthermore, not all components shown in the embodiments are necessarily essential components of the present disclosure. Furthermore, each drawing is a schematic diagram and is not necessarily a precise illustration.

[0009] <Configuration of System 1> System 1 in the present disclosure is an information processing system that provides an information processing service for evaluating interactions between multiple users. System 1 includes information processing devices: a server 10, a first user terminal 20, a second user terminal 30, a CRM system 50, and a voice server (PBX) 60, all connected via a network N. Fig. 1 is a block diagram showing the functional configuration of system 1. Fig. 2 is a block diagram showing the functional configuration of the server 10. Fig. 3 is a block diagram showing the functional configuration of the first user terminal 20. Fig. 4 is a block diagram showing the functional configuration of the second user terminal 30.

[0010] Each information processing device is configured by a computer equipped with an arithmetic unit and a storage device. The basic hardware configuration of the computer and the basic functional configuration of the computer realized by this hardware configuration will be described later. For each of the server 10, the first user terminal 20, the second user terminal 30, the CRM system 50, and the voice server (PBX) 60, descriptions that overlap with the basic hardware configuration and basic functional configuration of the computer will be omitted.

[0011] <Configuration of Server 10> The server 10 is an information processing device that provides a service of storing and managing data (dialogue data) related to a dialogue (call) between a first user and a second user. The server 10 in the present disclosure is an information processing device that provides an online dialogue service (online dialogue service) between a first user who is an operator and a second user who is a customer. Note that the server 10 in the present disclosure may also be capable of providing an online dialogue service between three or more users, including not only the first user and the second user, but also one or more other users. The server 10 includes a storage unit 101 and a control unit 104.

[0012] <Configuration of Storage Unit 101 of Server 10 > The storage unit 101 of the server 10 includes an application program 1011 , a user table 1012 , a group table 1013 , a dialogue table 1014 , a label table 1015 , a voice segment table 1016 , and an object table 1021 .

[0013] The application programs 1011 are programs for causing the control unit 104 of the server 10 to function as each functional unit. The application programs 1011 include applications such as a web browser application.

[0014] The user table 1012 is a table that stores and manages information about member users (hereinafter referred to as users) who use the service. When a user registers to use the service, the user's information is stored in a new record in the user table 1012. This allows the user to use the service according to the present disclosure. The user table 1012 is a table that uses the user ID as the primary key and has columns for the user ID, group ID, and user name. FIG. 5 is a diagram showing the data structure of the user table 1012.

[0015] The user ID is an item that stores user identification information for identifying a user. The user identification information is an item in which a unique value is set for each user. The group ID is an item that stores group identification information for identifying a group. The user name is an item that stores the name of the user. Instead of the user name, any character string such as a nickname may be set.

[0016] The group table 1013 is a table that stores and manages information (group information) about groups to which users belong. Groups include any group, such as a company, corporation, corporate group, club, or various organizations. Groups may also be defined for more detailed subgroups, such as company departments (sales department, general affairs department, customer support department). The group table 1013 is a table that uses the group ID as the primary key and has columns for group ID, group name, and group attributes. FIG. 6 is a diagram showing the data structure of the group table 1013.

[0017] The group ID is an item that stores group identification information for identifying a group. The group identification information is an item in which a unique value is set for each piece of group information. The group name is an item that stores the name of the group. Any character string can be set as the group name. The group attribute is an item that stores information about the attributes of the group, such as the group type (company, corporate group, other organization, etc.) and industry (real estate, finance, etc.).

[0018] The dialogue table 1014 is a table for storing and managing information (dialogue information) related to dialogues between users and customers. The dialogue table 1014 is a table having a dialogue ID as a primary key and columns for dialogue ID, user ID, customer ID, dialogue category, sending / receiving type, dialogue data, and video data. Figure 7 shows the data structure of the dialogue table 1014.

[0019] The conversation ID is an item that stores conversation identification information for identifying a conversation. The conversation identification information is an item that is assigned a unique value for each conversation information. The user ID is an item that stores user identification information for identifying a user in a conversation between a user and a customer. Multiple user IDs may be associated with each conversation information. The customer ID is an item that stores user identification information for identifying a customer in a conversation between a user and a customer. Multiple user IDs may be associated with each conversation information. The conversation category is an item that stores the type (category) of a conversation between a user and a customer. The conversation data is classified by the conversation category. The conversation category stores values ​​such as telephone operator, telemarketing, customer support, and technical support depending on the purpose of the conversation between the user and the customer. The call type is an item that stores information for distinguishing whether the conversation between the user and the customer was initiated by the user (outbound) or received by the user (inbound). In addition, when a conversation involves three or more users, the call type "room" is stored. The dialogue data is an item that stores dialogue data related to a dialogue. The dialogue data item stores voice data collected by a microphone. The dialogue data item includes text-to-speech obtained by analyzing the voice data. Reference information (path) for voice data files and text data files located in other locations may also be stored. The voice data may be in any format, such as AAC, ATRAC, MP3, or MP4, and the text data may be in any format, such as TXT, CSV, or DOC. The dialogue data is data that includes at least one of voice data, video data, and text-to-speech based on the voice data and video data. The dialogue data includes audio data recorded from a user's dialogue, video data recorded, and any data, such as text-to-speech obtained by converting the voice data and video data. The dialogue data may be data in a format in which identifiers are set that allow the user's voice and the customer's voice to be independently identified.In this case, the control unit 104 of the server 10 can perform independent analysis processes on the user's voice and the customer's voice. Furthermore, the user IDs of the user and the customer can be identified based on the user's and customer's dialogue data. In the present disclosure, video data containing audio information may be used instead of audio data. Furthermore, audio data in the present disclosure also includes audio data contained in video data. Video data is an item that stores video data captured by a camera or the like. Reference information (path) for a video data file located in another location may also be stored. The video data may be in any data format, such as MP4, MOV, WMV, AVI, or AVCHD. The video data may be in a format in which identifiers are set that allow the user's video and the customer's video to be independently identifiable. In this case, the control unit 104 of the server 10 can perform independent analysis processes on the user's video and the customer's video. Furthermore, the user IDs of the user and the customer can be identified based on the user's and customer's video data.

[0020] The label table 1015 is a table for storing and managing information about labels (label information). The label table 1015 has columns for conversation IDs and label data. FIG. 8 shows the data structure of the label table 1015.

[0021] The dialogue ID is an item that stores dialogue identification information for identifying a dialogue. The label data is an item that stores label information for managing dialogues. The label information is additional information for managing dialogue information, such as a classification name, label, classification label, or tag. The label data may be a character string indicating the name of the label information, or may be a label ID for referencing the name of label information stored in another table. The label data includes classification information according to the emotional state of a speaker in a specific dialogue. The classification data includes classification information for classifying the speaker's response in a specific dialogue as good or bad.

[0022] The voice segment table 1016 is a table for storing and managing information (voice segment information) relating to multiple voice segments included in dialogue information. The voice segment table 1016 is a table having a section ID as a primary key and columns for section ID, dialogue ID, speaker ID, start date and time, end date and time, section voice data, section video data, and section read-aloud text. Fig. 9 is a diagram showing the data structure of the voice segment table 1016.

[0023] The section ID is an item that stores section identification information for identifying a voice section. The section identification information is an item in which a unique value is set for each voice section information. The dialogue ID is an item that stores dialogue identification information for identifying a dialogue with which the voice section information is associated. The speaker ID is an item that stores speaker identification information for identifying a speaker with which the voice section information is associated. Specifically, the speaker ID is an item that stores the user IDs of multiple users who participated in the dialogue. The start date and time is an item that stores the start date and time of a voice section or a video section. The end date and time is an item that stores the end date and time of a voice section or a video section. The section audio data is an item that stores audio data included in a voice section. Reference information (path) to an audio data file located in another location may be stored. Furthermore, a reference to audio data for the period from the start date and time to the end date and time of the audio data in the dialogue table 1014 may be stored based on the start date and time and the end date and time. Furthermore, the section audio data may include audio data included in the section video data. The audio data may be in any data format, such as AAC, ATRAC, MP3, or MP4. The section video data is an item that stores video data included in the audio section. It may also store reference information (path) to a video data file located in another location. Furthermore, a reference to video data for a period from the start date and time to the end date and time of the video data in the dialogue table 1014 may be stored based on the start date and time and the end date and time. The video data may be in any data format, such as MP4, MOV, WMV, AVI, or AVCHD. The section reading text is an item that stores text information about the content spoken by a speaker in the section audio data included in the audio section. Specifically, the section reading text may be generated manually or by using a learning model, such as machine learning or deep learning, based on the section audio data and section video data.

[0024] The object table 1021 is a table for storing and managing information about any target object (object) included in the video data of the dialogue information. Objects include information about people, such as the faces and body parts of users participating in the dialogue. Objects include information about objects, such as information about information terminals, electrical appliances, and other items such as furniture. Objects include information about slides (slide information) presented to participants in the dialogue. For example, this includes information about character strings (text), images, photographs, etc. included in the slides, and information about subjects included in the images, photographs, etc. In addition, in the present disclosure, for slides consisting of multiple pages, object information is stored and managed using object information to which different object identification information is assigned for each page. Objects include any document information, such as the contents of word processing software or spreadsheet software, presented to participants in the dialogue. For example, this includes information about word processing files, character strings (text), images, photographs, etc. included in spreadsheet files, and information about subjects included in the images, photographs, etc. The object table 1021 is a table with an object ID as a primary key and columns for object ID, dialogue ID, section ID, speaker ID, start date and time, end date and time, and object data. FIG. 10 is a diagram showing the data structure of the object table 1021. As shown in FIG.

[0025] The object ID is an item that stores object identification information for identifying an object. The object identification information is an item that has a unique value set for each piece of object information. The dialogue ID is an item that stores dialogue identification information for identifying a dialogue with which object information is associated. The section ID is an item that stores section identification information for voice section information associated with object information. Specifically, one or more section IDs of voice section information included in the period from the start to the end of presentation of object information to participants are stored. The speaker ID is an item that stores user identification information for identifying a speaker. Specifically, user identification information of a user associated with an object identified by object identification information in a dialogue, etc. is stored. For example, if the object is information about a person, user identification information for identifying the person is stored. For example, if the object is slide information or document information, user identification information of a user explaining the slide, etc. is stored. The start date and time is an item that stores the presentation start date and time when an object identified by object identification information in a dialogue is presented to participants. The end date and time is an item that stores the presentation end date and time when an object identified by object identification information in a dialogue is presented to participants. Specifically, the object table 1021 stores information about objects identified by object identification information that are presented to conference participants from the start date and time to the end date and time. Object data is an item that stores information about objects (object information). Object data about a person includes information about the face or the person's body. Object data about a person includes features related to a face image (such as the shape of the face, the position and size of the eyes, the shape of the nose and mouth), fingerprint data, retinal data, and other body features. Object data about a person may also include attribute information about a person, such as the name, age, sex, address, occupation, contact information, hobbies, educational history, and family relationships of an individual identified based on the features. Object data about an object includes information about the name, type, and shape of the object.Other information may include specifications, model numbers, and the like of objects identified based on the objects. Object data related to slides includes image data of slides presented to participants in a dialogue. Object data related to word processing software and spreadsheet software includes the contents of word processing files and spreadsheet files presented to participants in a dialogue. Object data related to slides and documents may store character strings recognized by applying known OCR (Optical Character Reader) technology based on slide images. Object data related to slides may be identified by referencing slide files such as PowerPoint and Keynote and character strings contained in the slide files. Object data related to documents may be identified by referencing word processing files and spreadsheet files such as Word, Ichitaro, Google Document, Excel, and Google Spreadsheet and character strings contained in the files.

[0026] <Configuration of control unit 104 of server 10> The control unit 104 of the server 10 includes a user registration control unit 1041, an analysis unit 1042, a search unit 1043, and a presentation unit 1044. The control unit 104 executes an application program 1011 stored in the storage unit 101, thereby realizing each functional unit.

[0027] The user registration control unit 1041 performs a process of storing information about users who wish to use the service disclosed herein in the user table 1012. Information stored in the user table 1012 is generated when a user opens a webpage operated by the service provider from an arbitrary information processing terminal, enters information into a predetermined input form, and transmits the information to the server 10. The user registration control unit 1041 stores the received information in a new record in the user table 1012, completing the user registration. This allows the user stored in the user table 1012 to use the service. Prior to the user registration control unit 1041 registering the user information in the user table 1012, the service provider may conduct a predetermined screening process to restrict whether the user can use the service. The user ID may be any string or number that can identify the user, any string or number desired by the user, or an arbitrary string or number may be automatically set by the user registration control unit 1041.

[0028] The analysis unit 1042 executes an object analysis process, the details of which will be described later.

[0029] The search unit 1043 executes a search process, the details of which will be described later.

[0030] The presentation unit 1044 executes a presentation process, the details of which will be described later.

[0031] <Configuration of First User Terminal 20> The first user terminal 20 is an information processing device operated by a first user who uses a service. The first user terminal 20 may be, for example, a desktop personal computer (PC) or a laptop PC, or may be a mobile terminal such as a smartphone or a tablet. It may also be a wearable terminal such as an HMD (Head Mount Display) or a wristwatch terminal. The first user terminal 20 includes a storage unit 201, a control unit 204, an input device 206, and an output device 208.

[0032] <Configuration of Storage Unit 201 of First User Terminal 20> The storage unit 201 of the first user terminal 20 includes a first user ID 2011 and an application program 2012.

[0033] The first user ID 2011 stores user identification information of the first user. The user transmits the first user ID 2011 from the first user terminal 20 to the voice server (PBX) 60. The voice server (PBX) 60 identifies the first user based on the first user ID 2011 and provides the first user with the service described herein. The first user ID 2011 includes information such as a session ID temporarily assigned by the voice server (PBX) 60 to identify the user using the first user terminal 20.

[0034] The application program 2012 may be pre-stored in the storage unit 201, or may be downloaded from a web server operated by a service provider via a communication IF. The application program 2012 includes an application such as a web browser application. The application program 2012 includes an interpreter-type programming language such as JavaScript (registered trademark) that is executed on the web browser application stored in the first user terminal 20.

[0035] <Configuration of control unit 204 of first user terminal 20> The control unit 204 of the first user terminal 20 includes an input control unit 2041 and an output control unit 2042. The control unit 204 executes an application program 2012 stored in the storage unit 201, thereby realizing each functional unit.

[0036] <Configuration of Input Device 206 of First User Terminal 20 > The input device 206 of the first user terminal 20 includes a camera 2061 , a microphone 2062 , a position information sensor 2063 , a motion sensor 2064 , and a keyboard 2065 .

[0037] <Configuration of Output Device 208 of First User Terminal 20 > The output device 208 of the first user terminal 20 includes a display 2081 and a speaker 2082 .

[0038] <Configuration of Second User Terminal 30> The second user terminal 30 is an information processing device operated by a second user who uses a service. The second user terminal 30 may be, for example, a mobile terminal such as a smartphone or tablet, or may be a stationary personal computer (PC) or laptop PC. It may also be a wearable terminal such as a head-mounted display (HMD) or a wristwatch-type terminal. The second user terminal 30 includes a storage unit 301, a control unit 304, an input device 306, and an output device 308.

[0039] <Configuration of Storage Unit 301 of Second User Terminal 30 > The storage unit 301 of the second user terminal 30 includes an application program 3012 and a telephone number 3013 .

[0040] The application program 3012 may be pre-stored in the storage unit 301, or may be downloaded from a web server operated by a service provider via a communication IF. The application program 3012 includes an application such as a web browser application. The application program 3012 includes an interpreter-type programming language such as JavaScript (registered trademark) that is executed on the web browser application stored in the second user terminal 30.

[0041] <Configuration of the control unit 304 of the second user terminal 30> The control unit 304 of the second user terminal 30 includes an input control unit 3041 and an output control unit 3042. The control unit 304 executes an application program 3012 stored in the storage unit 301, thereby realizing each functional unit.

[0042] <Configuration of Input Device 306 of Second User Terminal 30 > The input device 306 of the second user terminal 30 includes a camera 3061 , a microphone 3062 , a position information sensor 3063 , a motion sensor 3064 , and a touch device 3065 .

[0043] <Configuration of Output Device 308 of Second User Terminal 30 > The output device 308 of the second user terminal 30 includes a display 3081 and a speaker 3082 .

[0044] <Configuration of CRM System 50> The CRM system 50 is an information processing device managed and operated by a business providing CRM (Customer Relationship Management) services (CRM business). CRM services include Salesforce, HubSpot, Zoho CRM, and Kintone. The CRM system 50 includes a storage unit 501 and a control unit 504.

[0045] <Configuration of Storage Unit 501 of CRM System 50 > The storage unit 501 of the CRM system 50 includes an application program 5011 and a customer table 5012 .

[0046] The application programs 5011 are programs for causing the control unit 504 of the CRM system 50 to function as each functional unit. The application programs 5011 include applications such as a web browser application.

[0047] The customer table 5012 is a table for storing and managing user information (customer information) related to customers. The customer table 5012 is a table having the customer ID as a primary key and columns for the customer ID, user ID, name, telephone number, and speaker type. FIG. 11 is a diagram showing the data structure of the customer table 5012.

[0048] The customer ID is an item that stores the user identification information of a customer. The user identification information is an item in which a unique value is set for each customer. The user ID is an item that stores the user identification information of the user who manages the customer. The name is an item that stores the customer's name. The phone number is an item that stores the customer's phone number. A user can make a call to the customer's phone number from the first user terminal 20 by accessing a website provided by the CRM system, selecting the customer to whom they want to make a call, and performing a specified operation such as "call." The speaker type is an item that stores the speaker type of the user identified by the customer ID.

[0049] <Configuration of the control unit 504 of the CRM system 50> The control unit 504 of the CRM system 50 includes a user registration control unit 5041. The control unit 504 executes an application program 5011 stored in the storage unit 501, thereby realizing each functional unit.

[0050] The user registration control unit 5041 performs processing to store customer information in the customer table 5012 in the service according to the present disclosure. The information stored in the customer table 5012 is generated when a user opens a web page operated by the service provider from any information processing terminal, enters information into a specified input form, and transmits the information to the CRM system 50. The user registration control unit 5041 stores the received information in a new record in the customer table 5012, completing the customer registration. This allows the customer information to be stored in association with the user ID of the user who manages the customer. The customer ID may be any character string or number that can identify the user, or may be any character string or number desired by the user, or may be automatically set by the user registration control unit 5041.

[0051] <Configuration of Voice Server (PBX) 60> The voice server (PBX) 60 is an information processing device that functions as a switch that connects the network N and the telephone network T to each other, thereby enabling conversation between the first user terminal 20 and the second user terminal 30. The voice server (PBX) 60 includes a storage unit 601.

[0052] <Configuration of Storage Unit 601 of Voice Server (PBX) 60 > The storage unit 601 of the voice server (PBX) 60 includes an application program 6011 .

[0053] The application programs 6011 are programs for causing the control unit 604 of the voice (PBX) server 60 to function as each functional unit. The application programs 6011 include applications such as a web browser application.

[0054] <Operation of System 1> Each process of system 1 will be described below. Fig. 12 is a flowchart showing the operation of incoming call processing. Fig. 13 is a flowchart showing the operation of slide analysis processing. Fig. 14 is a flowchart showing the operation of search processing. Fig. 15 is an example screen showing the operation of search processing. Fig. 16 is a flowchart showing the operation of presentation processing (first embodiment). Fig. 17 is a flowchart showing the operation of presentation processing (second embodiment). Fig. 18 is an example screen showing the operation of presentation processing.

[0055] <Dialogue Processing> The following describes processes that enable a first user and a second user to interact through an incoming call process in which a first user (operator) receives a call from a second user (customer) and an outgoing call process in which the first user (operator) makes an outgoing call to the second user (customer). Other methods for enabling a first user and a second user to interact are not limited to these. For example, a process in which a first user interacts with a second user in a virtual interaction space called a room will be described as room interaction processing. The invention disclosed herein is applicable to incoming call processing, outgoing call processing, and any method for enabling a first user and a second user to interact.

[0056] <Room Interaction Processing> One method involves creating a virtual interaction space called a room on the server 10 for interaction between a first user and a second user. The first and second users access the room via a web browser or application program stored in the first user terminal 20 and the second user terminal 30, respectively, to enable interaction. In this case, the voice server (PBX) 60 is not required. Specifically, the first user, who will be the host of the interaction, operates the input device 206 of the first user terminal 20 to send a request to the server 10 to host the interaction. Upon receiving the request, the control unit 104 of the server 10 issues room identification information such as a unique room ID and transmits a response to the first user terminal 20. The first user then transmits the received room identification information to the second user, with whom the interaction is intended, via any communication means, such as email or chat. The first user can enter the room by operating the input device 206 of the first user terminal 20, using a web browser or the like to access a URL that provides a room-related service on the server 10, and entering the room identification information. Similarly, a second user can enter a room by operating the input device 306 of the second user terminal 30, accessing a URL that provides a room-related service on the server 10 using a web browser or the like, and entering the room identification information. This allows the first user and the second user to converse via the first user terminal 20 and the second user terminal 30, respectively, in a virtual conversation space called a room associated with each other by the room identification information. By entering the room identification information, one or more other users can enter a single room in addition to the first user and the second user. This allows three or more users to converse via their respective user terminals in a virtual conversation space called a room associated with each other by the room identification information.

[0057] Furthermore, the interactive processing does not necessarily have to be performed by having all participants join the room. For example, in a conference held in a conference room or the like and attended by multiple participants, the interactive processing may be performed by having multiple participants enter the room via a single information terminal. Furthermore, the interactive processing does not necessarily have to be performed online, and may be performed using an information terminal that acquires video and audio of the contents of a conference held in a conference room or the like and attended by multiple participants. For example, the interactive processing may be performed in an application that facilitates the conference.

[0058] <Video Interaction> The system 1 according to the present disclosure may provide an online interaction service (video interaction service) that includes video data. For example, the control unit 204 of the first user terminal 20 and the control unit 304 of the second user terminal 30 transmit video data captured by the camera 2061 of the first user terminal 20 and the camera 3061 of the second user terminal 30, respectively, to the server 10. Based on the received video data, the server 10 transmits the video data captured by the camera 2061 of the first user terminal 20 to the second user terminal 30 and the video data captured by the camera 3061 of the second user terminal 30 to the first user terminal 20. The control unit 204 of the first user terminal 20 displays the received video data captured by the camera 3061 of the second user terminal 30 on the display 2081. The control unit 304 of the second user terminal 30 displays the received video data captured by the camera 2061 of the first user terminal 20 on the display 3081. The server 10 may transmit video data of some or all of the multiple users participating in the online dialogue to the first user terminal 20 and the second user terminal 30. In this case, the control unit 204 of the first user terminal 20 displays the received video data of some or all of the multiple users participating in the online dialogue on a single screen on the display 2081 of the first user terminal 20. This allows the dialogue status of the multiple users participating in the online dialogue to be confirmed. A similar process may also be performed in the second user terminal 30.

[0059] In the outgoing call processing and room interaction processing, when an interaction between the user and the customer is initiated, an interaction storage process is executed in the same manner as in the incoming call processing. The interaction storage process is the same as step S104 in the incoming call processing, and therefore will not be described here.

[0060] The room interaction process may be performed by an online conference service or the like operated by a business operator different from the information processing service according to the present disclosure. Examples of online conference services include Zoom, Google Meet, Microsoft Teams, and the like.

[0061] <Incoming call processing> Incoming call processing is processing in which the user receives an incoming call (call) from a customer.

[0062] <Overview of Incoming Call Processing> Incoming call processing is a series of processes that, when a customer calls the user while the user has an application running on the first user terminal 20, identify a call handling rule to be applied to the customer, execute an incoming call determination process based on the identified call handling rule, and execute a process to connect to the user based on the determination result. Note that, in the present disclosure, an incoming call processing by telephone is described as an example, but the present disclosure is also applicable to incoming call processing using any online interactive service, etc.

[0063] <Details of Incoming Call Processing> An incoming call processing of the system 1 when the user receives an incoming call (call) from a customer will be described.

[0064] When the user receives a call from a customer, the following process is performed in the system 1.

[0065] In step S101, the user operates the first user terminal 20 to launch a web browser and access the website of the CRM service provided by the CRM system 50. At this time, the user is assumed to be logged in to the CRM system 50 using his or her own account in the web browser and is on standby. Note that the user only needs to be logged in to the CRM system 50, and may be performing other tasks related to the CRM service.

[0066] In step S102, the customer operates the second user terminal 30, inputs a predetermined telephone number assigned to the voice server (PBX) 60, and makes a call to the voice server (PBX) 60. The voice server (PBX) 60 receives the call from the second user terminal 30 as an incoming call event. The voice server (PBX) 60 transmits the incoming call event to the server 10. Specifically, the voice server (PBX) 60 transmits an incoming call request including the customer's telephone number 3011 to the server 10.

[0067] In step S103, the first user terminal 20 accepts a response operation by the user. The response operation is realized, for example, by lifting the receiver (not shown) of the first user terminal 20, or by the user operating the mouse 2066 to press a button labeled "Answer Call" on the display 2081 of the first user terminal 20. When the first user terminal 20 accepts the response operation, it transmits a response request to the voice server (PBX) 60 via the CRM system 50 and the server 10. The voice server (PBX) 60 receives the transmitted response request and establishes voice communication. This enables the first user terminal 20 to interact with the second user terminal 30. The display 2081 of the first user terminal 20 displays information indicating that an interaction is taking place. For example, the display 2081 of the first user terminal 20 may display the words "Interactive."

[0068] <Dialogue Storage Process> Dialogue storage process is executed in step S104. The dialogue storage process is a process for storing data relating to a dialogue between a user and a customer.

[0069] <Outline of Dialogue Storage Processing> The dialogue storage processing is a series of processes for storing data relating to a dialogue in the dialogue table 1014 when a dialogue is started between a user and a customer.

[0070] <Details of Dialogue Storage Process> In step S104, the control unit 104 of the server 10 executes a dialogue acquisition step of acquiring dialogue data related to the dialogue. Specifically, when a dialogue is initiated between the user and the customer, the voice server (PBX) 60 records voice data related to the dialogue between the user and the customer and transmits it to the server 10. Upon receiving the voice data, the control unit 104 of the server 10 creates a new record in the dialogue table 1014 and stores the voice data in the field for dialogue data related to the dialogue between the user and the customer. Specifically, the control unit 104 of the server 10 stores the user ID, customer ID, dialogue category, call reception / transmission type, and dialogue data content in the new record in the dialogue table 1014. Note that the control unit 104 of the server 10 may include text read aloud obtained by analyzing the voice data in the field for dialogue data in the dialogue table 1014.

[0071] The control unit 104 of the server 10 acquires the first user ID 2011 of the user from the first user terminal 20 during outgoing call processing or incoming call processing, and stores it in the user ID field of a new record in the dialogue table 1014. The control unit 104 of the server 10 queries the CRM system 50 based on the telephone number during outgoing call processing or incoming call processing. The CRM system 50 acquires the customer ID by searching the customer table 5012 by telephone number, and sends it to the server 10. The control unit 104 of the server 10 stores the acquired customer ID in the customer ID field of the new record in the dialogue table 1014. The control unit 104 of the server 10 stores the value of the dialogue category set in advance for each user or customer in the dialogue category field of the new record in the dialogue table 1014. Note that the dialogue category may be stored by the user selecting and inputting a value for each dialogue. The control unit 104 of the server 10 identifies whether the ongoing conversation was initiated by the user or the customer, and stores either the value of outbound (initiated by the user) or inbound (initiated by the customer) in the incoming / outgoing call type field of the new record in the conversation table 1014.

[0072] The control unit 104 of the server 10 stores the voice data received from the voice server (PBX) 60 in the dialogue data item of a new record in the dialogue table 1014. Note that the voice data may be stored as a voice data file in another location, and reference information (path) for the voice data file may be stored after the dialogue ends. The control unit 104 of the server 10 may also be configured to store the voice data after the dialogue ends. Note that the control unit 104 of the server 10 may include text to be read obtained by analyzing the voice data in the dialogue data item of the dialogue table 1014.

[0073] Furthermore, in the video dialogue service, the control unit 104 of the server 10 stores video data received from the first user terminal 20 and the second user terminal 30 in the video data field of a new record in the dialogue table 1014. Note that the video data may be stored as a video data file in another location, and reference information (path) for the video data file may be stored after the dialogue ends. The control unit 104 of the server 10 may also be configured to store the video data after the dialogue ends.

[0074] The control unit 104 of the server 10 executes an audio extraction step of extracting multiple pieces of section audio data for each speech section from the dialogue data acquired in the dialogue acquisition step. The audio extraction step includes a step of identifying a speaker for each piece of section audio data. Specifically, the control unit 104 of the server 10 acquires (accepts) the dialogue ID, dialogue data, and video data stored in the dialogue table 1014. The control unit 104 of the server 10 detects sections (speech sections) in which speech is continuously present from the acquired (accepted) dialogue data and video data, and extracts the dialogue data and video data for each speech section as section audio data and section video data, respectively. For example, the section audio data and section video data may be extracted by dividing the dialogue data and video data by silent sections in which no speech is present. Alternatively, the section audio data and section video data may be extracted by dividing the dialogue data and video data by sentence units, such as segments, sentences, and paragraphs, for the content of speech contained in the dialogue data and video data. The section voice data and section video data are associated with the speaker's user ID, the start date and time of the speech section, and the end date and time of the speech section for each speech section.

[0075] The control unit 104 of the server 10 executes a text generation step of generating a plurality of section reading texts, which are text information of the content spoken by the speaker, for each of the plurality of section audio data extracted in the audio extraction step. Specifically, the control unit 104 of the server 10 performs text recognition on the speech content of the extracted section audio data and section video data, thereby converting the section audio data and section video data into section reading texts, which are characters (text), and transcribing them. Note that the specific method of text recognition is not particularly limited. For example, conversion may be performed using signal processing technology, machine learning using AI (artificial intelligence), deep learning, or the like.

[0076] The control unit 104 of the server 10 stores the dialogue ID to be processed, the speaker's user ID (first user ID 2011 or second user ID 3011), start date and time, end date and time, section audio data, section video data, and section reading text in the dialogue ID, speaker ID, start date and time, end date and time, section audio data, section video data, and section reading text fields of a new record in the audio section table 1016, respectively.

[0077] The section reading text for each utterance section of the dialogue data is stored as continuous time-series data in association with the start date and time and the speaker in the voice section table 1016. By checking the section reading text stored in the voice section table 1016, the user can check the dialogue content as text information without checking the content of the dialogue data.

[0078] In addition, during the text recognition process, information that is meaningless in understanding the conversation between the user and the customer, such as fillers contained in the text, may be excluded from the text in advance, and the speech recognition information may be stored in the speech segment table 1016.

[0079] <Call Processing> The call processing is processing in which a user (first user) calls (places a call) to a customer (second user).

[0080] <Outline of Call Processing> The call processing is a series of processes in which the user selects a customer to whom they wish to make a call from among multiple customers displayed on the screen of the first user terminal 20 and performs a call operation to make a call to the customer. In the present disclosure, a case in which a second user is selected as a customer will be described as an example.

[0081] <Details of Call Processing> The call processing of the system 1 when a user calls a customer will be described.

[0082] When a user makes a call to a customer, the following process is executed in the system 1.

[0083] The user operates the first user terminal 20 to launch a web browser and access the website of the CRM service provided by the CRM system 50. The user can display a list of their own customers on the display 2081 of the first user terminal 20 by opening a customer management screen provided by the CRM service. Specifically, the first user terminal 20 sends the CRM ID 2013 and a request to display a list of customers to the CRM system 50. Upon receiving the request, the CRM system 50 searches the customer table 5012 and sends information about the user's customers, such as the customer ID, name, telephone number, customer attributes, customer organization name, and customer organization attributes, to the first user terminal 20. The first user terminal 20 displays the received information about the customers on the display 2081 of the first user terminal 20.

[0084] The user presses and selects a customer (second user) to whom they wish to make a call from the list of customers displayed on the display 2081 of the first user terminal 20. With the customer selected, the user presses the "Call" button or the phone number button displayed on the display 2081 of the first user terminal 20 to send a request including the phone number to the CRM system 50. The CRM system 50, which receives the request, sends the request including the phone number to the server 10. The server 10, which receives the request, sends a call request to the voice server (PBX) 60. When the voice server (PBX) 60 receives the call request, it makes a call (call) to the second user terminal 30 based on the received phone number.

[0085] In response to this, the first user terminal 20 controls the speaker 2082 or the like to make a sound indicating that an outgoing call (call) is being made by the voice server (PBX) 60. The display 2081 of the first user terminal 20 also displays information indicating that an outgoing call (call) is being made to the customer by the voice server (PBX) 60. For example, the display 2081 of the first user terminal 20 may display the words "Calling."

[0086] The customer lifts the receiver (not shown) of the second user terminal 30 or presses an "answer" button or the like that is displayed on the input device 306 of the second user terminal 30 when a call arrives, thereby placing the second user terminal 30 in a conversation-enabled state. Accordingly, the voice server (PBX) 60 transmits information indicating that the second user terminal 30 has responded (hereinafter referred to as a "response event") to the first user terminal 20 via the server 10, the CRM system 50, or the like. This allows the user and the customer to engage in a conversation using the first user terminal 20 and the second user terminal 30, respectively, and allows a conversation between the user and the customer. Specifically, the user's voice collected by the microphone 2062 of the first user terminal 20 is output from the speaker 3082 of the second user terminal 30. Similarly, the customer's voice collected by the microphone 3062 of the second user terminal 30 is output from the speaker 2082 of the first user terminal 20.

[0087] When the display 2081 of the first user terminal 20 becomes ready for interaction, the display 2081 receives the response event and displays information indicating that an interaction is taking place. For example, the display 2081 of the first user terminal 20 may display the words "Responding."

[0088] The object analysis process is a process for analyzing video data related to a dialogue performed in a dialogue service according to the present disclosure. In the present disclosure, an analysis process is described for video data including objects such as facial expressions and emotions of meeting participants, other arbitrary objects, and presentation slides for sales, etc.

[0089] <Overview of Object Analysis Processing> Object analysis processing is a series of processes that acquire video data related to a dialogue between multiple users, identify the objects contained in the video data, identify the section of the video in which the objects are contained, and store the objects in association with dialogue data related to the objects.

[0090] <Details of Object Analysis Processing> Details of the object analysis processing will be described below.

[0091] In step S301, the analysis unit 1042 of the server 10 executes a video acquisition step of acquiring video data related to the dialogue. Specifically, the analysis unit 1042 of the server 10 acquires the video data by referring to the dialogue table 1014. In the present disclosure, a case where the object analysis process is executed as a process separate from the incoming call process is disclosed as an example, but is not limited to this. For example, in parallel with the start of the dialogue storage process for the incoming call process, the analysis unit 1042 of the server 10 may acquire video data related to the incoming call process and execute the object analysis process in parallel with the incoming call process.

[0092] In step S301, the video acquisition step executes a step of acquiring a plurality of different video data. Specifically, the analysis unit 1042 of the server 10 may refer to the dialogue table 1014 and acquire a plurality of different video data from a plurality of dialogue information records.

[0093] In step S302, the analysis unit 1042 of the server 10 executes an object identification step in which, based on the video data acquired in the video acquisition step, the analysis unit 1042 of the server 10 identifies object information regarding objects included in the video data. Specifically, the analysis unit 1042 of the server 10 analyzes the video data acquired in step S301 to identify the positions, areas, types, presentation timing, etc. of objects related to people, objects, and slides included in the video data. Details of the objects are described in the object table 1021. For example, the analysis unit 1042 of the server 10 applies frame images (still images) of the acquired video data at predetermined intervals to an object detection model (not shown) stored in the storage unit 101 of the server 10 to detect people, body parts of people, objects, and other objects. If the video includes presentation slides, the analysis unit 1042 of the server 10 detects the objects and identifies label information related to the objects. For example, the label information includes the class of the detected object (person, name of a person's body part such as a face, name of an object such as a chair or smartphone, information on a presentation slide, page, etc.), as well as the person's name, the object's model number, text, images, photos, etc. contained in the slide.

[0094] In step S302, the object identification step includes a page section identification step of identifying multiple pieces of object information related to multiple page sections for each page of a presentation slide included in the video data. The object identification step includes a string extraction step of extracting strings included in the page sections of the presentation slide for each of the multiple page sections identified in the page section identification step. The string extraction step executes a step of recognizing and extracting strings included in the image data by analyzing image data of the page of the presentation slide. Specifically, the analysis unit 1042 of the server 10 determines whether presentation slides are included in frame images (still images) for each predetermined period of the acquired video data. This determination may be made using any machine learning model, artificial intelligence model, or deep learning model. If the analysis unit 1042 of the server 10 determines that presentation slides are included in the acquired video data in step S301, it identifies each page of the presentation slide as object information. Furthermore, the analysis unit 1042 of the server 10 applies OCR (Optical Character Recognition) to the image data of each page, thereby including the characters, images, photographs, etc. contained in each page in the object information.

[0095] In step S302, the character string extraction step refers to a slide file previously linked to a page of the presentation slide, and analyzes the slide file to recognize and extract the position, type, character string, etc. of an object contained in the slide file. The analysis unit 1042 of the server 10 may identify any file created by a presentation slide, word processing software, spreadsheet software, or the like, contained in the video data acquired in step S301 based on the video data. For example, the analysis unit 1042 of the server 10 may identify a file by referring to the location (path) of the file contained in the video data acquired in step S301. The analysis unit 1042 of the server 10 may also identify a file by searching a specified shared drive, etc., based on image data contained in the video data. When using presentation slides, etc., prepared in advance for business negotiations, the analysis unit 1042 of the server 10 analyzes the identified file based on the path to extract and identify text, images, photos, etc. contained in the file as object information. In this way, the analysis unit 1042 of the server 10 may identify object information such as text, images, and photographs based not only on the video data but also on information stored in the original file that was presented to the participants in the video data. This allows the analysis unit 1042 of the server 10 to more accurately extract text included in slide pages and documents. It also allows for reference to clearer image, photograph, and other object information. When identifying object information based on information stored in a file, meta information such as the slide title and slide number of the presentation slide may also be included.

[0096] In step S302, the analysis unit 1042 of the server 10 executes a comparison step for each piece of video data. The comparison step compares the character string corresponding to the page of the presentation slide in the specified page section received in the character string extraction step with the section reading text, which is text information about what was spoken by the speaker in the specified page section. The comparison step generates information indicating the degree of agreement between the character string and the section reading text for each piece of video data. Specifically, the analysis unit 1042 of the server 10 analyzes the dialogue data for each page of the presentation slide during the period in which the page was presented to the participants and the dialogue data to obtain the reading text. The reading text may be obtained by referring to the section reading text in the speech section table 1016. The analysis unit 1042 of the server 10 obtains a talk script stored in advance in association with each page of the presentation slide. A talk script is a prepared conversation guideline or framework used by sales representatives when interacting with customers via telephone, online chat, or the like. Talk scripts are used by sales representatives to convey consistent messages and conduct effective and efficient dialogue with customers. They include explanations of questions, needs, features and benefits, objection responses, closing statements, and other legally required explanations. In the present disclosure, talk scripts are assigned in advance to each page of presentation slides. Typically, when a sales representative gives a presentation to a customer, the presentation is made according to the talk script for each page of the presentation slides. The analysis unit 1042 of the server 10 compares the acquired spoken text with the talk script and calculates the degree of match. The degree of match may be calculated based on the degree of sentence match, which evaluates the degree of match between the grammar and wording of the spoken text and the talk script. For example, the analysis unit 1042 of the server 10 calculates the degree of match by calculating the number of exact matching sentences or the word match rate. The degree of match may also be calculated based on the degree of content match, which evaluates the degree of match between the spoken text and the talk script.For example, the degree of agreement may be calculated by evaluating whether the purpose of the talk script and the value provided are reflected in the spoken text. Alternatively, the degree of agreement may include an index indicating whether the dialogue with the customer is proceeding in accordance with the talk script. The analysis unit 1042 of the server 10 includes the calculated degree of agreement in the object data item of the corresponding presentation slide page. This allows the analysis unit 1042 of the server 10 to store, for each analyzed page of presentation slides, a degree of agreement indicating to what extent the sales representative who gave the presentation followed the talk script.

[0097] The analysis unit 1042 of the server 10 detects the date and time (start date and time) when presentation of one or more pieces of object information identified in step S302 to participants began and the date and time (end date and time) when presentation ended. Specifically, the analysis unit 1042 of the server 10 detects the date and time when an object is detected by an object detection model or the like as the start date and time, and the date and time when the object is no longer detected as the end date and time. The analysis unit 1042 of the server 10 associates the start date and time and the end date and time with each of the pieces of object information identified in step S302 and stores them.

[0098] In step S303, the analysis unit 1042 of the server 10 executes a speaker identification step of identifying a speaker who is speaking in the dialogue data stored in association with the object information stored in the object storage step. Specifically, the analysis unit 1042 of the server 10 analyzes the video data or dialogue data for each piece of object information over a period (a predetermined period) from the start date and time to the end date and time, thereby identifying a speaker (user) who is speaking over the predetermined period. For example, the analysis unit 1042 of the server 10 detects video data in which speech is being made over the predetermined period from multiple video data stored for multiple users participating in the dialogue, identifies the participating user based on an identifier assigned to the video data, and identifies the user as the speaker. The analysis unit 1042 of the server 10 may also store each user in the user table 1012 in association with voiceprint data, etc., and identify the speaker by comparing the voice detected over the predetermined period with the voiceprint data, etc. Alternatively, the analysis unit 1042 of the server 10 may receive an input operation from the user to identify a speaker, and identify the speaker in accordance with the input operation. If multiple speakers are identified, the analysis unit 1042 of the server 10 may identify the multiple speakers as speakers related to the object information. Furthermore, the analysis unit 1042 of the server 10 may identify the speaker with the longest speaking period or the speaker with the highest importance calculated based on an arbitrary importance calculation algorithm, etc., as the speaker related to the object information.

[0099] In step S303, the speaker identification step executes a step of identifying a speaker speaking in the dialogue data stored in association with the object information, based on the speaker extracted in the voice extraction step. Specifically, the analysis unit 1042 of the server 10 identifies a speaker related to the object information based on the voice segment information stored in the voice segment table 1016. Based on a predetermined period from the start date / time to the end date / time of the object information identified in step S302, the analysis unit 1042 of the server 10 searches the voice segment table 1016 for voice segment information including the start date / time and the end date / time, and acquires a section ID and a speaker ID. This allows the analysis unit 1042 of the server 10 to identify one or more pieces of voice segment information that were spoken throughout the presentation period of the identified multiple pieces of object information. The analysis unit 1042 of the server 10 identifies the speaker ID included in the voice segment information as the speaker of the identified object information. The analysis unit 1042 of the server 10 may identify multiple speaker IDs as speakers. In addition, the analysis unit 1042 of the server 10 may identify the speaker with the longest speaking period, the speaker with the longest voice interval, or the speaker with the highest importance calculated based on an arbitrary importance calculation algorithm, etc., as the speaker related to the object information.

[0100] In step S304, the analysis unit 1042 of the server 10 executes an object storage step of associating and storing the object information identified in the object identification step with the dialogue data acquired in the dialogue acquisition step related to the object information. The object storage step executes a step of associating and storing the object information identified in the object identification step with the plurality of section voice data extracted in the voice extraction step. Specifically, the analysis unit 1042 of the server 10 stores the dialogue ID of the dialogue information acquired in step S301, the object information identified in step S302, the start date and time, end date and time, section ID, and speaker user ID identified in step S303 in the dialogue ID, object data, start date and time, end date and time, section ID, and speaker ID fields in the object table 1021. As a result, the object data is stored in association with the voice section information in the voice section table 1016 by the section ID. The analysis unit 1042 of the server 10 may refer to the dialogue data and video data in the dialogue table 1014 and store the dialogue data from the start date and time to the end date and time of the object information in an item for storing audio information (not shown) in the object table 1021. In this way, the object information may be stored in association with the dialogue data related to the object information. It is not necessarily necessary to store the dialogue data in the object table 1021. Based on the start date and time and the end date and time stored in the object table 1021, the dialogue data may be stored in association with the object by referring to the dialogue data and video data in the dialogue table 1014 included in the period.

[0101] In step S304, an object storage step executes a step of associating the object information identified in the object identification step with the multiple section reading texts generated in the text generation step and storing them. Specifically, the analysis unit 1042 of the server 10 associates section identification information with object information in the object table 1021 and stores the object information, thereby associating each piece of object information with a section reading text in the speech section table 1016. This allows the user to search for object information and to check at a glance as a search result the reading text corresponding to the dialogue data stored in association with the object information.

[0102] In step S304, the object storage step includes a string storage step of associating and storing the strings extracted in the string extraction step for each of the page sections identified in the page section identification step. Specifically, when the analysis unit 1042 of the server 10 identifies each page of the presentation slide as object information in step S302, the analysis unit 1042 stores object information, such as text, images, and photos, contained on each page of the presentation slide, in the object data field of the object information record for each page. In other words, when a video includes presentation slides, the analysis unit 1042 of the server 10 stores each page of the presentation slide as object information in each record of the object table 1021. In this case, the object data field of each object information record includes text, images, photos, and the like contained on each page of the presentation slide. This allows the analysis unit 1042 to store the string information for each slide associated with the slides included in the video, as to what strings are contained on each page of the slides in the video.

[0103] <Search Process> The search process is a process for searching for voice information stored in the incoming call process and object information stored in the object analysis process.

[0104] <Overview of Search Processing> The search processing is a series of steps that accepts search conditions, including keywords, from the user, searches for object information based on the search conditions, searches for the spoken text based on the search conditions, obtains the search results, and presents the search results.

[0105] <Details of Search Processing> Details of the search processing will be described below.

[0106] In step S501, the search unit 1043 of the server 10 executes a search input step of accepting search criteria input from a user. Specifically, the first user operates the input device 206 of the first user terminal 20 to execute a browser application or the like, and opens the search page D1 by inputting the URL or the like of a web page (search page) for executing a search process. The control unit 204 of the first user terminal 20 transmits a request including the first user ID 2011 for opening the search page to the server 10.

[0107] Upon receiving the request, the server 10 generates a search page D1 and transmits it to the first user terminal 20. The control unit 204 of the first user terminal 20 displays and presents the search page D1 on the display 2081 of the first user terminal 20. FIG. 15 shows an example of a search screen in the search process. The search page D1 is displayed on the display 2081 of the first user terminal 20. The search page D1 includes an input field D101 for search conditions, and search results D111, D112, D113, etc. The search results D111, D112, and D113 include readout text D1111, D1121, and D1131. The search results D111 and D113 include object information D1112 and D1132. The search results D111, D112, and D113 include speakers D1113, D1123, and D1133.

[0108] The first user operates the input device 206 of the first user terminal 20 to input a character string (text) related to the search criteria into the input field D101. For example, the first user operates the input device 206 of the first user terminal 20 to input the character string "man-hour reduction" into the input field D101. The control unit 204 of the first user terminal 20 transmits the character string (search criteria) input into the input field D101 to the server 10. Note that the search criteria may also include a slide number or a slide title.

[0109] In step S502, the search unit 1043 of the server 10 executes an information search step of searching for object information stored in the object storage step based on the search criteria received in the search input step. The information search step executes a step of searching for object information containing information matching the search criteria based on the search criteria. Specifically, the search unit 1043 of the server 10 searches for object data items in the object table 1021 based on the string of search criteria received from the first user terminal 20. Specifically, the search unit 1043 of the server 10 identifies records of object information whose object data contains the string of search criteria. Note that the string of search criteria does not necessarily need to be included in the object data; a match can also be found when a record of object information is identified based on known matching criteria such as partial match, fuzzy search, concept search, or synonym search. Note that the search unit 1043 of the server 10 may also identify a type of object data (person, object, slide) as a search result based on the string of search criteria. The search unit 1043 of the server 10 may search for object data according to a character string related to a search condition relating to attribute information about a person, such as the person's name, age, sex, address, occupation, contact information, hobbies, educational history, family relationships, etc. The search unit 1043 of the server 10 may search for object data according to a character string related to information about the name, type, or shape of an object. The search unit 1043 of the server 10 may search for object data based on a character string, image, photograph, etc. included in a slide, document, etc.

[0110] <Speech Text Search (First Embodiment)> In step S503, the search unit 1043 of the server 10 searches for and identifies one or more pieces of speech segment information by searching for section IDs in the speech segment table 1016 based on the section IDs included in the object information searched for in step S502. This makes it possible to identify one or more pieces of speech segment information associated with each of the one or more pieces of object information searched for in step S502.

[0111] <Text Reproduction Search (Second Embodiment)> In step S503, the search unit 1043 of the server 10 executes an information search step of searching for the section reproduction text generated in the text generation step, based on the search conditions received in the search input step. The information search step executes a step of searching for speech segment information in which the section reproduction text contains information matching the search conditions, based on the search conditions. Specifically, the search unit 1043 of the server 10 searches for an item of the section reproduction text in the speech segment table 1016, based on the character string of the search conditions received from the first user terminal 20. Specifically, the search unit 1043 of the server 10 identifies a record of speech segment information in which the character string of the section reproduction text contains a character string matching the search conditions. Note that the character string matching the search conditions does not necessarily need to be included in the section reproduction text; a case in which a record of object information is identified according to known matching conditions, such as partial match, fuzzy search, concept search, or synonym search, also constitutes a case in which the search conditions are matched.

[0112] In step S504, the search unit 1043 of the server 10 transmits one or more object information records searched for in step S502 and one or more voice section information records searched for in step S503 to the first user terminal 20. The control unit 204 of the first user terminal 20 receives and acquires the one or more object information and the one or more voice section information as search results for the search conditions input in step S501.

[0113] <Search Result Presentation Process (First Embodiment)> In step S505, the control unit 204 of the first user terminal 20 executes an information presentation step of presenting, to the user, the dialogue data stored in association with the object information searched in the information search step as a search result. The information presentation step executes a step of presenting, to the user, a read-aloud text, which is text information of what the speaker has said in the dialogue data, as a search result. The information presentation step includes a step of presenting a search result including the object information searched in the information search step. Specifically, the control unit 204 of the first user terminal 20 displays each of the received one or more pieces of speech segment information in search results D111, D112, and D113 on the display 2081 of the first user terminal 20. The search results D111, D112, and D113 include read-aloud text D1111, D1121, and D1131 included in the speech segment information. If there is associated object information found in step S502, the search results D111 and D113 include object information D1112 and D1132. This allows the user to search for object information and also to check the audio data stored in association with the object information as the search result.

[0114] In step S505, the information presenting step executes a step of presenting dialogue data in association with speakers who are uttering the voices included in the dialogue data. Specifically, the control unit 204 of the first user terminal 20 displays user information identified based on speaker IDs included in the received one or more pieces of voice segment information on speakers D1113, D1123, and D1133 on the display 2081 of the first user terminal 20. Note that speakers D1113, D1123, and D1133 may be user information identified based on speaker IDs included in object information, rather than speaker IDs included in the voice segment information.

[0115] <Search Result Presentation Process (Second Embodiment)> In step S505, the control unit 204 of the first user terminal 20 may display each of the received one or more pieces of object information on the display 2081 of the first user terminal 20. In this case, the control unit 204 of the first user terminal 20 may display each of the one or more pieces of object information in association with section reading text of one or more pieces of speech section information identified based on the section ID included in the object information. Furthermore, the control unit 204 of the first user terminal 20 may display each of the one or more pieces of object information in association with user information identified based on the speaker ID included in the object information.

[0116] <Presentation process (first embodiment)> The presentation process (first embodiment) is a process of searching for and presenting other dialogue information that includes object information equivalent to the object information selected by the user, based on the analysis results obtained by executing the object analysis process.

[0117] <Overview of the presentation process (first embodiment)> The presentation process (first embodiment) is a series of processes that accepts input of dialogue identification information from a user, obtains one or more object information corresponding to the dialogue identification information and presents it to the user, accepts selection of object information from the user, searches for other dialogue information that includes object information equivalent to the selected object information according to the selected object information, and presents the dialogue information related to the search results to the user.

[0118] <Details of Presentation Processing (First Embodiment)> Details of the presentation processing (first embodiment) will be described below.

[0119] In step S701, the presentation unit 1044 of the server 10 executes an identification information input step of accepting input of dialogue identification information from a user. Specifically, the first user operates the input device 206 of the first user terminal 20 to execute a browser application or the like, and opens the presentation page D3 by inputting the URL or the like of a web page (presentation page) for executing the presentation process. The control unit 204 of the first user terminal 20 transmits a request including the first user ID 2011 for opening the presentation page to the server 10.

[0120] Upon receiving the request, the server 10 generates a presentation page D3 and transmits it to the first user terminal 20. The control unit 204 of the first user terminal 20 displays and presents the presentation page D3 on the display 2081 of the first user terminal 20. FIG. 18 shows an example of a presentation screen in the presentation process. The presentation page D3 (presentation screen) is displayed on the display 2081 of the first user terminal 20. The presentation page D3 includes an input field D301 for dialogue identification information, object information D311, D312, D313, etc. included in dialogue information identified according to the selected dialogue identification information, and dialogue information D321, D322, D323 related to the search results. The dialogue information D321, D322, D323 includes video data D3211, D3221, D3231. The dialogue information D321, D322, D323 includes playback times D3212, D3222, D3232. The dialogue information D321, D322, and D323 includes summaries D3213, D3223, and D3233. The dialogue information D321, D322, and D323 includes degrees of agreement D3214, D3224, and D3234 with the talk script.

[0121] The first user operates the input device 206 of the first user terminal 20 to input dialogue identification information for identifying dialogue information into the input field D301. Note that the input field D301 may be configured to display, as input candidates, multiple dialogue IDs identified by searching dialogue information previously stored in the dialogue table 1014, and the user may select a specific dialogue ID from the input candidates of the multiple dialogue IDs by operating the input device 206 of the first user terminal 20. The control unit 204 of the first user terminal 20 transmits the character string (dialogue identification information) input into the input field D301 to the server 10.

[0122] In step S702, the presentation unit 1044 of the server 10 searches for a dialogue ID in the object table 1021 and acquires one or more pieces of object information based on the dialogue identification information received from the first user terminal 20. As a result, the presentation unit 1044 of the server 10 acquires one or more pieces of object information included in the dialogue information selected in step S701.

[0123] The presentation unit 1044 of the server 10 transmits the acquired one or more pieces of object information to the first user terminal 20. The control unit 204 of the first user terminal 20 receives and acquires the one or more pieces of object information.

[0124] <Object Selection (First Embodiment)> The control unit 204 of the first user terminal 20 displays the acquired one or more pieces of object information D311, D312, and D313 in a selectable manner on the display 2081 of the first user terminal 20. The first user operates the input device 206 of the first user terminal 20 to select predetermined object information (object information D312, as an example) from the one or more pieces of object information D311, D312, and D313.

[0125] <Object Selection (Second Embodiment)> In step S702, the presentation unit 1044 of the server 10 executes a section receiving step in which input of a specific page section from among the multiple page sections identified in the page section identifying step is received. As an example, a case will be described in which the object information included in the dialogue information is each page of a presentation slide. In this case, each page of the presentation slide is displayed in the object information D311, D312, and D313 of the presented page D3. Each page is associated with a start date and time and an end date and time in the object table 1021, and each of the multiple pieces of object information D311, D312, and D313 corresponds to a page of the presentation slide. The first user operates the input device 206 of the first user terminal 20 to select a specific page (a specific page section in the object table 1021) from among the pages of the presentation slide displayed on the display 2081 of the first user terminal 20.

[0126] The control unit 204 of the first user terminal 20 transmits the object ID of the object information D312 selected by the first user to the server 10.

[0127] <Dialogue Information Search (First Embodiment)> In step S703, the presentation unit 1044 of the server 10 searches the object ID field in the object table 1021 based on the object ID received from the first user terminal 20 to acquire object data. Specifically, the presentation unit 1044 of the server 10 searches the object data field in the object table 1021 based on the acquired object data, and searches for and identifies one or more records of object information that have object data equivalent to the object data selected by the first user. Note that the presentation unit 1044 of the server 10 may identify the object information excluding the object information related to the dialogue identification information input in step S701. In other words, the object information may be searched for by excluding the dialogue information related to the dialogue identification information input in step S701. Note that "equivalent" of object data means that the object data is substantially equivalent, and this also includes cases where the object data is equivalent in terms of similar concepts such as spelling variations, expression variations, synonyms, and the like. The presentation unit 1044 of the server 10 searches the dialogue ID item in the dialogue table 1014 based on one or more dialogue IDs included in the identified one or more pieces of object information, and acquires one or more pieces of dialogue information. The presentation unit 1044 of the server 10 may identify dialogue information by excluding dialogue information having duplicate dialogue IDs. The presentation unit 1044 of the server 10 may identify dialogue information by excluding object information by a speaker identified based on a speaker ID included in the object information selected in step S702. In other words, the presentation unit 1044 of the server 10 may be configured to identify object information in a dialogue conducted by a speaker different from the selected object information. This makes it possible to identify other dialogue information that includes object data equivalent to the object data related to the object information selected in step S702.

[0128] <Dialogue Information Search (Second Embodiment)> In step S703, the presentation unit 1044 of the server 10 executes a video identification step in which it identifies multiple different video data associated with the same page of a presentation slide in the specified page section received in the section reception step. The phrase "identical" refers to a presentation slide being substantially identical. This includes cases where the presentation slides are identical in terms of similar concepts, such as variations in spelling, expression, synonyms, and similar terms. For example, presentation slides related to a specific sales document may be considered identical if they are recognized as being substantially the same page, even if there are slight differences in expression, such as different versions. As an example, a case will be described in which the object information included in the dialogue information is each page of a presentation slide. In this case, the presentation unit 1044 of the server 10 identifies other dialogue information including the page of the presentation slide selected in step S702 and the same page of the presentation slide. The dialogue information includes dialogue data and video data related to the dialogue. This allows the contents of the conference using each slide to be viewed at a glance. For example, if the content of a meeting is a business negotiation, videos of business negotiations by different sales representatives using the same presentation slides can be viewed and understood. Specifically, multiple business negotiation videos can be compared and checked.

[0129] In step S704, the control unit 204 of the first user terminal 20 executes a video presentation step of presenting the predetermined video data identified in the video identification step. Specifically, the presentation unit 1044 of the server 10 searches the dialogue ID item in the dialogue table 1014 based on the dialogue ID included in the dialogue information identified by the search in step S703, and acquires one or more pieces of dialogue information including video data. Note that the presentation unit 1044 of the server 10 may acquire thumbnail information indicating the video data instead of the video data. The presentation unit 1044 of the server 10 transmits the acquired one or more pieces of dialogue information to the first user terminal 20. The control unit 204 of the first user terminal 20 displays the video data D3211, D3221, and D3231 included in the received one or more pieces of dialogue information D321, D322, and D323, respectively, on the display 2081 of the first user terminal 20. The first user can select video data D3211, D3221, or D3231 by operating the input device 206 of the first user terminal 20, thereby playing back video data related to the dialogue information D321, D322, or D323. Note that video data D3211, D3221, or D3231 may be configured to play back only the portions of each video data related to the object information searched and identified in step S702. This allows, when the object information is each page of a presentation slide, easy confirmation of presentation portions of a video of a sales negotiation conducted by a specific sales representative that correspond to the page of the presentation slide selected in step S702. This allows a collective confirmation of the state of business negotiations conducted by multiple sales representatives on a specific page of a presentation slide.

[0130] In step S704, the video presenting step executes a step of associating, for each predetermined video data, information indicating the display period of the presentation slide pages in the predetermined page section received in the section receiving step, and presenting the associated information. Specifically, the presentation unit 1044 of the server 10 acquires the period from the start date and time to the end date and time included in the object information searched in step S703. That is, for business negotiation videos by different sales representatives using the same presentation slide, information indicating the display period of the presentation slide is acquired. The presentation unit 1044 of the server 10 transmits the acquired information indicating the display period of the presentation slide to the first user terminal 20. The control unit 204 of the first user terminal 20 displays information indicating the display period of the presentation slide (playback time D3212, D3222, D3232) for each of the acquired dialogue information D321, D322, D323 on the display 2081 of the first user terminal 20. Alternatively, the control unit 204 of the first user terminal 20 may be configured to display an indicator indicating the importance of the presentation slide in each of the acquired dialogue information D321, D322, and D323 on the display 2081 of the first user terminal 20. For example, a five-level indicator indicating the importance may be displayed on the display 2081 of the first user terminal 20. This makes it possible to correlate and understand the display times of slides in sales negotiation videos by different sales representatives who used the same presentation slides. It is possible to check how long each sales representative spent negotiating in each sales negotiation video.

[0131] In step S704, the presentation unit 1044 of the server 10 searches the speech segment table 1016 for a dialogue ID based on the dialogue ID included in the dialogue information searched for in step S703, and acquires one or more pieces of speech segment information. The presentation unit 1044 of the server 10 generates summary information for each dialogue information based on one or more pieces of speech segment information for each dialogue ID. For example, the presentation unit 1044 of the server 10 may extract a small number of speech segment information with high importance from the multiple pieces of speech segment information for each dialogue information, and use the extracted speech segment information as summary information. Alternatively, a summary document may be generated by applying a large-scale language model (LLM) to the section reading text included in the extracted speech segment information. The presentation unit 1044 of the server 10 associates the dialogue ID with the summary information and transmits them to the first user terminal 20. The control unit 204 of the first user terminal 20 displays the received summary document (summaries D3213, D3223, and D3233) on the display 2081 of the first user terminal 20.

[0132] In step S704, the video presentation step executes a step of associating and presenting, for each predetermined video data, information indicating a comparison result between a character string corresponding to a page of a presentation slide in a predetermined page section received in the character string extraction step and a section readout text, which is text information on what was spoken by the speaker in the predetermined page section. The video presentation step executes a step of associating and presenting, for each predetermined video data, information indicating a degree of match between the character string and the section readout text. As an example, a case will be described in which object information included in the dialogue information is each page of a presentation slide. In this case, the presentation unit 1044 of the server 10 acquires information indicating a degree of match with the talk script included in the object information searched in step S703. Note that, although an example in which the information indicating the degree of match with the talk script is executed in the object analysis process has been disclosed in the present disclosure, it may also be configured to be executed in step S704 of the presentation process (first embodiment). The presentation unit 1044 of the server 10 transmits information indicating the degree of match with the acquired talk script to the first user terminal 20. The control unit 204 of the first user terminal 20 displays information indicating the degree of match with the received talk script (degrees of match with the talk script D3214, D3224, D3234) on the display 2081 of the first user terminal 20.

[0133] <Presentation process (second embodiment)> The presentation process (second embodiment) is a process of searching for and presenting other dialogue information that includes the same speaker as the speaker selected by the user, based on the analysis results obtained by executing the object analysis process.

[0134] <Overview of the presentation process (second embodiment)> The presentation process (second embodiment) is a series of processes that accepts input of dialogue identification information from a user, obtains one or more speakers according to the dialogue identification information and presents them to the user, accepts speaker selection from the user, searches for other dialogue information that includes the same speaker as the selected speaker according to the selected speaker, and presents the dialogue information related to the search results to the user.

[0135] <Details of Presentation Process (Second Embodiment)> Details of the presentation process (second embodiment) will be described below.

[0136] Steps S801 and S804 are the same as those in the presentation process (first embodiment), and therefore a description thereof will be omitted.

[0137] <Speaker Selection> In step S802, the presentation unit 1044 of the server 10 executes a speaker receiving step of receiving speaker input. Specifically, the presentation unit 1044 of the server 10 searches for a dialogue ID in the object table 1021 based on the dialogue identification information received from the first user terminal 20, and acquires one or more speaker IDs. As a result, the presentation unit 1044 of the server 10 acquires one or more speaker IDs included in the dialogue information selected in step S701. The presentation unit 1044 of the server 10 searches for a user ID in the user table 1012 based on the acquired speaker ID, and acquires one or more pieces of user information. The presentation unit 1044 of the server 10 transmits the acquired one or more pieces of user information to the first user terminal 20. The control unit 204 of the first user terminal 20 displays the received one or more pieces of user information on the display 2081 of the first user terminal 20 in a selectable manner. The first user selects a predetermined speaker from one or more speakers by operating the input device 206 of the first user terminal 20. The control unit 204 of the first user terminal 20 transmits the speaker ID of the speaker selected by the first user to the server 10.

[0138] In step S803, the video identification step executes a step of identifying multiple different predetermined video data associated with the speaker based on the speaker accepted in the speaker acceptance step. Specifically, the presentation unit 1044 of the server 10 searches the speaker ID field in the object table 1021 based on the acquired speaker ID, and searches for and identifies one or more records of object information having object data associated with the speaker selected by the first user. The presentation unit 1044 of the server 10 may identify the object information by excluding object information related to the dialogue identification information input in step S801. The presentation unit 1044 of the server 10 may search the dialogue ID field in the dialogue table 1014 based on one or more dialogue IDs included in the identified one or more pieces of object information, and acquire one or more pieces of dialogue information. The presentation unit 1044 of the server 10 may identify the dialogue information by excluding dialogue information with duplicate dialogue IDs. This allows the contents of the meeting in which each speaker spoke to be viewed in a list. For example, if the meeting content is a business negotiation, videos of the business negotiation conducted by a specific sales representative can be easily viewed.

[0139] 19 is a block diagram showing the basic hardware configuration of a computer 90. The computer 90 includes at least a processor 901, a main storage device 902, an auxiliary storage device 903, and a communication IF 991 (interface), which are electrically connected to one another by a communication bus 921.

[0140] The processor 901 is hardware for executing an instruction set written in a program, and is composed of an arithmetic unit, a register, a peripheral circuit, and the like.

[0141] The main storage device 902 is used to temporarily store programs and data to be processed by the programs, etc. For example, it is a volatile memory such as a DRAM (Dynamic Random Access Memory).

[0142] The auxiliary storage device 903 is a storage device for saving data and programs, such as a flash memory, a hard disk drive (HDD), a magneto-optical disk, a CD-ROM, a DVD-ROM, or a semiconductor memory.

[0143] The communication IF 991 is an interface for inputting and outputting signals for communicating with other computers via a network using a wired or wireless communication standard. The network is composed of the Internet, a LAN, various mobile communication systems constructed using wireless base stations, etc. For example, the network includes 3G, 4G, and 5G mobile communication systems, LTE (Long Term Evolution), and wireless networks (e.g., Wi-Fi (registered trademark)) that can connect to the Internet via a predetermined access point. In the case of a wireless connection, communication protocols include, for example, Z-Wave (registered trademark), ZigBee (registered trademark), and Bluetooth (registered trademark). In the case of a wired connection, the network also includes a network that is directly connected using a USB (Universal Serial Bus) cable, etc.

[0144] It should be noted that the computer 90 can be virtually realized by distributing all or part of each hardware configuration across multiple computers 90 and interconnecting them via a network. In this way, the concept of the computer 90 includes not only a computer 90 housed in a single housing or case, but also a virtualized computer system.

[0145] <Basic Functional Configuration of Computer 90> A description will be given of the functional configuration of the computer realized by the basic hardware configuration (FIG. 19) of the computer 90. The computer includes at least the functional units of a control unit, a storage unit, and a communication unit.

[0146] The functional units of the computer 90 can also be realized by distributing all or part of the functional units among multiple computers 90 interconnected via a network. The computer 90 is a concept that includes not only a single computer 90 but also a virtualized computer system.

[0147] The control unit is realized by the processor 901 reading various programs stored in the auxiliary storage device 903, loading them into the main storage device 902, and executing processing in accordance with the programs. The control unit can realize functional units that perform various types of information processing depending on the type of program. In this way, the computer is realized as an information processing device that performs information processing.

[0148] The storage unit is realized by a main storage device 902 and an auxiliary storage device 903. The storage unit stores data, various programs, and various databases. The processor 901 can allocate a storage area corresponding to the storage unit in the main storage device 902 or the auxiliary storage device 903 in accordance with the programs. The control unit can cause the processor 901 to add, update, and delete data stored in the storage unit in accordance with the various programs.

[0149] The term "database" refers to a relational database, which manages data sets called tables and masters in a tabular format structurally defined by rows and columns, by associating them with each other. In a database, a table is called a table, a master, a column in a table is called a column, and a row in a table is called a record. In a relational database, relationships between tables and masters can be established and associated. Typically, each table and each master is assigned a column that serves as a primary key to uniquely identify a record, but assigning a primary key to a column is not required. The control unit can cause the processor 901 to add, delete, or update records in specific tables and masters stored in the storage unit according to various programs. Furthermore, by storing data, various programs, and various databases in the storage unit, the information processing device and information processing system according to the present disclosure can be considered to have been manufactured.

[0150] Note that the databases and masters in this disclosure may include any data structure in which information is structurally defined (such as a list, dictionary, associative array, or object). The data structure also includes data that can be considered as a data structure by combining data with functions, classes, methods, etc. written in any programming language.

[0151] The communication unit is realized by the communication IF 991. The communication unit realizes the function of communicating with other computers 90 via a network. The communication unit can receive information transmitted from other computers 90 and input the information to the control unit. The control unit can cause the processor 901 to execute information processing on the received information in accordance with various programs. Furthermore, the communication unit can transmit information output from the control unit to other computers 90.

[0152] <Additional Notes> The matters described in the above embodiments will be added below.

[0153] (Supplementary Note 1) A program for causing a computer including a processor and a storage unit to process information relating to a dialogue between multiple users, the program executing a video acquisition step (S301) of acquiring video data relating to the dialogue, an object identification step (S302) of identifying object information relating to an object included in the video data based on the video data acquired in the video acquisition step, a dialogue acquisition step (S104) of acquiring dialogue data relating to the dialogue, and an object storage step (S304) of storing the object information identified in the object identification step in association with the dialogue data acquired in the dialogue acquisition step relating to the object information. This makes it possible to store object information such as slides displayed to conference participants included in a conference video, character strings included in a document, people and objects such as conference participants, in association with conference audio.

[0154] (Supplementary Note 2) The program according to Supplementary Note 1, wherein a processor executes a search input step (S501) of accepting input of search conditions from a user, an information retrieval step (S502) of searching for object information stored in an object storage step based on the search conditions accepted in the search input step, and an information presentation step (S505) of presenting to the user, as a search result, dialogue data stored in association with the object information retrieved in the information retrieval step. This enables the user to search for object information and to confirm, as a search result, the dialogue data stored in association with the object information.

[0155] (Supplementary Note 3) The program according to Supplementary Note 2, wherein the information search step (S502) is a step of searching for object information that includes information that matches the search conditions, and the information presentation step (S505) is a step of presenting a text-to-speech that is text information of what a speaker has said in dialogue data to the user as a search result. This allows the user to search for object information and also allows the user to confirm at a glance the text-to-speech that corresponds to the dialogue data stored in association with the object information as a search result.

[0156] (Supplementary Note 4) The program according to Supplementary Note 2 or 3, wherein the information presentation step (S505) includes a step of presenting a search result including the object information searched in the information search step, thereby allowing the user to confirm the object information searched based on the search conditions as well as the search result.

[0157] (Supplementary Note 5) The program according to any one of Supplementary Notes 2 to 4, wherein the information presenting step (S505) is a step of presenting dialogue data in association with a speaker who has uttered a voice included in the dialogue data. This allows the user to check information about the speaker who has uttered the voice in the dialogue data.

[0158] (Supplementary Note 6) The program according to Supplementary Note 1, wherein the processor executes an audio extraction step (S104) of extracting multiple pieces of section audio data for each speech section from the dialogue data acquired in the dialogue acquisition step, and an object storage step (S304) of storing the multiple pieces of section audio data extracted in the audio extraction step in association with object information identified in the object identification step. This makes it possible to store unstructured information, such as slides displayed to conference participants included in the conference video, character strings included in documents, facial expressions and emotions of conference participants, in association with the section audio data for each audio section determined based on speech and the like in the conference.

[0159] (Supplementary Note 7) The program according to Supplementary Note 6, wherein the processor executes a text generation step (S104) of generating a plurality of section reading texts, which are text information of what was spoken by the speaker, for each of the plurality of section audio data extracted in the audio extraction step, and an object storage step (S304) of storing the plurality of section reading texts generated in the text generation step in association with the object information identified in the object identification step. This makes it possible to store, for each audio section determined based on the speech etc. in the conference, unstructured information such as slides displayed to the conference participants contained in the conference video, character strings included in documents, facial expressions and emotions of the conference participants, in association with the section reading text, indicating what the speaker said.

[0160] (Supplementary Note 8) The program according to Supplementary Note 6 or 7, wherein the processor executes a speaker identification step (S303) of identifying a speaker speaking in the dialogue data stored in association with the object information stored in the object storage step, wherein the voice extraction step (S104) includes a step of identifying a speaker for each of a plurality of section voice data, and the speaker identification step (S303) is a step of identifying a speaker speaking in the dialogue data stored in association with the object information based on the speaker extracted in the voice extraction step. This allows a speaker associated with the object information to be identified based on the speaker identified for each speech section. Speakers related to object information can be stored in association with each other.

[0161] (Supplementary Note 9) The program according to Supplementary Note 1, wherein the object identification step (S302) includes a page section identification step of identifying a plurality of object information related to a plurality of page sections for each page of presentation slides included in the video data, thereby making it possible to accurately grasp what slides are included in the conference video and the content of each page included in the slides.

[0162] (Supplementary Note 10) The program according to Supplementary Note 9, wherein the object specifying step (S302) includes a string extraction step of extracting, for each of the plurality of page sections specified in the page section specifying step, a string included in a page of the presentation slide in that page section, and the object storing step (S304) includes a string storing step (S304) of storing, in association with each of the plurality of page sections specified in the page section specifying step, the string extracted in the string extraction step in that page section. This makes it possible to store what strings are included in each page included in the slides of the conference video, and to store the slides included in the conference video in association with the string information for each slide.

[0163] (Supplementary Note 11) The program according to Supplementary Note 10, wherein the character string extraction step (S302) is a step of recognizing and extracting character strings included in image data by analyzing image data of a page of a presentation slide. This makes it possible to easily extract character strings included in a page of a slide.

[0164] (Supplementary Note 12) The program according to Supplementary Note 10, wherein the character string extraction step (S302) is a step of recognizing and extracting character strings included in a slide file by referencing the slide file linked to the page of the presentation slide and analyzing the slide file. This allows for more accurate extraction of character strings included in the slide page.

[0165] (Supplementary Note 13) The program according to any one of Supplementary Notes 9 to 12, wherein the video acquisition step (S301) is a step of acquiring multiple different video data, and the processor executes a section reception step (S702) of receiving input of a specific page section from among the multiple page sections identified in the page section identification step, a video identification step (S703) of identifying multiple different specific video data associated with the same page as the presentation slide page in the specific page section received in the section reception step, and a video presentation step (S704) of presenting the specific video data identified in the video identification step. This allows the contents of a meeting using each slide page to be listed and confirmed. For example, if the meeting content is a business negotiation, it is possible to list and understand business negotiation videos by different sales representatives using the same presentation slide. Specifically, it is possible to compare and confirm multiple business negotiation videos.

[0166] (Supplementary Note 14) The program according to Supplementary Note 13, wherein the processor executes a speaker receiving step (S802) of receiving speaker input, and a video identifying step (S803) of identifying multiple different predetermined video data associated with the speaker based on the speaker received in the speaker receiving step. This allows for a list of conference content in which each speaker spoke, to be checked. For example, if the conference content is a business negotiation, it is possible to easily check the video of the business negotiation conducted by a predetermined sales representative.

[0167] (Supplementary Note 15) The program according to Supplementary Note 13 or 14, wherein the video presentation step (S704) is a step of associating and presenting, for each predetermined video data, information indicating the period during which the presentation slide pages are displayed in the predetermined page section received in the section reception step. This makes it possible to correlate and understand the slide display times for sales negotiation videos by different sales representatives who use the same presentation slides. It is possible to check how long each sales representative spent negotiating in each sales negotiation video.

[0168] (Supplementary Note 16) The program according to any one of Supplementary Notes 13 to 15, wherein the video presentation step (S704) is a step of, for each predetermined video data, associating and presenting information indicating the comparison result between a character string corresponding to the page of the presentation slide in the predetermined page section received in the section reception step and a section readout text, which is text information of the content spoken by the speaker in the predetermined page section. This makes it possible to check, for example, the degree to which the talk script (character string) assigned to each slide matches the content actually spoken by the sales representative for sales negotiation videos using the same presentation slides. It is easy to understand whether the sales representative is conducting sales activities in accordance with the talk script.

[0169] (Supplementary Note 17) The video presentation step (S704) is a step of associating and presenting, for each predetermined video data, information indicating the degree of match between the character string and the section readout text, in the program according to Supplementary Note 16. This makes it possible to check, for example, for videos of sales negotiations by different sales representatives using the same presentation slides, the degree to which the talk script (character string) assigned in advance to each slide matches the content actually spoken by the sales representative.

[0170] (Supplementary Note 18) An information processing device including a processor and a storage unit, wherein the processor executes a program according to any one of Supplementary Notes 1 to 17. This makes it possible to more accurately grasp what slides are included in the meeting content.

[0171] (Supplementary Note 19) A method for manufacturing an information processing system including an information processing device having a processor and a storage unit, the method including providing the information processing device with a program according to any one of Supplementary Notes 1 to 17. This makes it possible to more accurately grasp what slides are included in the meeting content.

[0172] (Supplementary Note 20) An information processing method executed by a computer including a processor and a storage unit, wherein the computer executes a program according to any one of Supplementary Notes 1 to 17. This makes it possible to more accurately grasp what slides are included in the meeting content.

[0173] 1 System, 10 Server, 101 Memory unit, 104 Control unit, 106 Input device, 108 Output device, 20 First user terminal, 201 Memory unit, 204 Control unit, 206 Input device, 208 Output device, 30 Second user terminal, 301 Memory unit, 304 Control unit, 306 Input device, 308 Output device, 50 CRM system, 501 Memory unit, 504 Control unit, 506 Input device, 508 Output device, 60 Voice server (PBX), 601 Memory unit, 604 Control unit, 606 Input device, 608 Output device

Claims

1. A program that causes a computer having a processor and a storage unit to process information regarding interactions between a plurality of users, the processor: a video acquisition step of acquiring video data relating to the dialogue; an object identification step of identifying object information related to an object included in the video data based on the video data acquired in the video acquisition step; a dialogue acquisition step of acquiring dialogue data relating to the dialogue; an object storage step of storing the object information identified in the object identification step in association with the dialogue data regarding the object information acquired in the dialogue acquisition step; Run the object specifying step includes a page section specifying step of specifying a plurality of pieces of object information relating to a plurality of page sections for each page of a presentation slide included in the video data; program.

2. the processor: a search input step for accepting input of search conditions from a user; an information retrieval step of retrieving the object information stored in the object storage step based on the search conditions received in the search input step; an information presenting step of presenting to the user, as a search result, the dialogue data stored in association with the object information searched in the information searching step; To execute The program according to claim 1.

3. the information search step is a step of searching for object information that includes information that matches the search conditions based on the search conditions, the information presenting step is a step of presenting a read-aloud text, which is text information of the content uttered by a speaker in the dialogue data, to the user as a search result. The program according to claim 2.

4. the information presenting step includes a step of presenting the search results including the object information searched in the information searching step.

4. The program according to claim 2 or 3.

5. the information presenting step is a step of presenting the dialogue data in association with a speaker who is uttering a voice included in the dialogue data.

4. The program according to claim 2 or 3.

6. the processor: a voice extraction step of extracting a plurality of section voice data for each utterance section from the dialogue data acquired in the dialogue acquisition step; Run the object storing step is a step of storing the object information identified in the object identifying step in association with the plurality of section audio data extracted in the audio extracting step. The program according to claim 1.

7. the processor: a text generation step of generating a plurality of section reading texts, which are text information of the contents uttered by the speaker, for each of the plurality of section voice data extracted in the voice extraction step; Run the object storing step is a step of storing the object information identified in the object identifying step in association with the plurality of section reading texts generated in the text generating step. The program according to claim 6.

8. the processor: a speaker identifying step of identifying a speaker who is speaking in the dialogue data stored in association with the object information stored in the object storing step; Run the voice extraction step includes a step of identifying a speaker for each of the plurality of section voice data; the speaker identifying step is a step of identifying the speaker who is speaking in the dialogue data stored in association with the object information, based on the speaker extracted in the voice extracting step.

8. The program according to claim 6 or 7.

9. the object specifying step includes a character string extracting step of extracting, for each of the plurality of page sections specified in the page section specifying step, a character string included in a page of the presentation slide in the page section; the object storing step includes a string storing step of storing the string extracted in the string extracting step in each of the plurality of page sections identified in the page section identifying step in association with the page section. The program according to claim 1.

10. the character string extraction step is a step of recognizing and extracting character strings included in image data by analyzing image data of the page of the presentation slide; The program according to claim 9.

11. the character string extraction step is a step of recognizing and extracting character strings included in a slide file by referencing a slide file linked to the page of the presentation slide and analyzing the slide file; The program according to claim 9.

12. the video acquisition step is a step of acquiring a plurality of different video data; the processor: a section receiving step of receiving an input of a predetermined page section from among the plurality of page sections identified in the page section identifying step; a moving image specifying step of specifying a plurality of different predetermined moving image data associated with the same page as the page of the presentation slide in the predetermined page section received in the section receiving step; a video presentation step of presenting the predetermined video data identified in the video identification step; To execute 12. The program according to claim 1, 2, 3, 6, 7, 9, 10, 11, or 12.

13. the processor: a speaker receiving step of receiving an input from a speaker; Run the moving image identifying step is a step of identifying the plurality of different predetermined moving image data associated with the speaker based on the speaker accepted in the speaker accepting step. The program according to claim 12.

14. the video presenting step is a step of associating information indicating a period during which pages of the presentation slides in the predetermined page section received in the section receiving step are displayed with each of the predetermined video data and presenting the information. The program according to claim 12.

15. the video presenting step is a step of presenting, for each of the predetermined video data, information indicating a comparison result between a character string corresponding to a page of the presentation slide in the predetermined page section received in the section receiving step and a section reading text, which is text information of the content spoken by a speaker in the predetermined page section, in association with the character string; The program according to claim 12.

16. the video presentation step is a step of presenting information indicating a degree of match between the character string and the section reading text in association with each of the predetermined video data. The program according to claim 15.

17. An information processing device including a processor and a storage unit, The processor executes a program according to any one of claims 1 to 3, 6, 7, 9 to 11. Information processing device.

18. A method for manufacturing an information processing system including an information processing device having a processor and a storage unit, providing the information processing device with a program according to any one of claims 1 to 3, 6, 7, 9 to 11, A method for manufacturing an information processing system.

19. An information processing method executed by a computer including a processor and a storage unit, The computer executes the program according to any one of claims 1 to 3, 6, 7, 9 to 11. Information processing methods.