System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving conference setting information and automatically adding a virtual participant to a conference; means for collecting speech of the participant and projected materials in real time during the conference; means for analyzing the collected data and generating a summary; means for automatically creating a summary video of a short period of time based on the generated summary; and means for providing the summary video to a user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, there are several problems with the methods for sharing and storing meeting content in online meetings. Specifically, recording meetings requires a large amount of storage space, and watching the recording takes as long as the actual meeting time, making it inefficient. Furthermore, creating meeting minutes is time-consuming and labor-intensive, and prone to errors or oversights. To solve these problems, there is a need for a method to efficiently summarize meeting content and provide it in a format that can be quickly understood. [Means for solving the problem]

[0005] To solve the above-mentioned problems, the present invention provides the following means. The system includes: means for receiving conference setting information and automatically adding virtual participants to a conference; means for collecting participant remarks and projected materials in real time during the conference; means for analyzing the collected data to generate a summary; means for automatically creating a short summary video based on the generated summary; and means for providing the summary video to a user. This eliminates the need for recording the conference and creating minutes, and enables users to efficiently grasp the key points of the conference. The system of the present invention further includes means for converting remark data collected during the conference into text using speech recognition technology and means for acquiring projected materials using image capture technology. The system may also include means for storing the collected data in cloud storage and backing it up. A machine learning model is used to generate the summary, and a video template is used to generate the summary video based on the summary, providing the user with a highly efficient and visually easy-to-understand format. The server notifies the user when the summary video is complete and provides an access link, allowing the user to easily view it.

[0006] "Conference setting information" is information provided by a user when setting up a Web conference, and includes the date and time of the conference, a list of participants, an invitation link, and the like.

[0007] "Virtual participant" refers to an artificial intelligence or software agent that participates in a web conference and collects and analyzes comments and materials in real time.

[0008] "Speech" refers to what participants say orally during a web conference, and is recorded as audio.

[0009] "Projected materials" refers to digital content such as presentation slides, documents, and images that are displayed on the screen during a web conference.

[0010] A "summary" refers to a concise text that extracts and summarizes the main points from the statements and materials collected during a meeting.

[0011] A "summary video" is a short video automatically created based on the generated summary, and provides the main points of the meeting in a visually understandable format.

[0012] "Speech recognition technology" refers to artificial intelligence technology for converting voice data into text data, and is used to convert statements made during meetings into text.

[0013] "Image capture technology" refers to technology that captures materials displayed on a screen as digital images and is used to automatically capture projected materials.

[0014] "Cloud storage" refers to a remote server that stores and accesses data over the internet and is used to store data collected during a meeting.

[0015] "Backup" refers to the process of creating and storing a copy of data in case of data loss or corruption.

[0016] A "machine learning model" is a type of artificial intelligence that can learn through data analysis and pattern recognition to perform specific tasks, and is used in the present invention to generate summaries.

[0017] A "video template" is a format that defines the basic layout and structure when creating a video, and is used to automatically generate a summary video based on the summary.

[0018] "Notification" refers to a message or alert that notifies the user that the summary video is complete.

[0019] The "access link" refers to a URL on the Internet that allows a user to view the summary video. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] This invention is a system that invites a generation AI to an online meeting as a virtual participant, collects and analyzes comments and projected materials during the meeting, and automatically generates a short summary video after the meeting. This system eliminates the need to record meetings and create minutes, and allows users to efficiently grasp the key points of the meeting.

[0042] Overall concept of the system

[0043] The system consists of the following elements:

[0044] 1. User Initial Setup: The user adds the generated AI to the meeting invitation list using their web conferencing account.

[0045] 2. AGI participation by the server: The server automatically joins the virtual participant (AGI) to the conference at the conference start time.

[0046] 3. Data collection and analysis using AGI: Collect and analyze statements and materials during meetings in real time.

[0047] 4. Summary video generation by the server: After the meeting ends, a summary video is generated based on the collected and analyzed data.

[0048] 5. Providing summary videos to users: Providing summary videos to users to help them understand the contents of the meeting efficiently.

[0049] Program Processing Details

[0050] User Initial Settings

[0051] Users simply log in to their web conferencing system and add the generated AI (virtual participant) to the invitation list when setting up a meeting. This process is an additional step in the normal operation of scheduling a meeting, and users can easily perform it.

[0052] Server AGI participation

[0053] When the meeting starts, the server automatically joins the AGI to the meeting. The server uses the specified meeting URL or invitation link to associate virtual participants with the meeting. At this stage, the server verifies that the AGI connection was successful.

[0054] AGI data collection and analysis

[0055] During the meeting, the AGI converts participants' speech into text data using speech recognition technology and records the speech in real time. It also captures presentations and materials shared by participants on their screens using image capture technology. The captured data is then stored in cloud storage.

[0056] Server-generated summary video

[0057] When the meeting ends, the server uses a machine learning model to generate a summary based on the data collected and analyzed by the AGI. This summary extracts the main points of the meeting. The server then automatically generates a summary video using a video template based on this summary. The generated video is then saved back to cloud storage.

[0058] Providing summary videos to users

[0059] Finally, the server notifies the user that the summary video is complete. This notification includes an access link to the summary video, which the user can click to watch. This allows the user to quickly grasp the main points of the meeting.

[0060] Specific examples

[0061] For example, User A schedules a monthly meeting and adds AGI to the invitation list. When the meeting starts, the server automatically invites AGI to join. During the meeting, AGI converts speeches into text and captures projected materials. When the meeting ends, AGI creates a summary from the collected data, and the server generates a summary video based on this summary. A few minutes later, User A receives a notification that the summary video is complete and can click the provided link to watch the video. This process allows User A to efficiently understand the main points of the meeting.

[0062] The specific embodiment of the present invention has been described above. This system allows users to quickly and efficiently grasp important information without having to go through the trouble of recording meetings and creating minutes.

[0063] The processing flow will be explained below.

[0064] Step 1:

[0065] A user logs into their own web conferencing system and sets up a new conference. At that time, they enter conference setting information such as the conference date and time, participant list, and invitation link, and add the generated AI (AGI) to the invitation list as a virtual participant.

[0066] Step 2:

[0067] The server schedules the AGI's participation based on the conference setting information received from the user. At the conference start time, the server sets a trigger to automatically join the AGI.

[0068] Step 3:

[0069] When it is time for the meeting to start, the server uses the meeting URL or invitation link to invite the AGI to join. The server verifies that the AGI has successfully connected to the meeting.

[0070] Step 4:

[0071] During the meeting, the terminal (AGI projection system) converts participants' speech into text in real time using speech recognition technology, and also captures presentations and materials shared on the screen by users and other participants using image capture technology.

[0072] Step 5:

[0073] The server stores the speech text and image capture data obtained during the meeting in cloud storage, which ensures consistent storage and access of the data.

[0074] Step 6:

[0075] Once the meeting is over, the server begins analyzing the data stored in cloud storage, using machine learning models to extract key points from the speech and materials and generate a summary text.

[0076] Step 7:

[0077] The server selects a video template based on the generated summary text and materials, and automatically generates a summary video based on the summary. The generated summary video is then saved in cloud storage.

[0078] Step 8:

[0079] Once the summary video is complete, the server sends a notification to the user, which includes a link to access the summary video, allowing the user to easily watch the video.

[0080] Step 9:

[0081] Users can click on the link provided by the server to watch the summary video, which allows them to quickly and efficiently grasp the main points of the conference.

[0082] Specific examples

[0083] As an example, let's explain the process of User A setting up a monthly meeting. In step 1, User A adds the generation AI to the invitation list and completes the meeting setup. From step 2 to step 9, the server automatically invites the AGI to the meeting and collects the content of comments and materials during the meeting. After the meeting ends, the AGI analyzes the data and generates a summary, and the server automatically generates a summary video. Finally, User A receives an access link to the summary video and can efficiently understand the key points of the meeting by watching the video. This process frees User A from the hassle of recording meetings and creating minutes.

[0084] Example 1

[0085] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0086] In today's business environment, online meetings are becoming more frequent and consume a lot of time. Recording meetings and creating minutes requires time and effort, making it difficult to efficiently grasp the information. It is also not easy to accurately summarize the content of meetings and extract only the key points. These issues increase the risk of important information being overlooked after the meeting.

[0087] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0088] In this invention, the server includes means for receiving conference setting information and automatically adding virtual participants to the conference, means for collecting participants' remarks and projection materials in real time during the conference, means for analyzing the collected data and generating a summary using a generative AI model, means for automatically creating a short summary video based on the generated summary, and means for providing the summary video to the user. This eliminates the need to record the conference and create minutes, and enables users to efficiently grasp the key points of the conference.

[0089] "Meeting setting information" refers to information necessary to set up a meeting, such as the online meeting schedule, participant list, and invitation URL.

[0090] "Virtual participants" refer to participants who are not real people, but are created using artificial intelligence such as generative AI.

[0091] "Participant Remarks" refers to oral presentations, discussions, and comments made by participants during a meeting.

[0092] "Projected materials" refers to digital materials such as presentation slides, documents, and images that are shared on the screen during a meeting.

[0093] A "generative AI model" refers to an artificial intelligence algorithm that has been trained to perform a specific task.

[0094] "Speech recognition technology" refers to technology for converting voice data into text.

[0095] "Image capture technology" refers to technology for capturing visual information from a screen or digital data.

[0096] "Cloud storage" refers to a service for storing and managing data on remote servers on the Internet.

[0097] A "summary video" is a video created to convey the main points of a meeting in a short amount of time.

[0098] A "prompt" is a textual instruction entered into a generative AI model to perform a specific task.

[0099] This invention is a system that invites a generation AI as a virtual participant in an online meeting, collects and analyzes the content of comments made during the meeting and the materials projected, and automatically generates a short summary video after the meeting ends. This invention eliminates the need to record meetings and create minutes, and allows users to efficiently grasp the key points of the meeting.

[0100] Overall concept of the system

[0101] The system consists of the following elements:

[0102] 1. User Initial Setup: The user adds the generated AI to the meeting invitation list using their web conferencing account.

[0103] 2. AGI participation by the server: The server automatically joins the virtual participant (AGI) to the conference at the conference start time.

[0104] 3. Data collection and analysis using AGI: Collect and analyze statements and materials during meetings in real time.

[0105] 4. Generation of summary video by the server: After the meeting ends, a summary video is generated based on the collected and analyzed data.

[0106] 5. Providing users with video summaries: Providing users with video summaries allows them to efficiently understand the content of the meeting.

[0107] Program Processing Details

[0108] User Initial Settings

[0109] The user logs in to their own web conferencing system (e.g., a general web conferencing system) and invites the generated AI (virtual participant) when setting up a conference. On the setting screen, they enter the generated AI's email address or ID and save it. This completes the setting for the generated AI to automatically join the conference.

[0110] Server AGI participation

[0111] As the meeting starts, the server automatically joins the AGI configured as a virtual participant to the meeting. The server connects the AGI using the meeting URL or invitation link provided by the user. At this stage, the server verifies that the AGI has successfully joined the meeting.

[0112] AGI data collection and analysis

[0113] When a meeting begins, the AGI performs the following tasks:

[0114] Speech Recognition: Use speech recognition technology (e.g., Google Speech-to-Text API, common speech recognition technology) to convert participants' speech into text in real time.

[0115] Document capture: Capture presentations and documents shared by participants on their screens using image capture technology (e.g., common image processing technology).

[0116] The acquired data is stored in cloud storage (e.g., a general cloud storage service).

[0117] Server-based generation of summary videos

[0118] After the meeting ends, the server generates a summary based on the data collected and analyzed by the AGI. To do this, it uses a generative AI model (e.g., a general machine learning model). The summary extracts the main points of the meeting, and the server automatically generates a summary video based on this summary using a video template (e.g., general video editing software). The generated video is then saved back to cloud storage.

[0119] Providing users with video summaries

[0120] The server notifies the user that the summary video is complete. This notification includes an access link to the summary video, which the user can view by clicking the link. This allows the user to quickly grasp the main points of the meeting.

[0121] Specific examples

[0122] For example, User A schedules a monthly meeting and adds a generated AI (virtual participant) to the invitation list. When the meeting starts, the server automatically invites the AGI to the meeting, where it performs real-time speech recognition and captures projected materials. When the meeting ends, the server generates a summary based on the collected data and creates a summary video based on the content. Finally, User A receives a notification a few minutes later that the summary video is ready, and can click the provided link to watch the video. This allows User A to quickly and efficiently understand the key points of the meeting.

[0123] Prompt Sentence Examples

[0124] Here are some example prompts for a generative AI model to generate a summary:

[0125] Please summarize the following meeting in a concise manner, extracting the main points and conclusions.

[0126] 1. Speaker A: "Today's meeting agenda is as follows..."

[0127] 2. Speaker B: "Regarding the budget review..."

[0128] 3. Speaker C: "I'll report on the progress of the following project..."

[0129] [Detailed minutes of the meeting follow]

[0130] Please summarize the main points and conclusions of this meeting in bullet points.

[0131] The specific embodiment of the present invention has been described above. This system allows users to quickly and efficiently grasp important information without having to go through the trouble of recording meetings and creating minutes.

[0132] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0133] Step 1: User Initial Setup

[0134] The user logs in to their own web conferencing system and schedules a meeting. At this time, they add the generated AI as a virtual participant. Specifically, they enter the generated AI's email address or ID on the meeting setting screen and add it to the invitation list. The input is the meeting setting information, and the output is the schedule data for the meeting to which the virtual participant has been added.

[0135] Step 2: Server joins AGI

[0136] As the start time of the conference approaches, the server automatically invites the generated AI (AGI) to join the conference. The server obtains the conference URL or invitation link based on the conference information set by the user and connects the AGI to that URL. This allows the AGI to join the conference. The input is the set conference URL and AGI connection information, and the output is status information indicating that the AGI has successfully joined the conference.

[0137] Step 3: AGI collects data

[0138] When a meeting begins, the AGI collects speech and projections in real time. Specifically, it performs the following processes:

[0139] 1. Speech recognition: Converting what is being said in a meeting into text data in real time, using tools such as the Google Speech-to-Text API.

[0140] 2. Document capture: Slides and documents shared during meetings are captured using image capture technology, and their contents are then analyzed and converted into text data using OCR (Optical Character Recognition) technology.

[0141] The input is the conference audio and shared screen data, and the output is the textual utterance data and captured material data.

[0142] Step 4: Save your data

[0143] The collected text data and image data are stored in cloud storage by the server. Data is stored using a common cloud storage service (e.g., Amazon S3). The input is the data collected by the AGI, and the output is the data stored in cloud storage.

[0144] Step 5: Generate a summary

[0145] After the meeting ends, the server analyzes the data stored in cloud storage and generates a summary using a generative AI model. Specifically, it uses a machine learning model (e.g., GPT-4) to extract the main points of the meeting based on the prompt. The input is the saved meeting data and the prompt, and the output is the summary text.

[0146] Step 6: Generate summary video

[0147] Based on the generated summary text, the server uses a video template to create a summary video. Using video editing software (e.g., general video editing software), a video that integrates text, audio narration, and images is automatically generated. The input is the summary text and an existing video template, and the output is the completed summary video.

[0148] Step 7: Provide the summary video to the user

[0149] The server notifies the user that the summary video is complete. The notification includes an access link to the summary video, which the user can click to watch. The input is the generated summary video, and the output is the link to provide the summary video and a notification message.

[0150] This series of processes allows users to efficiently grasp important information without having to go through the trouble of recording meetings and creating minutes.

[0151] (Application example 1)

[0152] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0153] In the operation management of autonomous vehicles, it is difficult to efficiently grasp the operation status and important events of multiple vehicles. In particular, as a large amount of data is generated from camera footage and various sensors during operation, operation managers are required to quickly and accurately analyze this data and make appropriate decisions. However, current technology lacks the means to efficiently collect and analyze this data and provide a summary of the operation status.

[0154] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0155] In this invention, the server includes means for receiving conference setting information and automatically adding virtual participants to the conference, means for collecting participant remarks and projection materials in real time during the conference, means for analyzing camera and sensor data collected while the vehicle is in operation to generate a summary, means for automatically creating a short summary video based on the generated summary, and means for providing the summary video to the user. This enables the operation manager of the autonomous vehicle to quickly and accurately grasp important events and driving conditions during operation and take prompt action.

[0156] "Conference setup information" is information necessary to automatically add virtual participants to a conference.

[0157] "Virtual participants" refers to generative AI that automatically participates in meetings and car rides, collecting and analyzing data.

[0158] The "means for automatically adding a virtual participant to a conference" is a method for automatically adding a virtual participant to a conference when the conference starts.

[0159] "Means of collecting information in real time" refers to methods of collecting information such as speech content, projected materials, camera footage, and sensor data on the spot during meetings or while driving a car.

[0160] A "camera" is an image capturing device mounted on a vehicle that records images while the vehicle is in motion.

[0161] A "sensor" is a data acquisition device installed in a vehicle that detects and records various physical information during operation.

[0162] "Means for analyzing and generating a summary" refers to a method for analyzing collected data and concisely summarizing its contents.

[0163] The "means for automatically creating a short summary video" is a means for automatically creating a short video of a meeting or operation status based on the summarized content.

[0164] The "means for providing to the user" refers to a method for providing the summary video in a form that allows the user to easily access it.

[0165] "Speech recognition technology" is a technology that converts voice into text data.

[0166] "Image capture technology" is a technology that captures projected materials or images as still images.

[0167] "Cloud storage" is a storage service for saving and managing data on the Internet.

[0168] "Means for backing up stored data" refers to a method for replicating data stored in cloud storage on a regular or ad-hoc basis to prepare for data loss.

[0169] This invention is a data collection and analysis system for managing the operation of autonomous vehicles. It analyzes camera footage and sensor data collected from the vehicle during operation, summarizes the results, and generates and provides a short summary video, allowing the operation manager to quickly understand the situation. Specific system configurations for implementing this invention are described below.

[0170] Overall concept of the system

[0171] The system consists of the following main components:

[0172] Camera image collection module (installed in autonomous vehicles)

[0173] Sensor data collection module (installed in autonomous vehicles)

[0174] Data analysis server

[0175] Generative AI Models

[0176] Summary video generation module

[0177] Cloud Storage

[0178] User Interface

[0179] Camera footage and sensor data collection

[0180] Autonomous vehicles are equipped with multiple cameras and sensors. These devices collect video and sensor data in real time while driving. The cameras record the vehicle's surroundings as video, while the sensors detect various data such as speed, position, acceleration, and the presence or absence of obstacles.

[0181] Data analysis server

[0182] The data analysis server receives and analyzes data collected from autonomous vehicles in real time. A generative AI model is used to analyze the collected camera footage and sensor data to generate summaries. For example, it generates summaries that explain traffic conditions and the occurrence of important events (such as sudden braking or sudden stops).

[0183] Summary video generation

[0184] The summary video generation module generates a summary video based on the analyzed data. The summary video includes images showing important events and situations during operation, along with related text information. This allows users to quickly understand the operation status.

[0185] Cloud storage and user provisioning

[0186] The generated summary video is stored in cloud storage, allowing the dispatcher to easily access it from anywhere. Through the user interface, the dispatcher can access the summary video and quickly obtain the information they need.

[0187] Specific examples

[0188] For example, when an autonomous vehicle passes through multiple intersections, it detects traffic conditions and events that occur at the intersections (such as a sudden vehicle stop or the appearance of an unexpected obstacle) from camera footage and sensor data. The generative AI model analyzes this data in real time and summarizes important points such as "A sudden stop occurred at intersection 1" or "A pedestrian ran out into the road at intersection 2" into text. Next, the summary video generation module creates a short summary video based on the summary and saves it in cloud storage. The operation manager can view this summary video via the user interface.

[0189] Examples of prompt statements

[0190] Example prompt for a generative AI model:

[0191] "Please summarize any significant events or notable operational findings from this camera footage and sensor data."

[0192] Such a system allows operation managers to quickly grasp key points extracted from large amounts of data and respond promptly.

[0193] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0194] Step 1:

[0195] Initial Setup

[0196] The server prepares to add virtual participants (generative AI models) while each vehicle is in operation based on the operation schedule and setting information of the autonomous vehicle received from the operation manager. The input is operation schedule information, and the output is a notification that the virtual participants have been set up. The server saves the setting information so that the virtual participants can automatically join the operation when the operation starts.

[0197] Step 2:

[0198] Starting Data Collection

[0199] Once the vehicle begins operation, the cameras and sensors installed on the vehicle begin collecting data in real time. The cameras record images of the surrounding area, and the sensors acquire data such as speed, position, acceleration, and obstacles. The input is the camera images and sensor data, and the output is the collected data. The server periodically receives this data and prepares it for analysis.

[0200] Step 3:

[0201] Data analysis

[0202] The server receives camera footage and sensor data transmitted sequentially during operation and analyzes it using the generative AI model. The input is the camera footage and sensor data collected in step 2, and the output is the analyzed summary information. The server sends prompts to the generative AI model, asking for the following to summarize important events and situations:

[0203] "Please summarize any significant events or notable operational findings from this camera footage and sensor data."

[0204] Step 4:

[0205] Summary video generation

[0206] The server generates a summary video based on the summary information obtained in step 3. The summary video generation module combines the summarized text information with related video to create a video that can be viewed in a short time. The input is the analyzed summary information, and the output is a summary video. The server saves the generated video in cloud storage.

[0207] Step 5:

[0208] Providing a summary video

[0209] After the generated summary video is saved in cloud storage, the server sends a notification to the operations manager. The notification includes an access link to the video, which the manager can click to view. The input is the summary video information saved in cloud storage, and the output is a notification sent to the operations manager. By watching the summary video, the operations manager can instantly understand important events and situations that occurred during operations.

[0210] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0211] The present invention is a system that combines an emotion recognition engine with an online conference, and provides a means to understand the main points of the conference more deeply by analyzing the emotions of participants during the conference and reflecting the results in a summary video. This system is implemented by the following main steps.

[0212] Overall concept of the system

[0213] The system consists of the following elements:

[0214] 1. User Initial Setup: The user adds the generative AI and emotion recognition engine to the meeting invitation list using their web conferencing account.

[0215] 2. AGI participation by the server: The server automatically joins the virtual participant (AGI) to the conference at the conference start time.

[0216] 3. Data collection and analysis: Using an emotion recognition engine and voice recognition technology, the content of statements made during meetings and the emotions of participants are collected and analyzed in real time.

[0217] 4. Summary video generation: A summary video is generated based on the collected data and the results of sentiment analysis.

[0218] 5. Providing users with video summary: Providing users with video summary provides an efficient way to understand the main points of the meeting and its emotions.

[0219] Program Processing Details

[0220] User Initial Settings

[0221] A user logs into their own web conferencing system and sets up a new meeting. At that time, they input meeting setting information such as the meeting date and time, participant list, and invitation link, and add the generative AI (AGI) and emotion recognition engine to the invitation list as virtual participants.

[0222] Participation of AGI and emotion recognition engine by server

[0223] The server automatically joins the virtual participants (AGI and emotion recognition engine) to the meeting at the scheduled meeting time. It uses the meeting URL or invitation link to ensure they are properly connected.

[0224] Data collection and analysis

[0225] During the meeting, the device (AGI projection system) converts participants' speech into text in real time using speech recognition technology. It also captures presentations and materials shared on the screen by the user and other participants using image capture technology. Furthermore, the emotion recognition engine analyzes the content and tone of speech to recognize participants' emotions in real time.

[0226] Emotion data storage and analysis

[0227] The server stores the captured text, image captures, and sentiment data in cloud storage, and uses a machine learning model to analyze the content and sentiment data and generate a summary that also reflects the results of sentiment analysis.

[0228] Summary video generation

[0229] After the meeting ends, the server generates a summary based on the stored data and identifies key moments from the meeting based on the emotion data. The server then uses a video template to automatically generate a summary video that reflects the results of the summary and emotion analysis. The summary video includes graphs and highlights that visually show changes in participants' emotions.

[0230] Providing summary videos to users

[0231] When the summary video is completed, the server sends a notification to the user, which includes a link to access the summary video. The user can click the link to watch the video. This allows the user to quickly grasp the main points of the meeting and changes in participants' emotions.

[0232] Specific examples

[0233] Let's take the example of User B setting up a weekly report meeting. In Step 1, User B adds the generation AI and emotion recognition engine to the invitation list and completes the meeting setup. From Step 2 to Step 6, the server automatically adds virtual participants to the meeting and collects and analyzes comments, materials, and emotion data from the meeting. After the meeting ends, the server generates a summary video based on the summary and emotion analysis results. Finally, User B receives a link to access the summary video and can watch the meeting highlights, which reflect the emotion data, to efficiently understand the key points of the meeting and changes in participants' emotions. This process frees User B from the hassle of recording meetings and creating minutes, and also makes it easier for them to understand participants' emotions.

[0234] The above describes a specific embodiment of the present invention. This system allows users to save the time and effort of recording meetings and creating minutes, and enables them to efficiently grasp not only important information but also changes in the emotions of participants.

[0235] The processing flow will be explained below.

[0236] Step 1:

[0237] A user logs into their own web conferencing system and sets up a new online meeting. At this time, they input information such as the meeting date and time, participant list, and invitation link, and add the generative AI (AGI) and emotion recognition engine to the invitation list as virtual participants.

[0238] Step 2:

[0239] When the meeting starts, the server automatically joins the AGI and emotion recognition engine to the meeting, which involves connecting using the meeting URL or invitation link.

[0240] Step 3:

[0241] During the meeting, the terminal (AGI projection system) converts participants' speech into text in real time using voice recognition technology, and captures presentations and materials using image capture technology.

[0242] Step 4:

[0243] Similarly, during a meeting, the emotion recognition engine analyzes the content and tone of speech to identify participants' emotions in real time, and this emotional data is instantly saved as digital data.

[0244] Step 5:

[0245] After the meeting ends, the server stores the data collected by the AGI and emotion recognition engine (speech text, document capture, and emotion analysis data) in cloud storage.

[0246] Step 6:

[0247] The server analyzes the stored data and generates a summary using the speech content and emotion data, which includes important speech content and emotional changes.

[0248] Step 7:

[0249] The server automatically generates a summary video using a video template based on the generated summary and the results of sentiment analysis, highlighting key points and important emotional moments from the meeting.

[0250] Step 8:

[0251] Once the summary video is complete, the server sends a notification to the user, which includes a link to access the summary video.

[0252] Step 9:

[0253] Users receive a notification and click the provided link to watch the summary video, which allows them to quickly grasp the key points of the meeting and the changes in participants' emotions.

[0254] Specific examples

[0255] For example, if User B schedules a weekly reporting meeting, in Step 1, the generation AI and emotion recognition engine are added to the meeting. When the meeting begins, in Steps 2 to 4, the AGI and emotion recognition engine automatically join the meeting and collect the remarks, materials, and emotion data. After the meeting ends, in Steps 5 to 7, the server generates a summary and automatically generates a summary video that reflects the emotion data. Finally, in Steps 8 and 9, User B can watch the summary video and understand the highlights of the meeting that reflect the emotion data. This process frees User B from the hassle of recording meetings and creating minutes, and also makes it easier to understand the emotions of the participants.

[0256] Example 2

[0257] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0258] While current online meeting systems allow participants to share remarks and materials during meetings, it is difficult to efficiently grasp participants' emotions and key moments. Furthermore, current systems do not automatically generate summary videos that include meeting summaries or emotion analysis results. This makes it difficult to effectively follow up on meetings and grasp key points, and reviewing the content of long meetings afterward is particularly time-consuming and labor-intensive.

[0259] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving conference setting information and automatically adding virtual participants to the conference, means for collecting participants' comments and projection materials in real time during the conference, means for analyzing the collected comment content and emotion data to generate a summary, means for automatically creating a short summary video based on the generated summary and emotion analysis results, and means for providing the summary video to the user. This makes it possible to efficiently grasp the content of the conference and changes in participants' emotions, and to quickly understand the gist of important information and emotions.

[0260] "Conference setting information" is information necessary to hold a conference, such as the date and time of the online conference, a list of participants, and an invitation link.

[0261] A "virtual participant" is an agent or program that is not physically present but is brought into an online conference by the system.

[0262] "Speech content" refers to voice data spoken by participants during an online conference that has been converted into text using voice recognition technology.

[0263] "Projected materials" are visual materials such as presentations and documents that participants share during an online meeting.

[0264] "Means of collection" refers to technologies and systems for acquiring participants' comments and projected materials in real time during online meetings.

[0265] "Speech recognition technology" is a technology for converting voice data into text data.

[0266] "Image capture technology" is a technology that captures displayed visual materials as image data.

[0267] "Emotional data" is emotional information analyzed based on the content of participants' statements and tone of voice.

[0268] A "summary" is a concise summary of the main points of a meeting based on collected speech and sentiment data.

[0269] A "summary video" is a short video generated from collected data that visually shows key moments of the meeting and the results of sentiment analysis.

[0270] "Cloud storage" is an external storage service for storing and managing data via the Internet.

[0271] "Backup measures" are techniques and methods for regularly replicating and backing up stored data.

[0272] The present invention is a system that combines an online conference with an emotion recognition engine, analyzes the content of comments and emotions of participants during the conference, and generates a summary video based on the results. A specific embodiment of this system will be described below.

[0273] System Overview

[0274] The system consists of the following elements:

[0275] 1. Initial Setup: The user enters the meeting setup information and adds the virtual participant (AGI) to the meeting invitation list.

[0276] 2. AGI joining: The server automatically joins the virtual participants to the meeting at the meeting start time.

[0277] 3. Data collection and analysis: Using emotion recognition engines and voice recognition technology, the content and emotions expressed during meetings are collected and analyzed in real time.

[0278] 4. Summary video generation: A summary video is generated based on the collected data and the results of sentiment analysis.

[0279] 5. Providing summary videos: Providing users with summary videos provides a means to efficiently understand the content and emotional gist of the meeting.

[0280] Specific Examples

[0281] 1. Initial Setup

[0282] A user logs into their online conference system and sets up a new conference. At this time, they input conference setting information such as the conference date and time, participant list, and invitation link, and add the generative AI (AGI) and emotion recognition engine to the invitation list as virtual participants.

[0283] 2. AGI participation

[0284] The server automatically joins the virtual participants (generative AI and emotion recognition engine) to the meeting at the scheduled meeting time. The server verifies that they are properly connected using the meeting URL or invitation link.

[0285] 3. Data collection and analysis

[0286] During the meeting, the device (AGI projection system) converts participants' speech into text in real time using speech recognition technology (e.g., Google Speech-to-Text API). It also captures presentations and materials shared on the screen by the user or other participants using image capture technology (e.g., screen capture software). Furthermore, an emotion recognition engine analyzes the content and tone of speech to recognize participants' emotions (e.g., joy, sadness, surprise, etc.) in real time (e.g., Azure Emotion API).

[0287] 4. Emotion Data Storage and Analysis

[0288] The server stores the captured speech text, image captures, and sentiment data in cloud storage (e.g., Amazon S3), analyzes the speech content and sentiment data using a machine learning model (e.g., the BERT model), and generates a summary that also reflects the results of sentiment analysis.

[0289] 5. Summary video generation

[0290] After the meeting ends, the server generates a summary based on the stored data and identifies key moments from the meeting based on the emotion data. The server then uses a video template to automatically generate a summary video that reflects the results of the summary and emotion analysis. The summary video includes graphs and highlights that visually show changes in participants' emotions.

[0291] 6. Providing a summary video

[0292] When the summary video is completed, the server sends a notification to the user, which includes a link to access the summary video. The user can click the link to watch the video. This allows the user to quickly grasp the main points of the meeting and changes in participants' emotions.

[0293] Specific examples

[0294] For example, consider the case where User B sets up a weekly reporting meeting. User B adds the generation AI and emotion recognition engine to the invitation list during the initial setup and completes the meeting setup. When the meeting starts, the server automatically adds virtual participants to the meeting, shares comments and materials during the meeting, and collects and analyzes emotional data. After the meeting ends, the server generates a summary video based on the summary and emotion analysis results and provides User B with an access link. By watching the summary video, User B can efficiently understand the key points of the meeting and changes in participants' emotions.

[0295] Prompt Sentence Examples

[0296] Below is an example prompt for asking a generative AI model to summarize a meeting:

[0297] Sample prompt: "Based on the audio data and sentiment analysis results from this meeting, please create a summary that reflects key statements and changes in sentiment."

[0298] This allows the generative AI model to generate a summary based on the meeting's audio data and emotion analysis results and provide it to the user.

[0299] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0300] Step 1:

[0301] A user logs into the online conference system and sets up a new conference. As input, they provide the date and time of the conference, a list of participants, an invitation link, and virtual participants (generative AI and emotion recognition engine). Based on this information, the system sets up the conference and outputs the necessary configuration information.

[0302] Step 2:

[0303] The server automatically joins the virtual participants (generative AI and emotion recognition engine) to the meeting at the scheduled meeting time. It uses the meeting invitation link and setting information as input to connect the virtual participants to the meeting. As output, it obtains confirmation information of the virtual participants who have joined the meeting.

[0304] Step 3:

[0305] During the meeting, the terminal (AGI projection system) converts participants' speech into text in real time using speech recognition technology (for example, Google Speech-to-Text API). It receives participants' speech data as input and generates text data using speech recognition technology. It obtains real-time speech text data as output.

[0306] Step 4:

[0307] The device captures presentations and materials shared on the screen by the user or other participants using image capture technology (e.g., screen capture software). It captures image data of the shared screen as input and creates the captured image data as output.

[0308] Step 5:

[0309] The emotion recognition engine analyzes the content and tone of speech during a meeting to recognize participants' emotions in real time. It receives speech text data and voice tone data as input, analyzes them, and generates emotion data. The real-time emotion data is obtained as output.

[0310] Step 6:

[0311] The server stores the acquired utterance text, image capture data, and emotion data in cloud storage (e.g., Amazon S3). It receives all collected data as input and performs the storage process. As output, it receives confirmation information for the data that has been saved.

[0312] Step 7:

[0313] The server analyzes the stored data and generates summaries. It uses the stored text and sentiment data as input and analyzes the data using a machine learning model (e.g., the BERT model). As output, it generates summary data based on the content and sentiment analysis of the speech.

[0314] Step 8:

[0315] The server automatically generates a summary video based on the summary data and the sentiment analysis results. It uses the summary data and the sentiment analysis results as input and applies a video template. As output, it generates a summary video containing key moments from the meeting.

[0316] Step 9:

[0317] The server sends a notification to the user when the summary video is complete. As input, it creates link information for the summary video and sends a notification message. As output, it sends a notification containing an access link to the summary video.

[0318] As a concrete example, User B sets up a weekly reporting meeting and follows the process described above to receive a summary video after the meeting. User B can quickly grasp the main points of the meeting and the emotions expressed, and can report efficiently.

[0319] (Application example 2)

[0320] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0321] With current customer support in virtual stores, it is difficult to accurately grasp customer emotions and respond quickly and appropriately. Furthermore, reviewing the content of conversations between customers and support staff requires a huge amount of time and effort, making it difficult to implement effective support follow-up. Furthermore, there is a high possibility that important moments in conversations based on emotional changes may be missed.

[0322] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving conference setting information and automatically adding virtual participants to the conference, means for collecting participant comments and projection materials in real time during the conference, means for analyzing the collected data and generating a summary, means for acquiring and analyzing emotion data of participants during the conference, means for automatically creating a short summary video based on the generated summary and emotion data, and means for providing the summary video to the user. This makes it possible to provide an effective summary of the content of the conversation and emotion data to improve the quality of customer support.

[0323] "Conference setting information" refers to detailed information for setting up a conference, such as a participant list including virtual participants, the date and time of the conference, and an invitation link.

[0324] A "virtual participant" is a virtual agent, such as an artificial intelligence or generative AI model, that participates in meetings in the same way as a human participant and plays a role in collecting and analyzing data.

[0325] "Utterances" are words or sentences spoken by participants during a meeting to communicate.

[0326] "Projected materials" are visual information such as slides, documents, and images that are shared by participants during presentations during a meeting.

[0327] "Collecting in real time" means capturing data simultaneously while the meeting is taking place.

[0328] A "summary" is information that briefly summarizes the contents based on the collected meeting data.

[0329] "Emotional data" is data that indicates the emotional state of participants, obtained by analyzing their facial expressions and tone of voice.

[0330] A "summary video" is a short video that includes important moments and key points from a meeting, and is presented visually, incorporating emotional data.

[0331] "Voice recognition technology" is a technology that analyzes voice and converts it into text data.

[0332] "Image capture technology" is a technology that captures images of materials and documents displayed on a screen and acquires them as data.

[0333] This invention is a system that improves the quality of customer support by analyzing the emotions of participants during online meetings between customers and customer support teams in customer support sessions at virtual stores and reflecting the results in summary videos.

[0334] The main components of the system are:

[0335] 1. A means of receiving conference setup information and automatically adding virtual participants to a conference:

[0336] The server receives conference setting information (participant list, conference date and time, invitation link, etc.) from a user who logs in to the virtual store's customer support system, and automatically adds an artificial intelligence model or emotion recognition engine to the conference as a virtual participant.

[0337] 2. A means of collecting participants' comments and projections in real time during a meeting:

[0338] The server converts speech during the meeting into text in real time using voice recognition technology (e.g., Google Speech-to-Text), and also uses image capture technology to obtain projected and shared materials.

[0339] 3. Means of analyzing collected data and generating summaries:

[0340] The server analyzes the collected speech text and projection materials using a machine learning model and generates a summary.

[0341] 4. Means of acquiring and analyzing participants' emotional data during a meeting:

[0342] The server uses an emotion recognition engine (e.g., Affectiva) to analyze the facial expressions and tone of voice of customers and customer support members in real time to obtain emotional data.

[0343] 5. A method for automatically creating short summary videos based on the generated summaries and emotion data:

[0344] The server automatically generates a short summary video that reflects important moments and emotional changes based on the generated summary and emotion data, and visually emphasizes the highlights using video generation software (e.g., Adobe Premiere Pro).

[0345] 6. Means of providing summary video to users:

[0346] Once the summary video is generated, the server notifies the user and provides an access link to the video, allowing the user to efficiently grasp the key points of the customer support session and the customer's emotional state.

[0347] Specific examples

[0348] As an example, consider setting up an online session in a virtual store where a customer seeks support for a defective product. When setting up the session, the user adds artificial intelligence and an emotion recognition engine as virtual participants. During the session, the server converts comments into text in real time and also collects projected materials. The emotion recognition engine analyzes the customer's emotions and obtains data. After the session ends, the server analyzes this data and generates a summary video containing the emotion analysis results. Finally, the user receives a link to the summary video and can take appropriate measures to improve customer satisfaction.

[0349] Prompt Sentence Examples

[0350] scenario:

[0351] During online sessions to respond to customer inquiries

[0352] 1. Analyze the customer's facial expressions and voice to determine their emotions.

[0353] 2. Convert what is said into text in real time.

[0354] 3. After the session, a summary video is generated showing key points and emotional changes.

[0355] Product:

[0356] A highlight summary video that visually displays customer sentiment data.

[0357] Technology used:

[0358] Emotion recognition engine, voice recognition technology, video generation software

[0359] Processing Points:

[0360] 1. Real-time monitoring of online conversations and sentiment between customers and customer support agents.

[0361] 2. Emotion data and text data are stored in cloud storage and then analyzed.

[0362] 3. Extract necessary scenes and generate a summary video that visually shows changes in emotions.

[0363] The above is a specific embodiment for carrying out the invention. This system is expected to improve the quality of customer support in virtual stores and increase customer satisfaction.

[0364] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0365] Step 1:

[0366] A user logs in to the virtual store's customer support system and sets up a new support session. The user enters meeting setting information such as the session date and time, participant list, and invitation link, and adds the generative AI model and emotion recognition engine as virtual participants. The entered meeting setting information is sent to the server, which then automatically adds the virtual participants based on this information and completes the setup.

[0367] Step 2:

[0368] The server automatically joins the virtual participants (generative AI model and emotion recognition engine) to the session at the set session time. The server uses the invitation link to properly connect, and as soon as the session starts, the system starts collecting data. The inputs are the invitation link and the session start time, and the output is the session participation status of the virtual participants.

[0369] Step 3:

[0370] During the session, the device uses speech recognition technology to convert speech into text data in real time. The input is the speech of each participant, which the server collects as text data. This text data is used in the subsequent analysis step. The output is text data.

[0371] Step 4:

[0372] The device uses image capture technology to capture the projected materials shared during the session in real time. The input is the shared materials on the screen, and the output is image data. The server stores this image data along with other data from the session.

[0373] Step 5:

[0374] The server uses an emotion recognition engine to analyze participants' facial expressions and tone of voice in real time to collect emotional data. The input is the participants' facial images and voices, and the output is emotional data. This emotional data is reflected in the generated summary video.

[0375] Step 6:

[0376] The server analyzes the collected speech text, projection materials, and emotion data using a machine learning model. The input is the aforementioned text data, image data, and emotion data, and a summary is generated through data analysis. The output is summary data.

[0377] Step 7:

[0378] The server automatically generates a short summary video based on the generated summary and emotion data. The input is the summary data and emotion data, and the output is a summary video. Important moments and emotional changes are visually emphasized using video generation software.

[0379] Step 8:

[0380] The server notifies the user of the generated summary video and provides an access link. The input is the summary video and the user's contact information, and the output is the notification and the access link to the video. The user can watch the summary video via this link and grasp the main points and emotional changes of the customer support session.

[0381] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0382] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0383] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0384] [Second embodiment]

[0385] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0386] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0387] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0388] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0389] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0390] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0391] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0392] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0393] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0394] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0395] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0396] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0397] This invention is a system that invites a generation AI to an online meeting as a virtual participant, collects and analyzes comments and projected materials during the meeting, and automatically generates a short summary video after the meeting. This system eliminates the need to record meetings and create minutes, and allows users to efficiently grasp the key points of the meeting.

[0398] Overall concept of the system

[0399] The system consists of the following elements:

[0400] 1. User Initial Setup: The user adds the generated AI to the meeting invitation list using their web conferencing account.

[0401] 2. AGI participation by the server: The server automatically joins the virtual participant (AGI) to the conference at the conference start time.

[0402] 3. Data collection and analysis using AGI: Collect and analyze statements and materials during meetings in real time.

[0403] 4. Summary video generation by the server: After the meeting ends, a summary video is generated based on the collected and analyzed data.

[0404] 5. Providing summary videos to users: Providing summary videos to users to help them understand the contents of the meeting efficiently.

[0405] Program Processing Details

[0406] User Initial Settings

[0407] Users simply log in to their web conferencing system and add the generated AI (virtual participant) to the invitation list when setting up a meeting. This process is an additional step in the normal operation of scheduling a meeting, and users can easily perform it.

[0408] Server AGI participation

[0409] When the meeting starts, the server automatically joins the AGI to the meeting. The server uses the specified meeting URL or invitation link to associate virtual participants with the meeting. At this stage, the server verifies that the AGI connection was successful.

[0410] AGI data collection and analysis

[0411] During the meeting, the AGI converts participants' speech into text data using speech recognition technology and records the speech in real time. It also captures presentations and materials shared by participants on their screens using image capture technology. The captured data is then stored in cloud storage.

[0412] Server-generated summary video

[0413] When the meeting ends, the server uses a machine learning model to generate a summary based on the data collected and analyzed by the AGI. This summary extracts the main points of the meeting. The server then automatically generates a summary video using a video template based on this summary. The generated video is then saved back to cloud storage.

[0414] Providing summary videos to users

[0415] Finally, the server notifies the user that the summary video is complete. This notification includes an access link to the summary video, which the user can click to watch. This allows the user to quickly grasp the main points of the meeting.

[0416] Specific examples

[0417] For example, User A schedules a monthly meeting and adds AGI to the invitation list. When the meeting starts, the server automatically invites AGI to join. During the meeting, AGI converts speeches into text and captures projected materials. When the meeting ends, AGI creates a summary from the collected data, and the server generates a summary video based on this summary. A few minutes later, User A receives a notification that the summary video is complete and can click the provided link to watch the video. This process allows User A to efficiently understand the main points of the meeting.

[0418] The specific embodiment of the present invention has been described above. This system allows users to quickly and efficiently grasp important information without having to go through the trouble of recording meetings and creating minutes.

[0419] The processing flow will be explained below.

[0420] Step 1:

[0421] A user logs into their own web conferencing system and sets up a new conference. At that time, they enter conference setting information such as the conference date and time, participant list, and invitation link, and add the generated AI (AGI) to the invitation list as a virtual participant.

[0422] Step 2:

[0423] The server schedules the AGI's participation based on the conference setting information received from the user. At the conference start time, the server sets a trigger to automatically join the AGI.

[0424] Step 3:

[0425] When it is time for the meeting to start, the server uses the meeting URL or invitation link to invite the AGI to join. The server verifies that the AGI has successfully connected to the meeting.

[0426] Step 4:

[0427] During the meeting, the terminal (AGI projection system) converts participants' speech into text in real time using speech recognition technology, and also captures presentations and materials shared on the screen by users and other participants using image capture technology.

[0428] Step 5:

[0429] The server stores the speech text and image capture data obtained during the meeting in cloud storage, which ensures consistent storage and access of the data.

[0430] Step 6:

[0431] Once the meeting is over, the server begins analyzing the data stored in cloud storage, using machine learning models to extract key points from the speech and materials and generate a summary text.

[0432] Step 7:

[0433] The server selects a video template based on the generated summary text and materials, and automatically generates a summary video based on the summary. The generated summary video is then saved in cloud storage.

[0434] Step 8:

[0435] Once the summary video is complete, the server sends a notification to the user, which includes a link to access the summary video, allowing the user to easily watch the video.

[0436] Step 9:

[0437] Users can click on the link provided by the server to watch the summary video, which allows them to quickly and efficiently grasp the main points of the conference.

[0438] Specific examples

[0439] As an example, let's explain the process of User A setting up a monthly meeting. In step 1, User A adds the generation AI to the invitation list and completes the meeting setup. From step 2 to step 9, the server automatically invites the AGI to the meeting and collects the content of comments and materials during the meeting. After the meeting ends, the AGI analyzes the data and generates a summary, and the server automatically generates a summary video. Finally, User A receives an access link to the summary video and can efficiently understand the key points of the meeting by watching the video. This process frees User A from the hassle of recording meetings and creating minutes.

[0440] Example 1

[0441] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0442] In today's business environment, online meetings are becoming more frequent and consume a lot of time. Recording meetings and creating minutes requires time and effort, making it difficult to efficiently grasp the information. It is also not easy to accurately summarize the content of meetings and extract only the key points. These issues increase the risk of important information being overlooked after the meeting.

[0443] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0444] In this invention, the server includes means for receiving conference setting information and automatically adding virtual participants to the conference, means for collecting participants' remarks and projection materials in real time during the conference, means for analyzing the collected data and generating a summary using a generative AI model, means for automatically creating a short summary video based on the generated summary, and means for providing the summary video to the user. This eliminates the need to record the conference and create minutes, and enables users to efficiently grasp the key points of the conference.

[0445] "Meeting setting information" refers to information necessary to set up a meeting, such as the online meeting schedule, participant list, and invitation URL.

[0446] "Virtual participants" refer to participants who are not real people, but are created using artificial intelligence such as generative AI.

[0447] "Participant Remarks" refers to oral presentations, discussions, and comments made by participants during a meeting.

[0448] "Projected materials" refers to digital materials such as presentation slides, documents, and images that are shared on the screen during a meeting.

[0449] A "generative AI model" refers to an artificial intelligence algorithm that has been trained to perform a specific task.

[0450] "Speech recognition technology" refers to technology for converting voice data into text.

[0451] "Image capture technology" refers to technology for capturing visual information from a screen or digital data.

[0452] "Cloud storage" refers to a service for storing and managing data on remote servers on the Internet.

[0453] A "summary video" is a video created to convey the main points of a meeting in a short amount of time.

[0454] A "prompt" is a textual instruction entered into a generative AI model to perform a specific task.

[0455] This invention is a system that invites a generation AI as a virtual participant in an online meeting, collects and analyzes the content of comments made during the meeting and the materials projected, and automatically generates a short summary video after the meeting ends. This invention eliminates the need to record meetings and create minutes, and allows users to efficiently grasp the key points of the meeting.

[0456] Overall concept of the system

[0457] The system consists of the following elements:

[0458] 1. User Initial Setup: The user adds the generated AI to the meeting invitation list using their web conferencing account.

[0459] 2. AGI participation by the server: The server automatically joins the virtual participant (AGI) to the conference at the conference start time.

[0460] 3. Data collection and analysis using AGI: Collect and analyze statements and materials during meetings in real time.

[0461] 4. Generation of summary video by the server: After the meeting ends, a summary video is generated based on the collected and analyzed data.

[0462] 5. Providing users with video summaries: Providing users with video summaries allows them to efficiently understand the content of the meeting.

[0463] Program Processing Details

[0464] User Initial Settings

[0465] The user logs in to their own web conferencing system (e.g., a general web conferencing system) and invites the generated AI (virtual participant) when setting up a conference. On the setting screen, they enter the generated AI's email address or ID and save it. This completes the setting for the generated AI to automatically join the conference.

[0466] Server AGI participation

[0467] As the meeting starts, the server automatically joins the AGI configured as a virtual participant to the meeting. The server connects the AGI using the meeting URL or invitation link provided by the user. At this stage, the server verifies that the AGI has successfully joined the meeting.

[0468] AGI data collection and analysis

[0469] When a meeting begins, the AGI performs the following tasks:

[0470] Speech Recognition: Use speech recognition technology (e.g., Google Speech-to-Text API, common speech recognition technology) to convert participants' speech into text in real time.

[0471] Document capture: Capture presentations and documents shared by participants on their screens using image capture technology (e.g., common image processing technology).

[0472] The acquired data is stored in cloud storage (e.g., a general cloud storage service).

[0473] Server-based generation of summary videos

[0474] After the meeting ends, the server generates a summary based on the data collected and analyzed by the AGI. To do this, it uses a generative AI model (e.g., a general machine learning model). The summary extracts the main points of the meeting, and the server automatically generates a summary video based on this summary using a video template (e.g., general video editing software). The generated video is then saved back to cloud storage.

[0475] Providing users with video summaries

[0476] The server notifies the user that the summary video is complete. This notification includes an access link to the summary video, which the user can view by clicking the link. This allows the user to quickly grasp the main points of the meeting.

[0477] Specific examples

[0478] For example, User A schedules a monthly meeting and adds a generated AI (virtual participant) to the invitation list. When the meeting starts, the server automatically invites the AGI to the meeting, where it performs real-time speech recognition and captures projected materials. When the meeting ends, the server generates a summary based on the collected data and creates a summary video based on the content. Finally, User A receives a notification a few minutes later that the summary video is ready, and can click the provided link to watch the video. This allows User A to quickly and efficiently understand the key points of the meeting.

[0479] Prompt Sentence Examples

[0480] Here are some example prompts for a generative AI model to generate a summary:

[0481] Please summarize the following meeting in a concise manner, extracting the main points and conclusions.

[0482] 1. Speaker A: "Today's meeting agenda is as follows..."

[0483] 2. Speaker B: "Regarding the budget review..."

[0484] 3. Speaker C: "I'll report on the progress of the following project..."

[0485] [Detailed minutes of the meeting follow]

[0486] Please summarize the main points and conclusions of this meeting in bullet points.

[0487] The specific embodiment of the present invention has been described above. This system allows users to quickly and efficiently grasp important information without having to go through the trouble of recording meetings and creating minutes.

[0488] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0489] Step 1: User Initial Setup

[0490] The user logs in to their own web conferencing system and schedules a meeting. At this time, they add the generated AI as a virtual participant. Specifically, they enter the generated AI's email address or ID on the meeting setting screen and add it to the invitation list. The input is the meeting setting information, and the output is the schedule data for the meeting to which the virtual participant has been added.

[0491] Step 2: Server joins AGI

[0492] As the start time of the conference approaches, the server automatically invites the generated AI (AGI) to join the conference. The server obtains the conference URL or invitation link based on the conference information set by the user and connects the AGI to that URL. This allows the AGI to join the conference. The input is the set conference URL and AGI connection information, and the output is status information indicating that the AGI has successfully joined the conference.

[0493] Step 3: AGI collects data

[0494] When a meeting begins, the AGI collects speech and projections in real time. Specifically, it performs the following processes:

[0495] 1. Speech recognition: Converting what is being said in a meeting into text data in real time, using tools such as the Google Speech-to-Text API.

[0496] 2. Document capture: Slides and documents shared during meetings are captured using image capture technology, and their contents are then analyzed and converted into text data using OCR (Optical Character Recognition) technology.

[0497] The input is the conference audio and shared screen data, and the output is the textual utterance data and captured material data.

[0498] Step 4: Save your data

[0499] The collected text data and image data are stored in cloud storage by the server. Data is stored using a common cloud storage service (e.g., Amazon S3). The input is the data collected by the AGI, and the output is the data stored in cloud storage.

[0500] Step 5: Generate a summary

[0501] After the meeting ends, the server analyzes the data stored in cloud storage and generates a summary using a generative AI model. Specifically, it uses a machine learning model (e.g., GPT-4) to extract the main points of the meeting based on the prompt. The input is the saved meeting data and the prompt, and the output is the summary text.

[0502] Step 6: Generate summary video

[0503] Based on the generated summary text, the server uses a video template to create a summary video. Using video editing software (e.g., general video editing software), a video that integrates text, audio narration, and images is automatically generated. The input is the summary text and an existing video template, and the output is the completed summary video.

[0504] Step 7: Provide the summary video to the user

[0505] The server notifies the user that the summary video is complete. The notification includes an access link to the summary video, which the user can click to watch. The input is the generated summary video, and the output is the link to provide the summary video and a notification message.

[0506] This series of processes allows users to efficiently grasp important information without having to go through the trouble of recording meetings and creating minutes.

[0507] (Application example 1)

[0508] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0509] In the operation management of autonomous vehicles, it is difficult to efficiently grasp the operation status and important events of multiple vehicles. In particular, as a large amount of data is generated from camera footage and various sensors during operation, operation managers are required to quickly and accurately analyze this data and make appropriate decisions. However, current technology lacks the means to efficiently collect and analyze this data and provide a summary of the operation status.

[0510] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0511] In this invention, the server includes means for receiving conference setting information and automatically adding virtual participants to the conference, means for collecting participant remarks and projection materials in real time during the conference, means for analyzing camera and sensor data collected while the vehicle is in operation to generate a summary, means for automatically creating a short summary video based on the generated summary, and means for providing the summary video to the user. This enables the operation manager of the autonomous vehicle to quickly and accurately grasp important events and driving conditions during operation and take prompt action.

[0512] "Conference setup information" is information necessary to automatically add virtual participants to a conference.

[0513] "Virtual participants" refers to generative AI that automatically participates in meetings and car rides, collecting and analyzing data.

[0514] The "means for automatically adding a virtual participant to a conference" is a method for automatically adding a virtual participant to a conference when the conference starts.

[0515] "Means of collecting information in real time" refers to methods of collecting information such as speech content, projected materials, camera footage, and sensor data on the spot during meetings or while driving a car.

[0516] A "camera" is an image capturing device mounted on a vehicle that records images while the vehicle is in motion.

[0517] A "sensor" is a data acquisition device installed in a vehicle that detects and records various physical information during operation.

[0518] "Means for analyzing and generating a summary" refers to a method for analyzing collected data and concisely summarizing its contents.

[0519] The "means for automatically creating a short summary video" is a means for automatically creating a short video of a meeting or operation status based on the summarized content.

[0520] The "means for providing to the user" refers to a method for providing the summary video in a form that allows the user to easily access it.

[0521] "Speech recognition technology" is a technology that converts voice into text data.

[0522] "Image capture technology" is a technology that captures projected materials or images as still images.

[0523] "Cloud storage" is a storage service for saving and managing data on the Internet.

[0524] "Means for backing up stored data" refers to a method for replicating data stored in cloud storage on a regular or ad-hoc basis to prepare for data loss.

[0525] This invention is a data collection and analysis system for managing the operation of autonomous vehicles. It analyzes camera footage and sensor data collected from the vehicle during operation, summarizes the results, and generates and provides a short summary video, allowing the operation manager to quickly understand the situation. Specific system configurations for implementing this invention are described below.

[0526] Overall concept of the system

[0527] The system consists of the following main components:

[0528] Camera image collection module (installed in autonomous vehicles)

[0529] Sensor data collection module (installed in autonomous vehicles)

[0530] Data analysis server

[0531] Generative AI Models

[0532] Summary video generation module

[0533] Cloud Storage

[0534] User Interface

[0535] Camera footage and sensor data collection

[0536] Autonomous vehicles are equipped with multiple cameras and sensors. These devices collect video and sensor data in real time while driving. The cameras record the vehicle's surroundings as video, while the sensors detect various data such as speed, position, acceleration, and the presence or absence of obstacles.

[0537] Data analysis server

[0538] The data analysis server receives and analyzes data collected from autonomous vehicles in real time. A generative AI model is used to analyze the collected camera footage and sensor data to generate summaries. For example, it generates summaries that explain traffic conditions and the occurrence of important events (such as sudden braking or sudden stops).

[0539] Summary video generation

[0540] The summary video generation module generates a summary video based on the analyzed data. The summary video includes images showing important events and situations during operation, along with related text information. This allows users to quickly understand the operation status.

[0541] Cloud storage and user provisioning

[0542] The generated summary video is stored in cloud storage, allowing the dispatcher to easily access it from anywhere. Through the user interface, the dispatcher can access the summary video and quickly obtain the information they need.

[0543] Specific examples

[0544] For example, when an autonomous vehicle passes through multiple intersections, it detects traffic conditions and events that occur at the intersections (such as a sudden vehicle stop or the appearance of an unexpected obstacle) from camera footage and sensor data. The generative AI model analyzes this data in real time and summarizes important points such as "A sudden stop occurred at intersection 1" or "A pedestrian ran out into the road at intersection 2" into text. Next, the summary video generation module creates a short summary video based on the summary and saves it in cloud storage. The operation manager can view this summary video via the user interface.

[0545] Examples of prompt statements

[0546] Example prompt for a generative AI model:

[0547] "Please summarize any significant events or notable operational findings from this camera footage and sensor data."

[0548] Such a system allows operation managers to quickly grasp key points extracted from large amounts of data and respond promptly.

[0549] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0550] Step 1:

[0551] Initial Setup

[0552] The server prepares to add virtual participants (generative AI models) while each vehicle is in operation based on the operation schedule and setting information of the autonomous vehicle received from the operation manager. The input is operation schedule information, and the output is a notification that the virtual participants have been set up. The server saves the setting information so that the virtual participants can automatically join the operation when the operation starts.

[0553] Step 2:

[0554] Starting Data Collection

[0555] Once the vehicle begins operation, the cameras and sensors installed on the vehicle begin collecting data in real time. The cameras record images of the surrounding area, and the sensors acquire data such as speed, position, acceleration, and obstacles. The input is the camera images and sensor data, and the output is the collected data. The server periodically receives this data and prepares it for analysis.

[0556] Step 3:

[0557] Data analysis

[0558] The server receives camera footage and sensor data transmitted sequentially during operation and analyzes it using the generative AI model. The input is the camera footage and sensor data collected in step 2, and the output is the analyzed summary information. The server sends prompts to the generative AI model, asking for the following to summarize important events and situations:

[0559] "Please summarize any significant events or notable operational findings from this camera footage and sensor data."

[0560] Step 4:

[0561] Summary video generation

[0562] The server generates a summary video based on the summary information obtained in step 3. The summary video generation module combines the summarized text information with related video to create a video that can be viewed in a short time. The input is the analyzed summary information, and the output is a summary video. The server saves the generated video in cloud storage.

[0563] Step 5:

[0564] Providing a summary video

[0565] After the generated summary video is saved in cloud storage, the server sends a notification to the operations manager. The notification includes an access link to the video, which the manager can click to view. The input is the summary video information saved in cloud storage, and the output is a notification sent to the operations manager. By watching the summary video, the operations manager can instantly understand important events and situations that occurred during operations.

[0566] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0567] The present invention is a system that combines an emotion recognition engine with an online conference, and provides a means to understand the main points of the conference more deeply by analyzing the emotions of participants during the conference and reflecting the results in a summary video. This system is implemented by the following main steps.

[0568] Overall concept of the system

[0569] The system consists of the following elements:

[0570] 1. User Initial Setup: The user adds the generative AI and emotion recognition engine to the meeting invitation list using their web conferencing account.

[0571] 2. AGI participation by the server: The server automatically joins the virtual participant (AGI) to the conference at the conference start time.

[0572] 3. Data collection and analysis: Using an emotion recognition engine and voice recognition technology, the content of statements made during meetings and the emotions of participants are collected and analyzed in real time.

[0573] 4. Summary video generation: A summary video is generated based on the collected data and the results of sentiment analysis.

[0574] 5. Providing users with video summary: Providing users with video summary provides an efficient way to understand the main points of the meeting and its emotions.

[0575] Program Processing Details

[0576] User Initial Settings

[0577] A user logs into their own web conferencing system and sets up a new meeting. At that time, they input meeting setting information such as the meeting date and time, participant list, and invitation link, and add the generative AI (AGI) and emotion recognition engine to the invitation list as virtual participants.

[0578] Participation of AGI and emotion recognition engine by server

[0579] The server automatically joins the virtual participants (AGI and emotion recognition engine) to the meeting at the scheduled meeting time. It uses the meeting URL or invitation link to ensure they are properly connected.

[0580] Data collection and analysis

[0581] During the meeting, the device (AGI projection system) converts participants' speech into text in real time using speech recognition technology. It also captures presentations and materials shared on the screen by the user and other participants using image capture technology. Furthermore, the emotion recognition engine analyzes the content and tone of speech to recognize participants' emotions in real time.

[0582] Emotion data storage and analysis

[0583] The server stores the captured text, image captures, and sentiment data in cloud storage, and uses a machine learning model to analyze the content and sentiment data and generate a summary that also reflects the results of sentiment analysis.

[0584] Summary video generation

[0585] After the meeting ends, the server generates a summary based on the stored data and identifies key moments from the meeting based on the emotion data. The server then uses a video template to automatically generate a summary video that reflects the results of the summary and emotion analysis. The summary video includes graphs and highlights that visually show changes in participants' emotions.

[0586] Providing summary videos to users

[0587] When the summary video is completed, the server sends a notification to the user, which includes a link to access the summary video. The user can click the link to watch the video. This allows the user to quickly grasp the main points of the meeting and changes in participants' emotions.

[0588] Specific examples

[0589] Let's take the example of User B setting up a weekly report meeting. In Step 1, User B adds the generation AI and emotion recognition engine to the invitation list and completes the meeting setup. From Step 2 to Step 6, the server automatically adds virtual participants to the meeting and collects and analyzes comments, materials, and emotion data from the meeting. After the meeting ends, the server generates a summary video based on the summary and emotion analysis results. Finally, User B receives a link to access the summary video and can watch the meeting highlights, which reflect the emotion data, to efficiently understand the key points of the meeting and changes in participants' emotions. This process frees User B from the hassle of recording meetings and creating minutes, and also makes it easier for them to understand participants' emotions.

[0590] The above describes a specific embodiment of the present invention. This system allows users to save the time and effort of recording meetings and creating minutes, and enables them to efficiently grasp not only important information but also changes in the emotions of participants.

[0591] The processing flow will be explained below.

[0592] Step 1:

[0593] A user logs into their own web conferencing system and sets up a new online meeting. At this time, they input information such as the meeting date and time, participant list, and invitation link, and add the generative AI (AGI) and emotion recognition engine to the invitation list as virtual participants.

[0594] Step 2:

[0595] When the meeting starts, the server automatically joins the AGI and emotion recognition engine to the meeting, which involves connecting using the meeting URL or invitation link.

[0596] Step 3:

[0597] During the meeting, the terminal (AGI projection system) converts participants' speech into text in real time using voice recognition technology, and captures presentations and materials using image capture technology.

[0598] Step 4:

[0599] Similarly, during a meeting, the emotion recognition engine analyzes the content and tone of speech to identify participants' emotions in real time, and this emotional data is instantly saved as digital data.

[0600] Step 5:

[0601] After the meeting ends, the server stores the data collected by the AGI and emotion recognition engine (speech text, document capture, and emotion analysis data) in cloud storage.

[0602] Step 6:

[0603] The server analyzes the stored data and generates a summary using the speech content and emotion data, which includes important speech content and emotional changes.

[0604] Step 7:

[0605] The server automatically generates a summary video using a video template based on the generated summary and the results of sentiment analysis, highlighting key points and important emotional moments from the meeting.

[0606] Step 8:

[0607] Once the summary video is complete, the server sends a notification to the user, which includes a link to access the summary video.

[0608] Step 9:

[0609] Users receive a notification and click the provided link to watch the summary video, which allows them to quickly grasp the key points of the meeting and the changes in participants' emotions.

[0610] Specific examples

[0611] For example, if User B schedules a weekly reporting meeting, in Step 1, the generation AI and emotion recognition engine are added to the meeting. When the meeting begins, in Steps 2 to 4, the AGI and emotion recognition engine automatically join the meeting and collect the remarks, materials, and emotion data. After the meeting ends, in Steps 5 to 7, the server generates a summary and automatically generates a summary video that reflects the emotion data. Finally, in Steps 8 and 9, User B can watch the summary video and understand the highlights of the meeting that reflect the emotion data. This process frees User B from the hassle of recording meetings and creating minutes, and also makes it easier to understand the emotions of the participants.

[0612] Example 2

[0613] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0614] While current online meeting systems allow participants to share remarks and materials during meetings, it is difficult to efficiently grasp participants' emotions and key moments. Furthermore, current systems do not automatically generate summary videos that include meeting summaries or emotion analysis results. This makes it difficult to effectively follow up on meetings and grasp key points, and reviewing the content of long meetings afterward is particularly time-consuming and labor-intensive.

[0615] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving conference setting information and automatically adding virtual participants to the conference, means for collecting participants' comments and projection materials in real time during the conference, means for analyzing the collected comment content and emotion data to generate a summary, means for automatically creating a short summary video based on the generated summary and emotion analysis results, and means for providing the summary video to the user. This makes it possible to efficiently grasp the content of the conference and changes in participants' emotions, and to quickly understand the gist of important information and emotions.

[0616] "Conference setting information" is information necessary to hold a conference, such as the date and time of the online conference, a list of participants, and an invitation link.

[0617] A "virtual participant" is an agent or program that is not physically present but is brought into an online conference by the system.

[0618] "Speech content" refers to voice data spoken by participants during an online conference that has been converted into text using voice recognition technology.

[0619] "Projected materials" are visual materials such as presentations and documents that participants share during an online meeting.

[0620] "Means of collection" refers to technologies and systems for acquiring participants' comments and projected materials in real time during online meetings.

[0621] "Speech recognition technology" is a technology for converting voice data into text data.

[0622] "Image capture technology" is a technology that captures displayed visual materials as image data.

[0623] "Emotional data" is emotional information analyzed based on the content of participants' statements and tone of voice.

[0624] A "summary" is a concise summary of the main points of a meeting based on collected speech and sentiment data.

[0625] A "summary video" is a short video generated from collected data that visually shows key moments of the meeting and the results of sentiment analysis.

[0626] "Cloud storage" is an external storage service for storing and managing data via the Internet.

[0627] "Backup measures" are techniques and methods for regularly replicating and backing up stored data.

[0628] The present invention is a system that combines an online conference with an emotion recognition engine, analyzes the content of comments and emotions of participants during the conference, and generates a summary video based on the results. A specific embodiment of this system will be described below.

[0629] System Overview

[0630] The system consists of the following elements:

[0631] 1. Initial Setup: The user enters the meeting setup information and adds the virtual participant (AGI) to the meeting invitation list.

[0632] 2. AGI joining: The server automatically joins the virtual participants to the meeting at the meeting start time.

[0633] 3. Data collection and analysis: Using emotion recognition engines and voice recognition technology, the content and emotions expressed during meetings are collected and analyzed in real time.

[0634] 4. Summary video generation: A summary video is generated based on the collected data and the results of sentiment analysis.

[0635] 5. Providing summary videos: Providing users with summary videos provides a means to efficiently understand the content and emotional gist of the meeting.

[0636] Specific Examples

[0637] 1. Initial Setup

[0638] A user logs into their online conference system and sets up a new conference. At this time, they input conference setting information such as the conference date and time, participant list, and invitation link, and add the generative AI (AGI) and emotion recognition engine to the invitation list as virtual participants.

[0639] 2. AGI participation

[0640] The server automatically joins the virtual participants (generative AI and emotion recognition engine) to the meeting at the scheduled meeting time. The server verifies that they are properly connected using the meeting URL or invitation link.

[0641] 3. Data collection and analysis

[0642] During the meeting, the device (AGI projection system) converts participants' speech into text in real time using speech recognition technology (e.g., Google Speech-to-Text API). It also captures presentations and materials shared on the screen by the user or other participants using image capture technology (e.g., screen capture software). Furthermore, an emotion recognition engine analyzes the content and tone of speech to recognize participants' emotions (e.g., joy, sadness, surprise, etc.) in real time (e.g., Azure Emotion API).

[0643] 4. Emotion Data Storage and Analysis

[0644] The server stores the captured speech text, image captures, and sentiment data in cloud storage (e.g., Amazon S3), analyzes the speech content and sentiment data using a machine learning model (e.g., the BERT model), and generates a summary that also reflects the results of sentiment analysis.

[0645] 5. Summary video generation

[0646] After the meeting ends, the server generates a summary based on the stored data and identifies key moments from the meeting based on the emotion data. The server then uses a video template to automatically generate a summary video that reflects the results of the summary and emotion analysis. The summary video includes graphs and highlights that visually show changes in participants' emotions.

[0647] 6. Providing a summary video

[0648] When the summary video is completed, the server sends a notification to the user, which includes a link to access the summary video. The user can click the link to watch the video. This allows the user to quickly grasp the main points of the meeting and changes in participants' emotions.

[0649] Specific examples

[0650] For example, consider the case where User B sets up a weekly reporting meeting. User B adds the generation AI and emotion recognition engine to the invitation list during the initial setup and completes the meeting setup. When the meeting starts, the server automatically adds virtual participants to the meeting, shares comments and materials during the meeting, and collects and analyzes emotional data. After the meeting ends, the server generates a summary video based on the summary and emotion analysis results and provides User B with an access link. By watching the summary video, User B can efficiently understand the key points of the meeting and changes in participants' emotions.

[0651] Prompt Sentence Examples

[0652] Below is an example prompt for asking a generative AI model to summarize a meeting:

[0653] Sample prompt: "Based on the audio data and sentiment analysis results from this meeting, please create a summary that reflects key statements and changes in sentiment."

[0654] This allows the generative AI model to generate a summary based on the meeting's audio data and emotion analysis results and provide it to the user.

[0655] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0656] Step 1:

[0657] A user logs into the online conference system and sets up a new conference. As input, they provide the date and time of the conference, a list of participants, an invitation link, and virtual participants (generative AI and emotion recognition engine). Based on this information, the system sets up the conference and outputs the necessary configuration information.

[0658] Step 2:

[0659] The server automatically joins the virtual participants (generative AI and emotion recognition engine) to the meeting at the scheduled meeting time. It uses the meeting invitation link and setting information as input to connect the virtual participants to the meeting. As output, it obtains confirmation information of the virtual participants who have joined the meeting.

[0660] Step 3:

[0661] During the meeting, the terminal (AGI projection system) converts participants' speech into text in real time using speech recognition technology (for example, Google Speech-to-Text API). It receives participants' speech data as input and generates text data using speech recognition technology. It obtains real-time speech text data as output.

[0662] Step 4:

[0663] The device captures presentations and materials shared on the screen by the user or other participants using image capture technology (e.g., screen capture software). It captures image data of the shared screen as input and creates the captured image data as output.

[0664] Step 5:

[0665] The emotion recognition engine analyzes the content and tone of speech during a meeting to recognize participants' emotions in real time. It receives speech text data and voice tone data as input, analyzes them, and generates emotion data. The real-time emotion data is obtained as output.

[0666] Step 6:

[0667] The server stores the acquired utterance text, image capture data, and emotion data in cloud storage (e.g., Amazon S3). It receives all collected data as input and performs the storage process. As output, it receives confirmation information for the data that has been saved.

[0668] Step 7:

[0669] The server analyzes the stored data and generates summaries. It uses the stored text and sentiment data as input and analyzes the data using a machine learning model (e.g., the BERT model). As output, it generates summary data based on the content and sentiment analysis of the speech.

[0670] Step 8:

[0671] The server automatically generates a summary video based on the summary data and the sentiment analysis results. It uses the summary data and the sentiment analysis results as input and applies a video template. As output, it generates a summary video containing key moments from the meeting.

[0672] Step 9:

[0673] The server sends a notification to the user when the summary video is complete. As input, it creates link information for the summary video and sends a notification message. As output, it sends a notification containing an access link to the summary video.

[0674] As a concrete example, User B sets up a weekly reporting meeting and follows the process described above to receive a summary video after the meeting. User B can quickly grasp the main points of the meeting and the emotions expressed, and can report efficiently.

[0675] (Application example 2)

[0676] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0677] With current customer support in virtual stores, it is difficult to accurately grasp customer emotions and respond quickly and appropriately. Furthermore, reviewing the content of conversations between customers and support staff requires a huge amount of time and effort, making it difficult to implement effective support follow-up. Furthermore, there is a high possibility that important moments in conversations based on emotional changes may be missed.

[0678] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving conference setting information and automatically adding virtual participants to the conference, means for collecting participant comments and projection materials in real time during the conference, means for analyzing the collected data and generating a summary, means for acquiring and analyzing emotion data of participants during the conference, means for automatically creating a short summary video based on the generated summary and emotion data, and means for providing the summary video to the user. This makes it possible to provide an effective summary of the content of the conversation and emotion data to improve the quality of customer support.

[0679] "Conference setting information" refers to detailed information for setting up a conference, such as a participant list including virtual participants, the date and time of the conference, and an invitation link.

[0680] A "virtual participant" is a virtual agent, such as an artificial intelligence or generative AI model, that participates in meetings in the same way as a human participant and plays a role in collecting and analyzing data.

[0681] "Utterances" are words or sentences spoken by participants during a meeting to communicate.

[0682] "Projected materials" are visual information such as slides, documents, and images that are shared by participants during presentations during a meeting.

[0683] "Collecting in real time" means capturing data simultaneously while the meeting is taking place.

[0684] A "summary" is information that briefly summarizes the contents based on the collected meeting data.

[0685] "Emotional data" is data that indicates the emotional state of participants, obtained by analyzing their facial expressions and tone of voice.

[0686] A "summary video" is a short video that includes important moments and key points from a meeting, and is presented visually, incorporating emotional data.

[0687] "Voice recognition technology" is a technology that analyzes voice and converts it into text data.

[0688] "Image capture technology" is a technology that captures images of materials and documents displayed on a screen and acquires them as data.

[0689] This invention is a system that improves the quality of customer support by analyzing the emotions of participants during online meetings between customers and customer support teams in customer support sessions at virtual stores and reflecting the results in summary videos.

[0690] The main components of the system are:

[0691] 1. A means of receiving conference setup information and automatically adding virtual participants to a conference:

[0692] The server receives conference setting information (participant list, conference date and time, invitation link, etc.) from a user who logs in to the virtual store's customer support system, and automatically adds an artificial intelligence model or emotion recognition engine to the conference as a virtual participant.

[0693] 2. A means of collecting participants' comments and projections in real time during a meeting:

[0694] The server converts speech during the meeting into text in real time using voice recognition technology (e.g., Google Speech-to-Text), and also uses image capture technology to obtain projected and shared materials.

[0695] 3. Means of analyzing collected data and generating summaries:

[0696] The server analyzes the collected speech text and projection materials using a machine learning model and generates a summary.

[0697] 4. Means of acquiring and analyzing participants' emotional data during a meeting:

[0698] The server uses an emotion recognition engine (e.g., Affectiva) to analyze the facial expressions and tone of voice of customers and customer support members in real time to obtain emotional data.

[0699] 5. A method for automatically creating short summary videos based on the generated summaries and emotion data:

[0700] The server automatically generates a short summary video that reflects important moments and emotional changes based on the generated summary and emotion data, and visually emphasizes the highlights using video generation software (e.g., Adobe Premiere Pro).

[0701] 6. Means of providing summary video to users:

[0702] Once the summary video is generated, the server notifies the user and provides an access link to the video, allowing the user to efficiently grasp the key points of the customer support session and the customer's emotional state.

[0703] Specific examples

[0704] As an example, consider setting up an online session in a virtual store where a customer seeks support for a defective product. When setting up the session, the user adds artificial intelligence and an emotion recognition engine as virtual participants. During the session, the server converts comments into text in real time and also collects projected materials. The emotion recognition engine analyzes the customer's emotions and obtains data. After the session ends, the server analyzes this data and generates a summary video containing the emotion analysis results. Finally, the user receives a link to the summary video and can take appropriate measures to improve customer satisfaction.

[0705] Prompt Sentence Examples

[0706] scenario:

[0707] During online sessions to respond to customer inquiries

[0708] 1. Analyze the customer's facial expressions and voice to determine their emotions.

[0709] 2. Convert what is said into text in real time.

[0710] 3. After the session, a summary video is generated showing key points and emotional changes.

[0711] Product:

[0712] A highlight summary video that visually displays customer sentiment data.

[0713] Technology used:

[0714] Emotion recognition engine, voice recognition technology, video generation software

[0715] Processing Points:

[0716] 1. Real-time monitoring of online conversations and sentiment between customers and customer support agents.

[0717] 2. Emotion data and text data are stored in cloud storage and then analyzed.

[0718] 3. Extract necessary scenes and generate a summary video that visually shows changes in emotions.

[0719] The above is a specific embodiment for carrying out the invention. This system is expected to improve the quality of customer support in virtual stores and increase customer satisfaction.

[0720] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0721] Step 1:

[0722] A user logs in to the virtual store's customer support system and sets up a new support session. The user enters meeting setting information such as the session date and time, participant list, and invitation link, and adds the generative AI model and emotion recognition engine as virtual participants. The entered meeting setting information is sent to the server, which then automatically adds the virtual participants based on this information and completes the setup.

[0723] Step 2:

[0724] The server automatically joins the virtual participants (generative AI model and emotion recognition engine) to the session at the set session time. The server uses the invitation link to properly connect, and as soon as the session starts, the system starts collecting data. The inputs are the invitation link and the session start time, and the output is the session participation status of the virtual participants.

[0725] Step 3:

[0726] During the session, the device uses speech recognition technology to convert speech into text data in real time. The input is the speech of each participant, which the server collects as text data. This text data is used in the subsequent analysis step. The output is text data.

[0727] Step 4:

[0728] The device uses image capture technology to capture the projected materials shared during the session in real time. The input is the shared materials on the screen, and the output is image data. The server stores this image data along with other data from the session.

[0729] Step 5:

[0730] The server uses an emotion recognition engine to analyze participants' facial expressions and tone of voice in real time to collect emotional data. The input is the participants' facial images and voices, and the output is emotional data. This emotional data is reflected in the generated summary video.

[0731] Step 6:

[0732] The server analyzes the collected speech text, projection materials, and emotion data using a machine learning model. The input is the aforementioned text data, image data, and emotion data, and a summary is generated through data analysis. The output is summary data.

[0733] Step 7:

[0734] The server automatically generates a short summary video based on the generated summary and emotion data. The input is the summary data and emotion data, and the output is a summary video. Important moments and emotional changes are visually emphasized using video generation software.

[0735] Step 8:

[0736] The server notifies the user of the generated summary video and provides an access link. The input is the summary video and the user's contact information, and the output is the notification and the access link to the video. The user can watch the summary video via this link and grasp the main points and emotional changes of the customer support session.

[0737] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0738] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0739] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0740] [Third embodiment]

[0741] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0742] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0743] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0744] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0745] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0746] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0747] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0748] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0749] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0750] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0751] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0752] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0753] This invention is a system that invites a generation AI to an online meeting as a virtual participant, collects and analyzes comments and projected materials during the meeting, and automatically generates a short summary video after the meeting. This system eliminates the need to record meetings and create minutes, and allows users to efficiently grasp the key points of the meeting.

[0754] Overall concept of the system

[0755] The system consists of the following elements:

[0756] 1. User Initial Setup: The user adds the generated AI to the meeting invitation list using their web conferencing account.

[0757] 2. AGI participation by the server: The server automatically joins the virtual participant (AGI) to the conference at the conference start time.

[0758] 3. Data collection and analysis using AGI: Collect and analyze statements and materials during meetings in real time.

[0759] 4. Summary video generation by the server: After the meeting ends, a summary video is generated based on the collected and analyzed data.

[0760] 5. Providing summary videos to users: Providing summary videos to users to help them understand the contents of the meeting efficiently.

[0761] Program Processing Details

[0762] User Initial Settings

[0763] Users simply log in to their web conferencing system and add the generated AI (virtual participant) to the invitation list when setting up a meeting. This process is an additional step in the normal operation of scheduling a meeting, and users can easily perform it.

[0764] Server AGI participation

[0765] When the meeting starts, the server automatically joins the AGI to the meeting. The server uses the specified meeting URL or invitation link to associate virtual participants with the meeting. At this stage, the server verifies that the AGI connection was successful.

[0766] AGI data collection and analysis

[0767] During the meeting, the AGI converts participants' speech into text data using speech recognition technology and records the speech in real time. It also captures presentations and materials shared by participants on their screens using image capture technology. The captured data is then stored in cloud storage.

[0768] Server-generated summary video

[0769] When the meeting ends, the server uses a machine learning model to generate a summary based on the data collected and analyzed by the AGI. This summary extracts the main points of the meeting. The server then automatically generates a summary video using a video template based on this summary. The generated video is then saved back to cloud storage.

[0770] Providing summary videos to users

[0771] Finally, the server notifies the user that the summary video is complete. This notification includes an access link to the summary video, which the user can click to watch. This allows the user to quickly grasp the main points of the meeting.

[0772] Specific examples

[0773] For example, User A schedules a monthly meeting and adds AGI to the invitation list. When the meeting starts, the server automatically invites AGI to join. During the meeting, AGI converts speeches into text and captures projected materials. When the meeting ends, AGI creates a summary from the collected data, and the server generates a summary video based on this summary. A few minutes later, User A receives a notification that the summary video is complete and can click the provided link to watch the video. This process allows User A to efficiently understand the main points of the meeting.

[0774] The specific embodiment of the present invention has been described above. This system allows users to quickly and efficiently grasp important information without having to go through the trouble of recording meetings and creating minutes.

[0775] The processing flow will be explained below.

[0776] Step 1:

[0777] A user logs into their own web conferencing system and sets up a new conference. At that time, they enter conference setting information such as the conference date and time, participant list, and invitation link, and add the generated AI (AGI) to the invitation list as a virtual participant.

[0778] Step 2:

[0779] The server schedules the AGI's participation based on the conference setting information received from the user. At the conference start time, the server sets a trigger to automatically join the AGI.

[0780] Step 3:

[0781] When it is time for the meeting to start, the server uses the meeting URL or invitation link to invite the AGI to join. The server verifies that the AGI has successfully connected to the meeting.

[0782] Step 4:

[0783] During the meeting, the terminal (AGI projection system) converts participants' speech into text in real time using speech recognition technology, and also captures presentations and materials shared on the screen by users and other participants using image capture technology.

[0784] Step 5:

[0785] The server stores the speech text and image capture data obtained during the meeting in cloud storage, which ensures consistent storage and access of the data.

[0786] Step 6:

[0787] Once the meeting is over, the server begins analyzing the data stored in cloud storage, using machine learning models to extract key points from the speech and materials and generate a summary text.

[0788] Step 7:

[0789] The server selects a video template based on the generated summary text and materials, and automatically generates a summary video based on the summary. The generated summary video is then saved in cloud storage.

[0790] Step 8:

[0791] Once the summary video is complete, the server sends a notification to the user, which includes a link to access the summary video, allowing the user to easily watch the video.

[0792] Step 9:

[0793] Users can click on the link provided by the server to watch the summary video, which allows them to quickly and efficiently grasp the main points of the conference.

[0794] Specific examples

[0795] As an example, let's explain the process of User A setting up a monthly meeting. In step 1, User A adds the generation AI to the invitation list and completes the meeting setup. From step 2 to step 9, the server automatically invites the AGI to the meeting and collects the content of comments and materials during the meeting. After the meeting ends, the AGI analyzes the data and generates a summary, and the server automatically generates a summary video. Finally, User A receives an access link to the summary video and can efficiently understand the key points of the meeting by watching the video. This process frees User A from the hassle of recording meetings and creating minutes.

[0796] Example 1

[0797] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0798] In today's business environment, online meetings are becoming more frequent and consume a lot of time. Recording meetings and creating minutes requires time and effort, making it difficult to efficiently grasp the information. It is also not easy to accurately summarize the content of meetings and extract only the key points. These issues increase the risk of important information being overlooked after the meeting.

[0799] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0800] In this invention, the server includes means for receiving conference setting information and automatically adding virtual participants to the conference, means for collecting participants' remarks and projection materials in real time during the conference, means for analyzing the collected data and generating a summary using a generative AI model, means for automatically creating a short summary video based on the generated summary, and means for providing the summary video to the user. This eliminates the need to record the conference and create minutes, and enables users to efficiently grasp the key points of the conference.

[0801] "Meeting setting information" refers to information necessary to set up a meeting, such as the online meeting schedule, participant list, and invitation URL.

[0802] "Virtual participants" refer to participants who are not real people, but are created using artificial intelligence such as generative AI.

[0803] "Participant Remarks" refers to oral presentations, discussions, and comments made by participants during a meeting.

[0804] "Projected materials" refers to digital materials such as presentation slides, documents, and images that are shared on the screen during a meeting.

[0805] A "generative AI model" refers to an artificial intelligence algorithm that has been trained to perform a specific task.

[0806] "Speech recognition technology" refers to technology for converting voice data into text.

[0807] "Image capture technology" refers to technology for capturing visual information from a screen or digital data.

[0808] "Cloud storage" refers to a service for storing and managing data on remote servers on the Internet.

[0809] A "summary video" is a video created to convey the main points of a meeting in a short amount of time.

[0810] A "prompt" is a textual instruction entered into a generative AI model to perform a specific task.

[0811] This invention is a system that invites a generation AI as a virtual participant in an online meeting, collects and analyzes the content of comments made during the meeting and the materials projected, and automatically generates a short summary video after the meeting ends. This invention eliminates the need to record meetings and create minutes, and allows users to efficiently grasp the key points of the meeting.

[0812] Overall concept of the system

[0813] The system consists of the following elements:

[0814] 1. User Initial Setup: The user adds the generated AI to the meeting invitation list using their web conferencing account.

[0815] 2. AGI participation by the server: The server automatically joins the virtual participant (AGI) to the conference at the conference start time.

[0816] 3. Data collection and analysis using AGI: Collect and analyze statements and materials during meetings in real time.

[0817] 4. Generation of summary video by the server: After the meeting ends, a summary video is generated based on the collected and analyzed data.

[0818] 5. Providing users with video summaries: Providing users with video summaries allows them to efficiently understand the content of the meeting.

[0819] Program Processing Details

[0820] User Initial Settings

[0821] The user logs in to their own web conferencing system (e.g., a general web conferencing system) and invites the generated AI (virtual participant) when setting up a conference. On the setting screen, they enter the generated AI's email address or ID and save it. This completes the setting for the generated AI to automatically join the conference.

[0822] Server AGI participation

[0823] As the meeting starts, the server automatically joins the AGI configured as a virtual participant to the meeting. The server connects the AGI using the meeting URL or invitation link provided by the user. At this stage, the server verifies that the AGI has successfully joined the meeting.

[0824] AGI data collection and analysis

[0825] When a meeting begins, the AGI performs the following tasks:

[0826] Speech Recognition: Use speech recognition technology (e.g., Google Speech-to-Text API, common speech recognition technology) to convert participants' speech into text in real time.

[0827] Document capture: Capture presentations and documents shared by participants on their screens using image capture technology (e.g., common image processing technology).

[0828] The acquired data is stored in cloud storage (e.g., a general cloud storage service).

[0829] Server-based generation of summary videos

[0830] After the meeting ends, the server generates a summary based on the data collected and analyzed by the AGI. To do this, it uses a generative AI model (e.g., a general machine learning model). The summary extracts the main points of the meeting, and the server automatically generates a summary video based on this summary using a video template (e.g., general video editing software). The generated video is then saved back to cloud storage.

[0831] Providing users with video summaries

[0832] The server notifies the user that the summary video is complete. This notification includes an access link to the summary video, which the user can view by clicking the link. This allows the user to quickly grasp the main points of the meeting.

[0833] Specific examples

[0834] For example, User A schedules a monthly meeting and adds a generated AI (virtual participant) to the invitation list. When the meeting starts, the server automatically invites the AGI to the meeting, where it performs real-time speech recognition and captures projected materials. When the meeting ends, the server generates a summary based on the collected data and creates a summary video based on the content. Finally, User A receives a notification a few minutes later that the summary video is ready, and can click the provided link to watch the video. This allows User A to quickly and efficiently understand the key points of the meeting.

[0835] Prompt Sentence Examples

[0836] Here are some example prompts for a generative AI model to generate a summary:

[0837] Please summarize the following meeting in a concise manner, extracting the main points and conclusions.

[0838] 1. Speaker A: "Today's meeting agenda is as follows..."

[0839] 2. Speaker B: "Regarding the budget review..."

[0840] 3. Speaker C: "I'll report on the progress of the following project..."

[0841] [Detailed minutes of the meeting follow]

[0842] Please summarize the main points and conclusions of this meeting in bullet points.

[0843] The specific embodiment of the present invention has been described above. This system allows users to quickly and efficiently grasp important information without having to go through the trouble of recording meetings and creating minutes.

[0844] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0845] Step 1: User Initial Setup

[0846] The user logs in to their own web conferencing system and schedules a meeting. At this time, they add the generated AI as a virtual participant. Specifically, they enter the generated AI's email address or ID on the meeting setting screen and add it to the invitation list. The input is the meeting setting information, and the output is the schedule data for the meeting to which the virtual participant has been added.

[0847] Step 2: Server joins AGI

[0848] As the start time of the conference approaches, the server automatically invites the generated AI (AGI) to join the conference. The server obtains the conference URL or invitation link based on the conference information set by the user and connects the AGI to that URL. This allows the AGI to join the conference. The input is the set conference URL and AGI connection information, and the output is status information indicating that the AGI has successfully joined the conference.

[0849] Step 3: AGI collects data

[0850] When a meeting begins, the AGI collects speech and projections in real time. Specifically, it performs the following processes:

[0851] 1. Speech recognition: Converting what is being said in a meeting into text data in real time, using tools such as the Google Speech-to-Text API.

[0852] 2. Document capture: Slides and documents shared during meetings are captured using image capture technology, and their contents are then analyzed and converted into text data using OCR (Optical Character Recognition) technology.

[0853] The input is the conference audio and shared screen data, and the output is the textual utterance data and captured material data.

[0854] Step 4: Save your data

[0855] The collected text data and image data are stored in cloud storage by the server. Data is stored using a common cloud storage service (e.g., Amazon S3). The input is the data collected by the AGI, and the output is the data stored in cloud storage.

[0856] Step 5: Generate a summary

[0857] After the meeting ends, the server analyzes the data stored in cloud storage and generates a summary using a generative AI model. Specifically, it uses a machine learning model (e.g., GPT-4) to extract the main points of the meeting based on the prompt. The input is the saved meeting data and the prompt, and the output is the summary text.

[0858] Step 6: Generate summary video

[0859] Based on the generated summary text, the server uses a video template to create a summary video. Using video editing software (e.g., general video editing software), a video that integrates text, audio narration, and images is automatically generated. The input is the summary text and an existing video template, and the output is the completed summary video.

[0860] Step 7: Provide the summary video to the user

[0861] The server notifies the user that the summary video is complete. The notification includes an access link to the summary video, which the user can click to watch. The input is the generated summary video, and the output is the link to provide the summary video and a notification message.

[0862] This series of processes allows users to efficiently grasp important information without having to go through the trouble of recording meetings and creating minutes.

[0863] (Application example 1)

[0864] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0865] In the operation management of autonomous vehicles, it is difficult to efficiently grasp the operation status and important events of multiple vehicles. In particular, as a large amount of data is generated from camera footage and various sensors during operation, operation managers are required to quickly and accurately analyze this data and make appropriate decisions. However, current technology lacks the means to efficiently collect and analyze this data and provide a summary of the operation status.

[0866] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0867] In this invention, the server includes means for receiving conference setting information and automatically adding virtual participants to the conference, means for collecting participant remarks and projection materials in real time during the conference, means for analyzing camera and sensor data collected while the vehicle is in operation to generate a summary, means for automatically creating a short summary video based on the generated summary, and means for providing the summary video to the user. This enables the operation manager of the autonomous vehicle to quickly and accurately grasp important events and driving conditions during operation and take prompt action.

[0868] "Conference setup information" is information necessary to automatically add virtual participants to a conference.

[0869] "Virtual participants" refers to generative AI that automatically participates in meetings and car rides, collecting and analyzing data.

[0870] The "means for automatically adding a virtual participant to a conference" is a method for automatically adding a virtual participant to a conference when the conference starts.

[0871] "Means of collecting information in real time" refers to methods of collecting information such as speech content, projected materials, camera footage, and sensor data on the spot during meetings or while driving a car.

[0872] A "camera" is an image capturing device mounted on a vehicle that records images while the vehicle is in motion.

[0873] A "sensor" is a data acquisition device installed in a vehicle that detects and records various physical information during operation.

[0874] "Means for analyzing and generating a summary" refers to a method for analyzing collected data and concisely summarizing its contents.

[0875] The "means for automatically creating a short summary video" is a means for automatically creating a short video of a meeting or operation status based on the summarized content.

[0876] The "means for providing to the user" refers to a method for providing the summary video in a form that allows the user to easily access it.

[0877] "Speech recognition technology" is a technology that converts voice into text data.

[0878] "Image capture technology" is a technology that captures projected materials or images as still images.

[0879] "Cloud storage" is a storage service for saving and managing data on the Internet.

[0880] "Means for backing up stored data" refers to a method for replicating data stored in cloud storage on a regular or ad-hoc basis to prepare for data loss.

[0881] This invention is a data collection and analysis system for managing the operation of autonomous vehicles. It analyzes camera footage and sensor data collected from the vehicle during operation, summarizes the results, and generates and provides a short summary video, allowing the operation manager to quickly understand the situation. Specific system configurations for implementing this invention are described below.

[0882] Overall concept of the system

[0883] The system consists of the following main components:

[0884] Camera image collection module (installed in autonomous vehicles)

[0885] Sensor data collection module (installed in autonomous vehicles)

[0886] Data analysis server

[0887] Generative AI Models

[0888] Summary video generation module

[0889] Cloud Storage

[0890] User Interface

[0891] Camera footage and sensor data collection

[0892] Autonomous vehicles are equipped with multiple cameras and sensors. These devices collect video and sensor data in real time while driving. The cameras record the vehicle's surroundings as video, while the sensors detect various data such as speed, position, acceleration, and the presence or absence of obstacles.

[0893] Data analysis server

[0894] The data analysis server receives and analyzes data collected from autonomous vehicles in real time. A generative AI model is used to analyze the collected camera footage and sensor data to generate summaries. For example, it generates summaries that explain traffic conditions and the occurrence of important events (such as sudden braking or sudden stops).

[0895] Summary video generation

[0896] The summary video generation module generates a summary video based on the analyzed data. The summary video includes images showing important events and situations during operation, along with related text information. This allows users to quickly understand the operation status.

[0897] Cloud storage and user provisioning

[0898] The generated summary video is stored in cloud storage, allowing the dispatcher to easily access it from anywhere. Through the user interface, the dispatcher can access the summary video and quickly obtain the information they need.

[0899] Specific examples

[0900] For example, when an autonomous vehicle passes through multiple intersections, it detects traffic conditions and events that occur at the intersections (such as a sudden vehicle stop or the appearance of an unexpected obstacle) from camera footage and sensor data. The generative AI model analyzes this data in real time and summarizes important points such as "A sudden stop occurred at intersection 1" or "A pedestrian ran out into the road at intersection 2" into text. Next, the summary video generation module creates a short summary video based on the summary and saves it in cloud storage. The operation manager can view this summary video via the user interface.

[0901] Examples of prompt statements

[0902] Example prompt for a generative AI model:

[0903] "Please summarize any significant events or notable operational findings from this camera footage and sensor data."

[0904] Such a system allows operation managers to quickly grasp key points extracted from large amounts of data and respond promptly.

[0905] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0906] Step 1:

[0907] Initial Setup

[0908] The server prepares to add virtual participants (generative AI models) while each vehicle is in operation based on the operation schedule and setting information of the autonomous vehicle received from the operation manager. The input is operation schedule information, and the output is a notification that the virtual participants have been set up. The server saves the setting information so that the virtual participants can automatically join the operation when the operation starts.

[0909] Step 2:

[0910] Starting Data Collection

[0911] Once the vehicle begins operation, the cameras and sensors installed on the vehicle begin collecting data in real time. The cameras record images of the surrounding area, and the sensors acquire data such as speed, position, acceleration, and obstacles. The input is the camera images and sensor data, and the output is the collected data. The server periodically receives this data and prepares it for analysis.

[0912] Step 3:

[0913] Data analysis

[0914] The server receives camera footage and sensor data transmitted sequentially during operation and analyzes it using the generative AI model. The input is the camera footage and sensor data collected in step 2, and the output is the analyzed summary information. The server sends prompts to the generative AI model, asking for the following to summarize important events and situations:

[0915] "Please summarize any significant events or notable operational findings from this camera footage and sensor data."

[0916] Step 4:

[0917] Summary video generation

[0918] The server generates a summary video based on the summary information obtained in step 3. The summary video generation module combines the summarized text information with related video to create a video that can be viewed in a short time. The input is the analyzed summary information, and the output is a summary video. The server saves the generated video in cloud storage.

[0919] Step 5:

[0920] Providing a summary video

[0921] After the generated summary video is saved in cloud storage, the server sends a notification to the operations manager. The notification includes an access link to the video, which the manager can click to view. The input is the summary video information saved in cloud storage, and the output is a notification sent to the operations manager. By watching the summary video, the operations manager can instantly understand important events and situations that occurred during operations.

[0922] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0923] The present invention is a system that combines an emotion recognition engine with an online conference, and provides a means to understand the main points of the conference more deeply by analyzing the emotions of participants during the conference and reflecting the results in a summary video. This system is implemented by the following main steps.

[0924] Overall concept of the system

[0925] The system consists of the following elements:

[0926] 1. User Initial Setup: The user adds the generative AI and emotion recognition engine to the meeting invitation list using their web conferencing account.

[0927] 2. AGI participation by the server: The server automatically joins the virtual participant (AGI) to the conference at the conference start time.

[0928] 3. Data collection and analysis: Using an emotion recognition engine and voice recognition technology, the content of statements made during meetings and the emotions of participants are collected and analyzed in real time.

[0929] 4. Summary video generation: A summary video is generated based on the collected data and the results of sentiment analysis.

[0930] 5. Providing users with video summary: Providing users with video summary provides an efficient way to understand the main points of the meeting and its emotions.

[0931] Program Processing Details

[0932] User Initial Settings

[0933] A user logs into their own web conferencing system and sets up a new meeting. At that time, they input meeting setting information such as the meeting date and time, participant list, and invitation link, and add the generative AI (AGI) and emotion recognition engine to the invitation list as virtual participants.

[0934] Participation of AGI and emotion recognition engine by server

[0935] The server automatically joins the virtual participants (AGI and emotion recognition engine) to the meeting at the scheduled meeting time. It uses the meeting URL or invitation link to ensure they are properly connected.

[0936] Data collection and analysis

[0937] During the meeting, the device (AGI projection system) converts participants' speech into text in real time using speech recognition technology. It also captures presentations and materials shared on the screen by the user and other participants using image capture technology. Furthermore, the emotion recognition engine analyzes the content and tone of speech to recognize participants' emotions in real time.

[0938] Emotion data storage and analysis

[0939] The server stores the captured text, image captures, and sentiment data in cloud storage, and uses a machine learning model to analyze the content and sentiment data and generate a summary that also reflects the results of sentiment analysis.

[0940] Summary video generation

[0941] After the meeting ends, the server generates a summary based on the stored data and identifies key moments from the meeting based on the emotion data. The server then uses a video template to automatically generate a summary video that reflects the results of the summary and emotion analysis. The summary video includes graphs and highlights that visually show changes in participants' emotions.

[0942] Providing summary videos to users

[0943] When the summary video is completed, the server sends a notification to the user, which includes a link to access the summary video. The user can click the link to watch the video. This allows the user to quickly grasp the main points of the meeting and changes in participants' emotions.

[0944] Specific examples

[0945] Let's take the example of User B setting up a weekly report meeting. In Step 1, User B adds the generation AI and emotion recognition engine to the invitation list and completes the meeting setup. From Step 2 to Step 6, the server automatically adds virtual participants to the meeting and collects and analyzes comments, materials, and emotion data from the meeting. After the meeting ends, the server generates a summary video based on the summary and emotion analysis results. Finally, User B receives a link to access the summary video and can watch the meeting highlights, which reflect the emotion data, to efficiently understand the key points of the meeting and changes in participants' emotions. This process frees User B from the hassle of recording meetings and creating minutes, and also makes it easier for them to understand participants' emotions.

[0946] The above describes a specific embodiment of the present invention. This system allows users to save the time and effort of recording meetings and creating minutes, and enables them to efficiently grasp not only important information but also changes in the emotions of participants.

[0947] The processing flow will be explained below.

[0948] Step 1:

[0949] A user logs into their own web conferencing system and sets up a new online meeting. At this time, they input information such as the meeting date and time, participant list, and invitation link, and add the generative AI (AGI) and emotion recognition engine to the invitation list as virtual participants.

[0950] Step 2:

[0951] When the meeting starts, the server automatically joins the AGI and emotion recognition engine to the meeting, which involves connecting using the meeting URL or invitation link.

[0952] Step 3:

[0953] During the meeting, the terminal (AGI projection system) converts participants' speech into text in real time using voice recognition technology, and captures presentations and materials using image capture technology.

[0954] Step 4:

[0955] Similarly, during a meeting, the emotion recognition engine analyzes the content and tone of speech to identify participants' emotions in real time, and this emotional data is instantly saved as digital data.

[0956] Step 5:

[0957] After the meeting ends, the server stores the data collected by the AGI and emotion recognition engine (speech text, document capture, and emotion analysis data) in cloud storage.

[0958] Step 6:

[0959] The server analyzes the stored data and generates a summary using the speech content and emotion data, which includes important speech content and emotional changes.

[0960] Step 7:

[0961] The server automatically generates a summary video using a video template based on the generated summary and the results of sentiment analysis, highlighting key points and important emotional moments from the meeting.

[0962] Step 8:

[0963] Once the summary video is complete, the server sends a notification to the user, which includes a link to access the summary video.

[0964] Step 9:

[0965] Users receive a notification and click the provided link to watch the summary video, which allows them to quickly grasp the key points of the meeting and the changes in participants' emotions.

[0966] Specific examples

[0967] For example, if User B schedules a weekly reporting meeting, in Step 1, the generation AI and emotion recognition engine are added to the meeting. When the meeting begins, in Steps 2 to 4, the AGI and emotion recognition engine automatically join the meeting and collect the remarks, materials, and emotion data. After the meeting ends, in Steps 5 to 7, the server generates a summary and automatically generates a summary video that reflects the emotion data. Finally, in Steps 8 and 9, User B can watch the summary video and understand the highlights of the meeting that reflect the emotion data. This process frees User B from the hassle of recording meetings and creating minutes, and also makes it easier to understand the emotions of the participants.

[0968] Example 2

[0969] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0970] While current online meeting systems allow participants to share remarks and materials during meetings, it is difficult to efficiently grasp participants' emotions and key moments. Furthermore, current systems do not automatically generate summary videos that include meeting summaries or emotion analysis results. This makes it difficult to effectively follow up on meetings and grasp key points, and reviewing the content of long meetings afterward is particularly time-consuming and labor-intensive.

[0971] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving conference setting information and automatically adding virtual participants to the conference, means for collecting participants' comments and projection materials in real time during the conference, means for analyzing the collected comment content and emotion data to generate a summary, means for automatically creating a short summary video based on the generated summary and emotion analysis results, and means for providing the summary video to the user. This makes it possible to efficiently grasp the content of the conference and changes in participants' emotions, and to quickly understand the gist of important information and emotions.

[0972] "Conference setting information" is information necessary to hold a conference, such as the date and time of the online conference, a list of participants, and an invitation link.

[0973] A "virtual participant" is an agent or program that is not physically present but is brought into an online conference by the system.

[0974] "Speech content" refers to voice data spoken by participants during an online conference that has been converted into text using voice recognition technology.

[0975] "Projected materials" are visual materials such as presentations and documents that participants share during an online meeting.

[0976] "Means of collection" refers to technologies and systems for acquiring participants' comments and projected materials in real time during online meetings.

[0977] "Speech recognition technology" is a technology for converting voice data into text data.

[0978] "Image capture technology" is a technology that captures displayed visual materials as image data.

[0979] "Emotional data" is emotional information analyzed based on the content of participants' statements and tone of voice.

[0980] A "summary" is a concise summary of the main points of a meeting based on collected speech and sentiment data.

[0981] A "summary video" is a short video generated from collected data that visually shows key moments of the meeting and the results of sentiment analysis.

[0982] "Cloud storage" is an external storage service for storing and managing data via the Internet.

[0983] "Backup measures" are techniques and methods for regularly replicating and backing up stored data.

[0984] The present invention is a system that combines an online conference with an emotion recognition engine, analyzes the content of comments and emotions of participants during the conference, and generates a summary video based on the results. A specific embodiment of this system will be described below.

[0985] System Overview

[0986] The system consists of the following elements:

[0987] 1. Initial Setup: The user enters the meeting setup information and adds the virtual participant (AGI) to the meeting invitation list.

[0988] 2. AGI joining: The server automatically joins the virtual participants to the meeting at the meeting start time.

[0989] 3. Data collection and analysis: Using emotion recognition engines and voice recognition technology, the content and emotions expressed during meetings are collected and analyzed in real time.

[0990] 4. Summary video generation: A summary video is generated based on the collected data and the results of sentiment analysis.

[0991] 5. Providing summary videos: Providing users with summary videos provides a means to efficiently understand the content and emotional gist of the meeting.

[0992] Specific Examples

[0993] 1. Initial Setup

[0994] A user logs into their online conference system and sets up a new conference. At this time, they input conference setting information such as the conference date and time, participant list, and invitation link, and add the generative AI (AGI) and emotion recognition engine to the invitation list as virtual participants.

[0995] 2. AGI participation

[0996] The server automatically joins the virtual participants (generative AI and emotion recognition engine) to the meeting at the scheduled meeting time. The server verifies that they are properly connected using the meeting URL or invitation link.

[0997] 3. Data collection and analysis

[0998] During the meeting, the device (AGI projection system) converts participants' speech into text in real time using speech recognition technology (e.g., Google Speech-to-Text API). It also captures presentations and materials shared on the screen by the user or other participants using image capture technology (e.g., screen capture software). Furthermore, an emotion recognition engine analyzes the content and tone of speech to recognize participants' emotions (e.g., joy, sadness, surprise, etc.) in real time (e.g., Azure Emotion API).

[0999] 4. Emotion Data Storage and Analysis

[1000] The server stores the captured speech text, image captures, and sentiment data in cloud storage (e.g., Amazon S3), analyzes the speech content and sentiment data using a machine learning model (e.g., the BERT model), and generates a summary that also reflects the results of sentiment analysis.

[1001] 5. Summary video generation

[1002] After the meeting ends, the server generates a summary based on the stored data and identifies key moments from the meeting based on the emotion data. The server then uses a video template to automatically generate a summary video that reflects the results of the summary and emotion analysis. The summary video includes graphs and highlights that visually show changes in participants' emotions.

[1003] 6. Providing a summary video

[1004] When the summary video is completed, the server sends a notification to the user, which includes a link to access the summary video. The user can click the link to watch the video. This allows the user to quickly grasp the main points of the meeting and changes in participants' emotions.

[1005] Specific examples

[1006] For example, consider the case where User B sets up a weekly reporting meeting. User B adds the generation AI and emotion recognition engine to the invitation list during the initial setup and completes the meeting setup. When the meeting starts, the server automatically adds virtual participants to the meeting, shares comments and materials during the meeting, and collects and analyzes emotional data. After the meeting ends, the server generates a summary video based on the summary and emotion analysis results and provides User B with an access link. By watching the summary video, User B can efficiently understand the key points of the meeting and changes in participants' emotions.

[1007] Prompt Sentence Examples

[1008] Below is an example prompt for asking a generative AI model to summarize a meeting:

[1009] Sample prompt: "Based on the audio data and sentiment analysis results from this meeting, please create a summary that reflects key statements and changes in sentiment."

[1010] This allows the generative AI model to generate a summary based on the meeting's audio data and emotion analysis results and provide it to the user.

[1011] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1012] Step 1:

[1013] A user logs into the online conference system and sets up a new conference. As input, they provide the date and time of the conference, a list of participants, an invitation link, and virtual participants (generative AI and emotion recognition engine). Based on this information, the system sets up the conference and outputs the necessary configuration information.

[1014] Step 2:

[1015] The server automatically joins the virtual participants (generative AI and emotion recognition engine) to the meeting at the scheduled meeting time. It uses the meeting invitation link and setting information as input to connect the virtual participants to the meeting. As output, it obtains confirmation information of the virtual participants who have joined the meeting.

[1016] Step 3:

[1017] During the meeting, the terminal (AGI projection system) converts participants' speech into text in real time using speech recognition technology (for example, Google Speech-to-Text API). It receives participants' speech data as input and generates text data using speech recognition technology. It obtains real-time speech text data as output.

[1018] Step 4:

[1019] The device captures presentations and materials shared on the screen by the user or other participants using image capture technology (e.g., screen capture software). It captures image data of the shared screen as input and creates the captured image data as output.

[1020] Step 5:

[1021] The emotion recognition engine analyzes the content and tone of speech during a meeting to recognize participants' emotions in real time. It receives speech text data and voice tone data as input, analyzes them, and generates emotion data. The real-time emotion data is obtained as output.

[1022] Step 6:

[1023] The server stores the acquired utterance text, image capture data, and emotion data in cloud storage (e.g., Amazon S3). It receives all collected data as input and performs the storage process. As output, it receives confirmation information for the data that has been saved.

[1024] Step 7:

[1025] The server analyzes the stored data and generates summaries. It uses the stored text and sentiment data as input and analyzes the data using a machine learning model (e.g., the BERT model). As output, it generates summary data based on the content and sentiment analysis of the speech.

[1026] Step 8:

[1027] The server automatically generates a summary video based on the summary data and the sentiment analysis results. It uses the summary data and the sentiment analysis results as input and applies a video template. As output, it generates a summary video containing key moments from the meeting.

[1028] Step 9:

[1029] The server sends a notification to the user when the summary video is complete. As input, it creates link information for the summary video and sends a notification message. As output, it sends a notification containing an access link to the summary video.

[1030] As a concrete example, User B sets up a weekly reporting meeting and follows the process described above to receive a summary video after the meeting. User B can quickly grasp the main points of the meeting and the emotions expressed, and can report efficiently.

[1031] (Application example 2)

[1032] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1033] With current customer support in virtual stores, it is difficult to accurately grasp customer emotions and respond quickly and appropriately. Furthermore, reviewing the content of conversations between customers and support staff requires a huge amount of time and effort, making it difficult to implement effective support follow-up. Furthermore, there is a high possibility that important moments in conversations based on emotional changes may be missed.

[1034] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving conference setting information and automatically adding virtual participants to the conference, means for collecting participant comments and projection materials in real time during the conference, means for analyzing the collected data and generating a summary, means for acquiring and analyzing emotion data of participants during the conference, means for automatically creating a short summary video based on the generated summary and emotion data, and means for providing the summary video to the user. This makes it possible to provide an effective summary of the content of the conversation and emotion data to improve the quality of customer support.

[1035] "Conference setting information" refers to detailed information for setting up a conference, such as a participant list including virtual participants, the date and time of the conference, and an invitation link.

[1036] A "virtual participant" is a virtual agent, such as an artificial intelligence or generative AI model, that participates in meetings in the same way as a human participant and plays a role in collecting and analyzing data.

[1037] "Utterances" are words or sentences spoken by participants during a meeting to communicate.

[1038] "Projected materials" are visual information such as slides, documents, and images that are shared by participants during presentations during a meeting.

[1039] "Collecting in real time" means capturing data simultaneously while the meeting is taking place.

[1040] A "summary" is information that briefly summarizes the contents based on the collected meeting data.

[1041] "Emotional data" is data that indicates the emotional state of participants, obtained by analyzing their facial expressions and tone of voice.

[1042] A "summary video" is a short video that includes important moments and key points from a meeting, and is presented visually, incorporating emotional data.

[1043] "Voice recognition technology" is a technology that analyzes voice and converts it into text data.

[1044] "Image capture technology" is a technology that captures images of materials and documents displayed on a screen and acquires them as data.

[1045] This invention is a system that improves the quality of customer support by analyzing the emotions of participants during online meetings between customers and customer support teams in customer support sessions at virtual stores and reflecting the results in summary videos.

[1046] The main components of the system are:

[1047] 1. A means of receiving conference setup information and automatically adding virtual participants to a conference:

[1048] The server receives conference setting information (participant list, conference date and time, invitation link, etc.) from a user who logs in to the virtual store's customer support system, and automatically adds an artificial intelligence model or emotion recognition engine to the conference as a virtual participant.

[1049] 2. A means of collecting participants' comments and projections in real time during a meeting:

[1050] The server converts speech during the meeting into text in real time using voice recognition technology (e.g., Google Speech-to-Text), and also uses image capture technology to obtain projected and shared materials.

[1051] 3. Means of analyzing collected data and generating summaries:

[1052] The server analyzes the collected speech text and projection materials using a machine learning model and generates a summary.

[1053] 4. Means of acquiring and analyzing participants' emotional data during a meeting:

[1054] The server uses an emotion recognition engine (e.g., Affectiva) to analyze the facial expressions and tone of voice of customers and customer support members in real time to obtain emotional data.

[1055] 5. A method for automatically creating short summary videos based on the generated summaries and emotion data:

[1056] The server automatically generates a short summary video that reflects important moments and emotional changes based on the generated summary and emotion data, and visually emphasizes the highlights using video generation software (e.g., Adobe Premiere Pro).

[1057] 6. Means of providing summary video to users:

[1058] Once the summary video is generated, the server notifies the user and provides an access link to the video, allowing the user to efficiently grasp the key points of the customer support session and the customer's emotional state.

[1059] Specific examples

[1060] As an example, consider setting up an online session in a virtual store where a customer seeks support for a defective product. When setting up the session, the user adds artificial intelligence and an emotion recognition engine as virtual participants. During the session, the server converts comments into text in real time and also collects projected materials. The emotion recognition engine analyzes the customer's emotions and obtains data. After the session ends, the server analyzes this data and generates a summary video containing the emotion analysis results. Finally, the user receives a link to the summary video and can take appropriate measures to improve customer satisfaction.

[1061] Prompt Sentence Examples

[1062] scenario:

[1063] During online sessions to respond to customer inquiries

[1064] 1. Analyze the customer's facial expressions and voice to determine their emotions.

[1065] 2. Convert what is said into text in real time.

[1066] 3. After the session, a summary video is generated showing key points and emotional changes.

[1067] Product:

[1068] A highlight summary video that visually displays customer sentiment data.

[1069] Technology used:

[1070] Emotion recognition engine, voice recognition technology, video generation software

[1071] Processing Points:

[1072] 1. Real-time monitoring of online conversations and sentiment between customers and customer support agents.

[1073] 2. Emotion data and text data are stored in cloud storage and then analyzed.

[1074] 3. Extract necessary scenes and generate a summary video that visually shows changes in emotions.

[1075] The above is a specific embodiment for carrying out the invention. This system is expected to improve the quality of customer support in virtual stores and increase customer satisfaction.

[1076] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1077] Step 1:

[1078] A user logs in to the virtual store's customer support system and sets up a new support session. The user enters meeting setting information such as the session date and time, participant list, and invitation link, and adds the generative AI model and emotion recognition engine as virtual participants. The entered meeting setting information is sent to the server, which then automatically adds the virtual participants based on this information and completes the setup.

[1079] Step 2:

[1080] The server automatically joins the virtual participants (generative AI model and emotion recognition engine) to the session at the set session time. The server uses the invitation link to properly connect, and as soon as the session starts, the system starts collecting data. The inputs are the invitation link and the session start time, and the output is the session participation status of the virtual participants.

[1081] Step 3:

[1082] During the session, the device uses speech recognition technology to convert speech into text data in real time. The input is the speech of each participant, which the server collects as text data. This text data is used in the subsequent analysis step. The output is text data.

[1083] Step 4:

[1084] The device uses image capture technology to capture the projected materials shared during the session in real time. The input is the shared materials on the screen, and the output is image data. The server stores this image data along with other data from the session.

[1085] Step 5:

[1086] The server uses an emotion recognition engine to analyze participants' facial expressions and tone of voice in real time to collect emotional data. The input is the participants' facial images and voices, and the output is emotional data. This emotional data is reflected in the generated summary video.

[1087] Step 6:

[1088] The server analyzes the collected speech text, projection materials, and emotion data using a machine learning model. The input is the aforementioned text data, image data, and emotion data, and a summary is generated through data analysis. The output is summary data.

[1089] Step 7:

[1090] The server automatically generates a short summary video based on the generated summary and emotion data. The input is the summary data and emotion data, and the output is a summary video. Important moments and emotional changes are visually emphasized using video generation software.

[1091] Step 8:

[1092] The server notifies the user of the generated summary video and provides an access link. The input is the summary video and the user's contact information, and the output is the notification and the access link to the video. The user can watch the summary video via this link and grasp the main points and emotional changes of the customer support session.

[1093] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1094] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1095] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1096] [Fourth embodiment]

[1097] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1098] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1099] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1100] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1101] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1102] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1103] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1104] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1105] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1106] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1107] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1108] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1109] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1110] This invention is a system that invites a generation AI to an online meeting as a virtual participant, collects and analyzes comments and projected materials during the meeting, and automatically generates a short summary video after the meeting. This system eliminates the need to record meetings and create minutes, and allows users to efficiently grasp the key points of the meeting.

[1111] Overall concept of the system

[1112] The system consists of the following elements:

[1113] 1. User Initial Setup: The user adds the generated AI to the meeting invitation list using their web conferencing account.

[1114] 2. AGI participation by the server: The server automatically joins the virtual participant (AGI) to the conference at the conference start time.

[1115] 3. Data collection and analysis using AGI: Collect and analyze statements and materials during meetings in real time.

[1116] 4. Summary video generation by the server: After the meeting ends, a summary video is generated based on the collected and analyzed data.

[1117] 5. Providing summary videos to users: Providing summary videos to users to help them understand the contents of the meeting efficiently.

[1118] Program Processing Details

[1119] User Initial Settings

[1120] Users simply log in to their web conferencing system and add the generated AI (virtual participant) to the invitation list when setting up a meeting. This process is an additional step in the normal operation of scheduling a meeting, and users can easily perform it.

[1121] Server AGI participation

[1122] When the meeting starts, the server automatically joins the AGI to the meeting. The server uses the specified meeting URL or invitation link to associate virtual participants with the meeting. At this stage, the server verifies that the AGI connection was successful.

[1123] AGI data collection and analysis

[1124] During the meeting, the AGI converts participants' speech into text data using speech recognition technology and records the speech in real time. It also captures presentations and materials shared by participants on their screens using image capture technology. The captured data is then stored in cloud storage.

[1125] Server-generated summary video

[1126] When the meeting ends, the server uses a machine learning model to generate a summary based on the data collected and analyzed by the AGI. This summary extracts the main points of the meeting. The server then automatically generates a summary video using a video template based on this summary. The generated video is then saved back to cloud storage.

[1127] Providing summary videos to users

[1128] Finally, the server notifies the user that the summary video is complete. This notification includes an access link to the summary video, which the user can click to watch. This allows the user to quickly grasp the main points of the meeting.

[1129] Specific examples

[1130] For example, User A schedules a monthly meeting and adds AGI to the invitation list. When the meeting starts, the server automatically invites AGI to join. During the meeting, AGI converts speeches into text and captures projected materials. When the meeting ends, AGI creates a summary from the collected data, and the server generates a summary video based on this summary. A few minutes later, User A receives a notification that the summary video is complete and can click the provided link to watch the video. This process allows User A to efficiently understand the main points of the meeting.

[1131] The specific embodiment of the present invention has been described above. This system allows users to quickly and efficiently grasp important information without having to go through the trouble of recording meetings and creating minutes.

[1132] The processing flow will be explained below.

[1133] Step 1:

[1134] A user logs into their own web conferencing system and sets up a new conference. At that time, they enter conference setting information such as the conference date and time, participant list, and invitation link, and add the generated AI (AGI) to the invitation list as a virtual participant.

[1135] Step 2:

[1136] The server schedules the AGI's participation based on the conference setting information received from the user. At the conference start time, the server sets a trigger to automatically join the AGI.

[1137] Step 3:

[1138] When it is time for the meeting to start, the server uses the meeting URL or invitation link to invite the AGI to join. The server verifies that the AGI has successfully connected to the meeting.

[1139] Step 4:

[1140] During the meeting, the terminal (AGI projection system) converts participants' speech into text in real time using speech recognition technology, and also captures presentations and materials shared on the screen by users and other participants using image capture technology.

[1141] Step 5:

[1142] The server stores the speech text and image capture data obtained during the meeting in cloud storage, which ensures consistent storage and access of the data.

[1143] Step 6:

[1144] Once the meeting is over, the server begins analyzing the data stored in cloud storage, using machine learning models to extract key points from the speech and materials and generate a summary text.

[1145] Step 7:

[1146] The server selects a video template based on the generated summary text and materials, and automatically generates a summary video based on the summary. The generated summary video is then saved in cloud storage.

[1147] Step 8:

[1148] Once the summary video is complete, the server sends a notification to the user, which includes a link to access the summary video, allowing the user to easily watch the video.

[1149] Step 9:

[1150] Users can click on the link provided by the server to watch the summary video, which allows them to quickly and efficiently grasp the main points of the conference.

[1151] Specific examples

[1152] As an example, let's explain the process of User A setting up a monthly meeting. In step 1, User A adds the generation AI to the invitation list and completes the meeting setup. From step 2 to step 9, the server automatically invites the AGI to the meeting and collects the content of comments and materials during the meeting. After the meeting ends, the AGI analyzes the data and generates a summary, and the server automatically generates a summary video. Finally, User A receives an access link to the summary video and can efficiently understand the key points of the meeting by watching the video. This process frees User A from the hassle of recording meetings and creating minutes.

[1153] Example 1

[1154] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1155] In today's business environment, online meetings are becoming more frequent and consume a lot of time. Recording meetings and creating minutes requires time and effort, making it difficult to efficiently grasp the information. It is also not easy to accurately summarize the content of meetings and extract only the key points. These issues increase the risk of important information being overlooked after the meeting.

[1156] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1157] In this invention, the server includes means for receiving conference setting information and automatically adding virtual participants to the conference, means for collecting participants' remarks and projection materials in real time during the conference, means for analyzing the collected data and generating a summary using a generative AI model, means for automatically creating a short summary video based on the generated summary, and means for providing the summary video to the user. This eliminates the need to record the conference and create minutes, and enables users to efficiently grasp the key points of the conference.

[1158] "Meeting setting information" refers to information necessary to set up a meeting, such as the online meeting schedule, participant list, and invitation URL.

[1159] "Virtual participants" refer to participants who are not real people, but are created using artificial intelligence such as generative AI.

[1160] "Participant Remarks" refers to oral presentations, discussions, and comments made by participants during a meeting.

[1161] "Projected materials" refers to digital materials such as presentation slides, documents, and images that are shared on the screen during a meeting.

[1162] A "generative AI model" refers to an artificial intelligence algorithm that has been trained to perform a specific task.

[1163] "Speech recognition technology" refers to technology for converting voice data into text.

[1164] "Image capture technology" refers to technology for capturing visual information from a screen or digital data.

[1165] "Cloud storage" refers to a service for storing and managing data on remote servers on the Internet.

[1166] A "summary video" is a video created to convey the main points of a meeting in a short amount of time.

[1167] A "prompt" is a textual instruction entered into a generative AI model to perform a specific task.

[1168] This invention is a system that invites a generation AI as a virtual participant in an online meeting, collects and analyzes the content of comments made during the meeting and the materials projected, and automatically generates a short summary video after the meeting ends. This invention eliminates the need to record meetings and create minutes, and allows users to efficiently grasp the key points of the meeting.

[1169] Overall concept of the system

[1170] The system consists of the following elements:

[1171] 1. User Initial Setup: The user adds the generated AI to the meeting invitation list using their web conferencing account.

[1172] 2. AGI participation by the server: The server automatically joins the virtual participant (AGI) to the conference at the conference start time.

[1173] 3. Data collection and analysis using AGI: Collect and analyze statements and materials during meetings in real time.

[1174] 4. Generation of summary video by the server: After the meeting ends, a summary video is generated based on the collected and analyzed data.

[1175] 5. Providing users with video summaries: Providing users with video summaries allows them to efficiently understand the content of the meeting.

[1176] Program Processing Details

[1177] User Initial Settings

[1178] The user logs in to their own web conferencing system (e.g., a general web conferencing system) and invites the generated AI (virtual participant) when setting up a conference. On the setting screen, they enter the generated AI's email address or ID and save it. This completes the setting for the generated AI to automatically join the conference.

[1179] Server AGI participation

[1180] As the meeting starts, the server automatically joins the AGI configured as a virtual participant to the meeting. The server connects the AGI using the meeting URL or invitation link provided by the user. At this stage, the server verifies that the AGI has successfully joined the meeting.

[1181] AGI data collection and analysis

[1182] When a meeting begins, the AGI performs the following tasks:

[1183] Speech Recognition: Use speech recognition technology (e.g., Google Speech-to-Text API, common speech recognition technology) to convert participants' speech into text in real time.

[1184] Document capture: Capture presentations and documents shared by participants on their screens using image capture technology (e.g., common image processing technology).

[1185] The acquired data is stored in cloud storage (e.g., a general cloud storage service).

[1186] Server-based generation of summary videos

[1187] After the meeting ends, the server generates a summary based on the data collected and analyzed by the AGI. To do this, it uses a generative AI model (e.g., a general machine learning model). The summary extracts the main points of the meeting, and the server automatically generates a summary video based on this summary using a video template (e.g., general video editing software). The generated video is then saved back to cloud storage.

[1188] Providing users with video summaries

[1189] The server notifies the user that the summary video is complete. This notification includes an access link to the summary video, which the user can view by clicking the link. This allows the user to quickly grasp the main points of the meeting.

[1190] Specific examples

[1191] For example, User A schedules a monthly meeting and adds a generated AI (virtual participant) to the invitation list. When the meeting starts, the server automatically invites the AGI to the meeting, where it performs real-time speech recognition and captures projected materials. When the meeting ends, the server generates a summary based on the collected data and creates a summary video based on the content. Finally, User A receives a notification a few minutes later that the summary video is ready, and can click the provided link to watch the video. This allows User A to quickly and efficiently understand the key points of the meeting.

[1192] Prompt Sentence Examples

[1193] Here are some example prompts for a generative AI model to generate a summary:

[1194] Please summarize the following meeting in a concise manner, extracting the main points and conclusions.

[1195] 1. Speaker A: "Today's meeting agenda is as follows..."

[1196] 2. Speaker B: "Regarding the budget review..."

[1197] 3. Speaker C: "I'll report on the progress of the following project..."

[1198] [Detailed minutes of the meeting follow]

[1199] Please summarize the main points and conclusions of this meeting in bullet points.

[1200] The specific embodiment of the present invention has been described above. This system allows users to quickly and efficiently grasp important information without having to go through the trouble of recording meetings and creating minutes.

[1201] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1202] Step 1: User Initial Setup

[1203] The user logs in to their own web conferencing system and schedules a meeting. At this time, they add the generated AI as a virtual participant. Specifically, they enter the generated AI's email address or ID on the meeting setting screen and add it to the invitation list. The input is the meeting setting information, and the output is the schedule data for the meeting to which the virtual participant has been added.

[1204] Step 2: Server joins AGI

[1205] As the start time of the conference approaches, the server automatically invites the generated AI (AGI) to join the conference. The server obtains the conference URL or invitation link based on the conference information set by the user and connects the AGI to that URL. This allows the AGI to join the conference. The input is the set conference URL and AGI connection information, and the output is status information indicating that the AGI has successfully joined the conference.

[1206] Step 3: AGI collects data

[1207] When a meeting begins, the AGI collects speech and projections in real time. Specifically, it performs the following processes:

[1208] 1. Speech recognition: Converting what is being said in a meeting into text data in real time, using tools such as the Google Speech-to-Text API.

[1209] 2. Document capture: Slides and documents shared during meetings are captured using image capture technology, and their contents are then analyzed and converted into text data using OCR (Optical Character Recognition) technology.

[1210] The input is the conference audio and shared screen data, and the output is the textual utterance data and captured material data.

[1211] Step 4: Save your data

[1212] The collected text data and image data are stored in cloud storage by the server. Data is stored using a common cloud storage service (e.g., Amazon S3). The input is the data collected by the AGI, and the output is the data stored in cloud storage.

[1213] Step 5: Generate a summary

[1214] After the meeting ends, the server analyzes the data stored in cloud storage and generates a summary using a generative AI model. Specifically, it uses a machine learning model (e.g., GPT-4) to extract the main points of the meeting based on the prompt. The input is the saved meeting data and the prompt, and the output is the summary text.

[1215] Step 6: Generate summary video

[1216] Based on the generated summary text, the server uses a video template to create a summary video. Using video editing software (e.g., general video editing software), a video that integrates text, audio narration, and images is automatically generated. The input is the summary text and an existing video template, and the output is the completed summary video.

[1217] Step 7: Provide the summary video to the user

[1218] The server notifies the user that the summary video is complete. The notification includes an access link to the summary video, which the user can click to watch. The input is the generated summary video, and the output is the link to provide the summary video and a notification message.

[1219] This series of processes allows users to efficiently grasp important information without having to go through the trouble of recording meetings and creating minutes.

[1220] (Application example 1)

[1221] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1222] In the operation management of autonomous vehicles, it is difficult to efficiently grasp the operation status and important events of multiple vehicles. In particular, as a large amount of data is generated from camera footage and various sensors during operation, operation managers are required to quickly and accurately analyze this data and make appropriate decisions. However, current technology lacks the means to efficiently collect and analyze this data and provide a summary of the operation status.

[1223] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1224] In this invention, the server includes means for receiving conference setting information and automatically adding virtual participants to the conference, means for collecting participant remarks and projection materials in real time during the conference, means for analyzing camera and sensor data collected while the vehicle is in operation to generate a summary, means for automatically creating a short summary video based on the generated summary, and means for providing the summary video to the user. This enables the operation manager of the autonomous vehicle to quickly and accurately grasp important events and driving conditions during operation and take prompt action.

[1225] "Conference setup information" is information necessary to automatically add virtual participants to a conference.

[1226] "Virtual participants" refers to generative AI that automatically participates in meetings and car rides, collecting and analyzing data.

[1227] The "means for automatically adding a virtual participant to a conference" is a method for automatically adding a virtual participant to a conference when the conference starts.

[1228] "Means of collecting information in real time" refers to methods of collecting information such as speech content, projected materials, camera footage, and sensor data on the spot during meetings or while driving a car.

[1229] A "camera" is an image capturing device mounted on a vehicle that records images while the vehicle is in motion.

[1230] A "sensor" is a data acquisition device installed in a vehicle that detects and records various physical information during operation.

[1231] "Means for analyzing and generating a summary" refers to a method for analyzing collected data and concisely summarizing its contents.

[1232] The "means for automatically creating a short summary video" is a means for automatically creating a short video of a meeting or operation status based on the summarized content.

[1233] The "means for providing to the user" refers to a method for providing the summary video in a form that allows the user to easily access it.

[1234] "Speech recognition technology" is a technology that converts voice into text data.

[1235] "Image capture technology" is a technology that captures projected materials or images as still images.

[1236] "Cloud storage" is a storage service for saving and managing data on the Internet.

[1237] "Means for backing up stored data" refers to a method for replicating data stored in cloud storage on a regular or ad-hoc basis to prepare for data loss.

[1238] This invention is a data collection and analysis system for managing the operation of autonomous vehicles. It analyzes camera footage and sensor data collected from the vehicle during operation, summarizes the results, and generates and provides a short summary video, allowing the operation manager to quickly understand the situation. Specific system configurations for implementing this invention are described below.

[1239] Overall concept of the system

[1240] The system consists of the following main components:

[1241] Camera image collection module (installed in autonomous vehicles)

[1242] Sensor data collection module (installed in autonomous vehicles)

[1243] Data analysis server

[1244] Generative AI Models

[1245] Summary video generation module

[1246] Cloud Storage

[1247] User Interface

[1248] Camera footage and sensor data collection

[1249] Autonomous vehicles are equipped with multiple cameras and sensors. These devices collect video and sensor data in real time while driving. The cameras record the vehicle's surroundings as video, while the sensors detect various data such as speed, position, acceleration, and the presence or absence of obstacles.

[1250] Data analysis server

[1251] The data analysis server receives and analyzes data collected from autonomous vehicles in real time. A generative AI model is used to analyze the collected camera footage and sensor data to generate summaries. For example, it generates summaries that explain traffic conditions and the occurrence of important events (such as sudden braking or sudden stops).

[1252] Summary video generation

[1253] The summary video generation module generates a summary video based on the analyzed data. The summary video includes images showing important events and situations during operation, along with related text information. This allows users to quickly understand the operation status.

[1254] Cloud storage and user provisioning

[1255] The generated summary video is stored in cloud storage, allowing the dispatcher to easily access it from anywhere. Through the user interface, the dispatcher can access the summary video and quickly obtain the information they need.

[1256] Specific examples

[1257] For example, when an autonomous vehicle passes through multiple intersections, it detects traffic conditions and events that occur at the intersections (such as a sudden vehicle stop or the appearance of an unexpected obstacle) from camera footage and sensor data. The generative AI model analyzes this data in real time and summarizes important points such as "A sudden stop occurred at intersection 1" or "A pedestrian ran out into the road at intersection 2" into text. Next, the summary video generation module creates a short summary video based on the summary and saves it in cloud storage. The operation manager can view this summary video via the user interface.

[1258] Examples of prompt statements

[1259] Example prompt for a generative AI model:

[1260] "Please summarize any significant events or notable operational findings from this camera footage and sensor data."

[1261] Such a system allows operation managers to quickly grasp key points extracted from large amounts of data and respond promptly.

[1262] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1263] Step 1:

[1264] Initial Setup

[1265] The server prepares to add virtual participants (generative AI models) while each vehicle is in operation based on the operation schedule and setting information of the autonomous vehicle received from the operation manager. The input is operation schedule information, and the output is a notification that the virtual participants have been set up. The server saves the setting information so that the virtual participants can automatically join the operation when the operation starts.

[1266] Step 2:

[1267] Starting Data Collection

[1268] Once the vehicle begins operation, the cameras and sensors installed on the vehicle begin collecting data in real time. The cameras record images of the surrounding area, and the sensors acquire data such as speed, position, acceleration, and obstacles. The input is the camera images and sensor data, and the output is the collected data. The server periodically receives this data and prepares it for analysis.

[1269] Step 3:

[1270] Data analysis

[1271] The server receives camera footage and sensor data transmitted sequentially during operation and analyzes it using the generative AI model. The input is the camera footage and sensor data collected in step 2, and the output is the analyzed summary information. The server sends prompts to the generative AI model, asking for the following to summarize important events and situations:

[1272] "Please summarize any significant events or notable operational findings from this camera footage and sensor data."

[1273] Step 4:

[1274] Summary video generation

[1275] The server generates a summary video based on the summary information obtained in step 3. The summary video generation module combines the summarized text information with related video to create a video that can be viewed in a short time. The input is the analyzed summary information, and the output is a summary video. The server saves the generated video in cloud storage.

[1276] Step 5:

[1277] Providing a summary video

[1278] After the generated summary video is saved in cloud storage, the server sends a notification to the operations manager. The notification includes an access link to the video, which the manager can click to view. The input is the summary video information saved in cloud storage, and the output is a notification sent to the operations manager. By watching the summary video, the operations manager can instantly understand important events and situations that occurred during operations.

[1279] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1280] The present invention is a system that combines an emotion recognition engine with an online conference, and provides a means to understand the main points of the conference more deeply by analyzing the emotions of participants during the conference and reflecting the results in a summary video. This system is implemented by the following main steps.

[1281] Overall concept of the system

[1282] The system consists of the following elements:

[1283] 1. User Initial Setup: The user adds the generative AI and emotion recognition engine to the meeting invitation list using their web conferencing account.

[1284] 2. AGI participation by the server: The server automatically joins the virtual participant (AGI) to the conference at the conference start time.

[1285] 3. Data collection and analysis: Using an emotion recognition engine and voice recognition technology, the content of statements made during meetings and the emotions of participants are collected and analyzed in real time.

[1286] 4. Summary video generation: A summary video is generated based on the collected data and the results of sentiment analysis.

[1287] 5. Providing users with video summary: Providing users with video summary provides an efficient way to understand the main points of the meeting and its emotions.

[1288] Program Processing Details

[1289] User Initial Settings

[1290] A user logs into their own web conferencing system and sets up a new meeting. At that time, they input meeting setting information such as the meeting date and time, participant list, and invitation link, and add the generative AI (AGI) and emotion recognition engine to the invitation list as virtual participants.

[1291] Participation of AGI and emotion recognition engine by server

[1292] The server automatically joins the virtual participants (AGI and emotion recognition engine) to the meeting at the scheduled meeting time. It uses the meeting URL or invitation link to ensure they are properly connected.

[1293] Data collection and analysis

[1294] During the meeting, the device (AGI projection system) converts participants' speech into text in real time using speech recognition technology. It also captures presentations and materials shared on the screen by the user and other participants using image capture technology. Furthermore, the emotion recognition engine analyzes the content and tone of speech to recognize participants' emotions in real time.

[1295] Emotion data storage and analysis

[1296] The server stores the captured text, image captures, and sentiment data in cloud storage, and uses a machine learning model to analyze the content and sentiment data and generate a summary that also reflects the results of sentiment analysis.

[1297] Summary video generation

[1298] After the meeting ends, the server generates a summary based on the stored data and identifies key moments from the meeting based on the emotion data. The server then uses a video template to automatically generate a summary video that reflects the results of the summary and emotion analysis. The summary video includes graphs and highlights that visually show changes in participants' emotions.

[1299] Providing summary videos to users

[1300] When the summary video is completed, the server sends a notification to the user, which includes a link to access the summary video. The user can click the link to watch the video. This allows the user to quickly grasp the main points of the meeting and changes in participants' emotions.

[1301] Specific examples

[1302] Let's take the example of User B setting up a weekly report meeting. In Step 1, User B adds the generation AI and emotion recognition engine to the invitation list and completes the meeting setup. From Step 2 to Step 6, the server automatically adds virtual participants to the meeting and collects and analyzes comments, materials, and emotion data from the meeting. After the meeting ends, the server generates a summary video based on the summary and emotion analysis results. Finally, User B receives a link to access the summary video and can watch the meeting highlights, which reflect the emotion data, to efficiently understand the key points of the meeting and changes in participants' emotions. This process frees User B from the hassle of recording meetings and creating minutes, and also makes it easier for them to understand participants' emotions.

[1303] The above describes a specific embodiment of the present invention. This system allows users to save the time and effort of recording meetings and creating minutes, and enables them to efficiently grasp not only important information but also changes in the emotions of participants.

[1304] The processing flow will be explained below.

[1305] Step 1:

[1306] A user logs into their own web conferencing system and sets up a new online meeting. At this time, they input information such as the meeting date and time, participant list, and invitation link, and add the generative AI (AGI) and emotion recognition engine to the invitation list as virtual participants.

[1307] Step 2:

[1308] When the meeting starts, the server automatically joins the AGI and emotion recognition engine to the meeting, which involves connecting using the meeting URL or invitation link.

[1309] Step 3:

[1310] During the meeting, the terminal (AGI projection system) converts participants' speech into text in real time using voice recognition technology, and captures presentations and materials using image capture technology.

[1311] Step 4:

[1312] Similarly, during a meeting, the emotion recognition engine analyzes the content and tone of speech to identify participants' emotions in real time, and this emotional data is instantly saved as digital data.

[1313] Step 5:

[1314] After the meeting ends, the server stores the data collected by the AGI and emotion recognition engine (speech text, document capture, and emotion analysis data) in cloud storage.

[1315] Step 6:

[1316] The server analyzes the stored data and generates a summary using the speech content and emotion data, which includes important speech content and emotional changes.

[1317] Step 7:

[1318] The server automatically generates a summary video using a video template based on the generated summary and the results of sentiment analysis, highlighting key points and important emotional moments from the meeting.

[1319] Step 8:

[1320] Once the summary video is complete, the server sends a notification to the user, which includes a link to access the summary video.

[1321] Step 9:

[1322] Users receive a notification and click the provided link to watch the summary video, which allows them to quickly grasp the key points of the meeting and the changes in participants' emotions.

[1323] Specific examples

[1324] For example, if User B schedules a weekly reporting meeting, in Step 1, the generation AI and emotion recognition engine are added to the meeting. When the meeting begins, in Steps 2 to 4, the AGI and emotion recognition engine automatically join the meeting and collect the remarks, materials, and emotion data. After the meeting ends, in Steps 5 to 7, the server generates a summary and automatically generates a summary video that reflects the emotion data. Finally, in Steps 8 and 9, User B can watch the summary video and understand the highlights of the meeting that reflect the emotion data. This process frees User B from the hassle of recording meetings and creating minutes, and also makes it easier to understand the emotions of the participants.

[1325] Example 2

[1326] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1327] While current online meeting systems allow participants to share remarks and materials during meetings, it is difficult to efficiently grasp participants' emotions and key moments. Furthermore, current systems do not automatically generate summary videos that include meeting summaries or emotion analysis results. This makes it difficult to effectively follow up on meetings and grasp key points, and reviewing the content of long meetings afterward is particularly time-consuming and labor-intensive.

[1328] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving conference setting information and automatically adding virtual participants to the conference, means for collecting participants' comments and projection materials in real time during the conference, means for analyzing the collected comment content and emotion data to generate a summary, means for automatically creating a short summary video based on the generated summary and emotion analysis results, and means for providing the summary video to the user. This makes it possible to efficiently grasp the content of the conference and changes in participants' emotions, and to quickly understand the gist of important information and emotions.

[1329] "Conference setting information" is information necessary to hold a conference, such as the date and time of the online conference, a list of participants, and an invitation link.

[1330] A "virtual participant" is an agent or program that is not physically present but is brought into an online conference by the system.

[1331] "Speech content" refers to voice data spoken by participants during an online conference that has been converted into text using voice recognition technology.

[1332] "Projected materials" are visual materials such as presentations and documents that participants share during an online meeting.

[1333] "Means of collection" refers to technologies and systems for acquiring participants' comments and projected materials in real time during online meetings.

[1334] "Speech recognition technology" is a technology for converting voice data into text data.

[1335] "Image capture technology" is a technology that captures displayed visual materials as image data.

[1336] "Emotional data" is emotional information analyzed based on the content of participants' statements and tone of voice.

[1337] A "summary" is a concise summary of the main points of a meeting based on collected speech and sentiment data.

[1338] A "summary video" is a short video generated from collected data that visually shows key moments of the meeting and the results of sentiment analysis.

[1339] "Cloud storage" is an external storage service for storing and managing data via the Internet.

[1340] "Backup measures" are techniques and methods for regularly replicating and backing up stored data.

[1341] The present invention is a system that combines an online conference with an emotion recognition engine, analyzes the content of comments and emotions of participants during the conference, and generates a summary video based on the results. A specific embodiment of this system will be described below.

[1342] System Overview

[1343] The system consists of the following elements:

[1344] 1. Initial Setup: The user enters the meeting setup information and adds the virtual participant (AGI) to the meeting invitation list.

[1345] 2. AGI joining: The server automatically joins the virtual participants to the meeting at the meeting start time.

[1346] 3. Data collection and analysis: Using emotion recognition engines and voice recognition technology, the content and emotions expressed during meetings are collected and analyzed in real time.

[1347] 4. Summary video generation: A summary video is generated based on the collected data and the results of sentiment analysis.

[1348] 5. Providing summary videos: Providing users with summary videos provides a means to efficiently understand the content and emotional gist of the meeting.

[1349] Specific Examples

[1350] 1. Initial Setup

[1351] A user logs into their online conference system and sets up a new conference. At this time, they input conference setting information such as the conference date and time, participant list, and invitation link, and add the generative AI (AGI) and emotion recognition engine to the invitation list as virtual participants.

[1352] 2. AGI participation

[1353] The server automatically joins the virtual participants (generative AI and emotion recognition engine) to the meeting at the scheduled meeting time. The server verifies that they are properly connected using the meeting URL or invitation link.

[1354] 3. Data collection and analysis

[1355] During the meeting, the device (AGI projection system) converts participants' speech into text in real time using speech recognition technology (e.g., Google Speech-to-Text API). It also captures presentations and materials shared on the screen by the user or other participants using image capture technology (e.g., screen capture software). Furthermore, an emotion recognition engine analyzes the content and tone of speech to recognize participants' emotions (e.g., joy, sadness, surprise, etc.) in real time (e.g., Azure Emotion API).

[1356] 4. Emotion Data Storage and Analysis

[1357] The server stores the captured speech text, image captures, and sentiment data in cloud storage (e.g., Amazon S3), analyzes the speech content and sentiment data using a machine learning model (e.g., the BERT model), and generates a summary that also reflects the results of sentiment analysis.

[1358] 5. Summary video generation

[1359] After the meeting ends, the server generates a summary based on the stored data and identifies key moments from the meeting based on the emotion data. The server then uses a video template to automatically generate a summary video that reflects the results of the summary and emotion analysis. The summary video includes graphs and highlights that visually show changes in participants' emotions.

[1360] 6. Providing a summary video

[1361] When the summary video is completed, the server sends a notification to the user, which includes a link to access the summary video. The user can click the link to watch the video. This allows the user to quickly grasp the main points of the meeting and changes in participants' emotions.

[1362] Specific examples

[1363] For example, consider the case where User B sets up a weekly reporting meeting. User B adds the generation AI and emotion recognition engine to the invitation list during the initial setup and completes the meeting setup. When the meeting starts, the server automatically adds virtual participants to the meeting, shares comments and materials during the meeting, and collects and analyzes emotional data. After the meeting ends, the server generates a summary video based on the summary and emotion analysis results and provides User B with an access link. By watching the summary video, User B can efficiently understand the key points of the meeting and changes in participants' emotions.

[1364] Prompt Sentence Examples

[1365] Below is an example prompt for asking a generative AI model to summarize a meeting:

[1366] Sample prompt: "Based on the audio data and sentiment analysis results from this meeting, please create a summary that reflects key statements and changes in sentiment."

[1367] This allows the generative AI model to generate a summary based on the meeting's audio data and emotion analysis results and provide it to the user.

[1368] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1369] Step 1:

[1370] A user logs into the online conference system and sets up a new conference. As input, they provide the date and time of the conference, a list of participants, an invitation link, and virtual participants (generative AI and emotion recognition engine). Based on this information, the system sets up the conference and outputs the necessary configuration information.

[1371] Step 2:

[1372] The server automatically joins the virtual participants (generative AI and emotion recognition engine) to the meeting at the scheduled meeting time. It uses the meeting invitation link and setting information as input to connect the virtual participants to the meeting. As output, it obtains confirmation information of the virtual participants who have joined the meeting.

[1373] Step 3:

[1374] During the meeting, the terminal (AGI projection system) converts participants' speech into text in real time using speech recognition technology (for example, Google Speech-to-Text API). It receives participants' speech data as input and generates text data using speech recognition technology. It obtains real-time speech text data as output.

[1375] Step 4:

[1376] The device captures presentations and materials shared on the screen by the user or other participants using image capture technology (e.g., screen capture software). It captures image data of the shared screen as input and creates the captured image data as output.

[1377] Step 5:

[1378] The emotion recognition engine analyzes the content and tone of speech during a meeting to recognize participants' emotions in real time. It receives speech text data and voice tone data as input, analyzes them, and generates emotion data. The real-time emotion data is obtained as output.

[1379] Step 6:

[1380] The server stores the acquired utterance text, image capture data, and emotion data in cloud storage (e.g., Amazon S3). It receives all collected data as input and performs the storage process. As output, it receives confirmation information for the data that has been saved.

[1381] Step 7:

[1382] The server analyzes the stored data and generates summaries. It uses the stored text and sentiment data as input and analyzes the data using a machine learning model (e.g., the BERT model). As output, it generates summary data based on the content and sentiment analysis of the speech.

[1383] Step 8:

[1384] The server automatically generates a summary video based on the summary data and the sentiment analysis results. It uses the summary data and the sentiment analysis results as input and applies a video template. As output, it generates a summary video containing key moments from the meeting.

[1385] Step 9:

[1386] The server sends a notification to the user when the summary video is complete. As input, it creates link information for the summary video and sends a notification message. As output, it sends a notification containing an access link to the summary video.

[1387] As a concrete example, User B sets up a weekly reporting meeting and follows the process described above to receive a summary video after the meeting. User B can quickly grasp the main points of the meeting and the emotions expressed, and can report efficiently.

[1388] (Application example 2)

[1389] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1390] With current customer support in virtual stores, it is difficult to accurately grasp customer emotions and respond quickly and appropriately. Furthermore, reviewing the content of conversations between customers and support staff requires a huge amount of time and effort, making it difficult to implement effective support follow-up. Furthermore, there is a high possibility that important moments in conversations based on emotional changes may be missed.

[1391] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving conference setting information and automatically adding virtual participants to the conference, means for collecting participant comments and projection materials in real time during the conference, means for analyzing the collected data and generating a summary, means for acquiring and analyzing emotion data of participants during the conference, means for automatically creating a short summary video based on the generated summary and emotion data, and means for providing the summary video to the user. This makes it possible to provide an effective summary of the content of the conversation and emotion data to improve the quality of customer support.

[1392] "Conference setting information" refers to detailed information for setting up a conference, such as a participant list including virtual participants, the date and time of the conference, and an invitation link.

[1393] A "virtual participant" is a virtual agent, such as an artificial intelligence or generative AI model, that participates in meetings in the same way as a human participant and plays a role in collecting and analyzing data.

[1394] "Utterances" are words or sentences spoken by participants during a meeting to communicate.

[1395] "Projected materials" are visual information such as slides, documents, and images that are shared by participants during presentations during a meeting.

[1396] "Collecting in real time" means capturing data simultaneously while the meeting is taking place.

[1397] A "summary" is information that briefly summarizes the contents based on the collected meeting data.

[1398] "Emotional data" is data that indicates the emotional state of participants, obtained by analyzing their facial expressions and tone of voice.

[1399] A "summary video" is a short video that includes important moments and key points from a meeting, and is presented visually, incorporating emotional data.

[1400] "Voice recognition technology" is a technology that analyzes voice and converts it into text data.

[1401] "Image capture technology" is a technology that captures images of materials and documents displayed on a screen and acquires them as data.

[1402] This invention is a system that improves the quality of customer support by analyzing the emotions of participants during online meetings between customers and customer support teams in customer support sessions at virtual stores and reflecting the results in summary videos.

[1403] The main components of the system are:

[1404] 1. A means of receiving conference setup information and automatically adding virtual participants to a conference:

[1405] The server receives conference setting information (participant list, conference date and time, invitation link, etc.) from a user who logs in to the virtual store's customer support system, and automatically adds an artificial intelligence model or emotion recognition engine to the conference as a virtual participant.

[1406] 2. A means of collecting participants' comments and projections in real time during a meeting:

[1407] The server converts speech during the meeting into text in real time using voice recognition technology (e.g., Google Speech-to-Text), and also uses image capture technology to obtain projected and shared materials.

[1408] 3. Means of analyzing collected data and generating summaries:

[1409] The server analyzes the collected speech text and projection materials using a machine learning model and generates a summary.

[1410] 4. Means of acquiring and analyzing participants' emotional data during a meeting:

[1411] The server uses an emotion recognition engine (e.g., Affectiva) to analyze the facial expressions and tone of voice of customers and customer support members in real time to obtain emotional data.

[1412] 5. A method for automatically creating short summary videos based on the generated summaries and emotion data:

[1413] The server automatically generates a short summary video that reflects important moments and emotional changes based on the generated summary and emotion data, and visually emphasizes the highlights using video generation software (e.g., Adobe Premiere Pro).

[1414] 6. Means of providing summary video to users:

[1415] Once the summary video is generated, the server notifies the user and provides an access link to the video, allowing the user to efficiently grasp the key points of the customer support session and the customer's emotional state.

[1416] Specific examples

[1417] As an example, consider setting up an online session in a virtual store where a customer seeks support for a defective product. When setting up the session, the user adds artificial intelligence and an emotion recognition engine as virtual participants. During the session, the server converts comments into text in real time and also collects projected materials. The emotion recognition engine analyzes the customer's emotions and obtains data. After the session ends, the server analyzes this data and generates a summary video containing the emotion analysis results. Finally, the user receives a link to the summary video and can take appropriate measures to improve customer satisfaction.

[1418] Prompt Sentence Examples

[1419] scenario:

[1420] During online sessions to respond to customer inquiries

[1421] 1. Analyze the customer's facial expressions and voice to determine their emotions.

[1422] 2. Convert what is said into text in real time.

[1423] 3. After the session, a summary video is generated showing key points and emotional changes.

[1424] Product:

[1425] A highlight summary video that visually displays customer sentiment data.

[1426] Technology used:

[1427] Emotion recognition engine, voice recognition technology, video generation software

[1428] Processing Points:

[1429] 1. Real-time monitoring of online conversations and sentiment between customers and customer support agents.

[1430] 2. Emotion data and text data are stored in cloud storage and then analyzed.

[1431] 3. Extract necessary scenes and generate a summary video that visually shows changes in emotions.

[1432] The above is a specific embodiment for carrying out the invention. This system is expected to improve the quality of customer support in virtual stores and increase customer satisfaction.

[1433] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1434] Step 1:

[1435] A user logs in to the virtual store's customer support system and sets up a new support session. The user enters meeting setting information such as the session date and time, participant list, and invitation link, and adds the generative AI model and emotion recognition engine as virtual participants. The entered meeting setting information is sent to the server, which then automatically adds the virtual participants based on this information and completes the setup.

[1436] Step 2:

[1437] The server automatically joins the virtual participants (generative AI model and emotion recognition engine) to the session at the set session time. The server uses the invitation link to properly connect, and as soon as the session starts, the system starts collecting data. The inputs are the invitation link and the session start time, and the output is the session participation status of the virtual participants.

[1438] Step 3:

[1439] During the session, the device uses speech recognition technology to convert speech into text data in real time. The input is the speech of each participant, which the server collects as text data. This text data is used in the subsequent analysis step. The output is text data.

[1440] Step 4:

[1441] The device uses image capture technology to capture the projected materials shared during the session in real time. The input is the shared materials on the screen, and the output is image data. The server stores this image data along with other data from the session.

[1442] Step 5:

[1443] The server uses an emotion recognition engine to analyze participants' facial expressions and tone of voice in real time to collect emotional data. The input is the participants' facial images and voices, and the output is emotional data. This emotional data is reflected in the generated summary video.

[1444] Step 6:

[1445] The server analyzes the collected speech text, projection materials, and emotion data using a machine learning model. The input is the aforementioned text data, image data, and emotion data, and a summary is generated through data analysis. The output is summary data.

[1446] Step 7:

[1447] The server automatically generates a short summary video based on the generated summary and emotion data. The input is the summary data and emotion data, and the output is a summary video. Important moments and emotional changes are visually emphasized using video generation software.

[1448] Step 8:

[1449] The server notifies the user of the generated summary video and provides an access link. The input is the summary video and the user's contact information, and the output is the notification and the access link to the video. The user can watch the summary video via this link and grasp the main points and emotional changes of the customer support session.

[1450] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1451] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1452] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1453] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1454] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1455] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1456] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1457] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1458] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1459] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1460] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1461] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1462] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1463] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1464] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1465] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1466] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1467] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1468] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1469] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1470] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1471] The following is further disclosed regarding the above embodiment.

[1472] (Claim 1)

[1473] means for receiving conference setup information and automatically adding virtual participants to the conference;

[1474] A means of collecting participants' comments and projected materials in real time during the meeting;

[1475] a means for analyzing the collected data and generating a summary;

[1476] A means for automatically creating a short summary video based on the generated summary;

[1477] A means for providing a summary video to a user;

[1478] A system including:

[1479] (Claim 2)

[1480] A means for converting speech data collected during a meeting into text using speech recognition technology;

[1481] means for acquiring the projected material by image capture technology;

[1482] The system of claim 1 further comprising:

[1483] (Claim 3)

[1484] A means for storing the collected data in cloud storage;

[1485] A means for backing up the stored data;

[1486] The system of claim 1 further comprising:

[1487] (Claim 4)

[1488] a means for using a machine learning model for summary generation;

[1489] a means for using the video template in generating a summary video based on the summary generation;

[1490] The system of claim 1 further comprising:

[1491] (Claim 5)

[1492] A means for the server to notify the user that the summary video has been completed;

[1493] a means for providing a link to access the video along with the notification;

[1494] The system of claim 1 further comprising:

[1495] "Example 1"

[1496] (Claim 1)

[1497] means for receiving conference setup information and automatically adding virtual participants to the conference;

[1498] A means of collecting participants' comments and projected materials in real time during the meeting;

[1499] A means for analyzing the collected data and generating summaries using a generative AI model; and

[1500] A means for automatically creating a short summary video based on the generated summary;

[1501] a means for providing a summary video to a user;

[1502] A system including:

[1503] (Claim 2)

[1504] A means for converting speech data collected during a meeting into text using speech recognition technology;

[1505] means for acquiring the projected material by image capture technology;

[1506] A means for storing data in cloud storage;

[1507] The system of claim 1 further comprising:

[1508] (Claim 3)

[1509] a means for initiating the summarization process using a prompt sentence when generating a summary using the generative AI model;

[1510] A means for creating a summary video using a video template based on the generated summary data;

[1511] The system of claim 1 further comprising:

[1512] "Application Example 1"

[1513] (Claim 1)

[1514] means for receiving conference setup information and automatically adding virtual participants to the conference;

[1515] A means of collecting participants' comments and projected materials in real time during the meeting;

[1516] means for analyzing and generating summaries of camera and sensor data collected during operation of the vehicle;

[1517] A means for automatically creating a short summary video based on the generated summary;

[1518] A means for providing a summary video to a user;

[1519] A system including:

[1520] (Claim 2)

[1521] A means for converting speech data collected during a meeting into text using speech recognition technology;

[1522] A means for acquiring and analyzing video from a camera mounted on the vehicle;

[1523] The system of claim 1 further comprising:

[1524] (Claim 3)

[1525] A means for storing the collected data in cloud storage;

[1526] A means for backing up the stored data;

[1527] The system of claim 1 further comprising:

[1528] "Example 2: Combining Emotion Engines"

[1529] (Claim 1)

[1530] means for receiving conference setup information and automatically adding virtual participants to the conference;

[1531] A means of collecting participants' comments and projected materials in real time during the meeting;

[1532] A means for analyzing the collected speech content and emotion data to generate a summary;

[1533] A means for automatically creating a short summary video based on the generated summary and the results of sentiment analysis;

[1534] A means for providing a summary video to a user;

[1535] Online conferencing system including.

[1536] (Claim 2)

[1537] A means for converting speech data collected during a meeting into text using speech recognition technology;

[1538] means for acquiring the projected material by image capture technology;

[1539] A means to recognize participants' emotions in real time by analyzing the content and tone of voice spoken during a meeting;

[1540] The online conference system according to claim 1, further comprising:

[1541] (Claim 3)

[1542] A means for storing the collected data and emotion data in cloud storage;

[1543] A means for backing up the stored data;

[1544] The online conference system according to claim 1, further comprising:

[1545] "Application example 2 when combining emotion engines"

[1546] (Claim 1)

[1547] means for receiving conference setup information and automatically adding virtual participants to the conference;

[1548] A means of collecting participants' comments and projected materials in real time during the meeting;

[1549] a means for analyzing the collected data and generating a summary;

[1550] A means for acquiring and analyzing emotion data of participants during a meeting;

[1551] A means for automatically creating a short summary video based on the generated summary and emotion data;

[1552] A means for providing a summary video to a user;

[1553] A system including:

[1554] (Claim 2)

[1555] A means for converting speech data collected during a meeting into text using speech recognition technology;

[1556] means for acquiring the projected material by image capture technology;

[1557] a means for analyzing participants' emotions in real time using an emotion recognition engine;

[1558] The system of claim 1 further comprising:

[1559] (Claim 3)

[1560] A means for storing the collected data in cloud storage;

[1561] A means for backing up the stored data;

[1562] The system of claim 1 further comprising: [Explanation of symbols]

[1563] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving conference setup information and automatically adding virtual participants to the conference; A means of collecting participants' comments and projected materials in real time during the meeting; a means for analyzing the collected data and generating a summary; A means for automatically creating a short summary video based on the generated summary; A means for providing a summary video to a user; A system including:

2. A means for converting speech data collected during a meeting into text using speech recognition technology; means for acquiring the projected material by image capture technology; The system of claim 1 further comprising:

3. A means for storing the collected data in cloud storage; A means for backing up the stored data; The system of claim 1 further comprising:

4. a means for using a machine learning model for summary generation; a means for using the video template in generating a summary video based on the summary generation; The system of claim 1 further comprising:

5. A means for the server to notify the user that the summary video has been completed; a means for providing a link to access the video along with the notification; The system of claim 1 further comprising:

Images