System

Abstract

A system is provided.SOLUTION: A system comprising: natural language processing means for analyzing comments received from viewers; filtering means for filtering offensive comments and spam comments from the comments analyzed by the natural language processing means; summarizing means for summarizing the comments filtered by the filtering means; and transmitting means for transmitting the comments summarized by the summarizing means to a distributor.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] When video streamers deal with comments from a large number of viewers, they have to manually filter out offensive and spam comments, which places a heavy mental burden on them. It is also difficult to efficiently analyze viewer comments and communicate with them based on summarized information. Furthermore, comments may contain many irrelevant comments or overlapping questions, making it difficult for streamers to respond efficiently. [Means for solving the problem]

[0005] The present invention provides a system including a natural language processing means for analyzing comments received from viewers, a filtering means for filtering out offensive comments and spam comments from the comments analyzed by the natural language processing means, a summarizing means for summarizing the comments filtered by the filtering means, and a sending means for sending the comments summarized by the summarizing means to a broadcaster. Furthermore, the filtering means includes a means for excluding irrelevant comments, and the summarizing means includes a means for consolidating similar comments from multiple viewers and a means for prioritizing comments based on importance, thereby enabling broadcasters to communicate with viewers more efficiently. Furthermore, the sending means sends the summarized comments to the broadcaster's terminal in real time, allowing the broadcaster to immediately respond to viewers' needs.

[0006] "Natural language processing means" refers to technology that analyzes the content of comments received from viewers and understands their meaning and emotions.

[0007] "Filtering measures" refers to technology used to automatically filter out offensive or spam comments from the analyzed comments.

[0008] "Summarization means" refers to technology that processes filtered comments, integrates important information and similar comments from many viewers, and summarizes them in a form that is useful to the broadcaster.

[0009] "Transmission means" refers to the technology that serves to transmit summarized comments and information to the broadcaster's terminal in real time.

[0010] "Irrelevant comments" refer to comments that are deemed to be of little importance to the streamer or viewer and do not require a response.

[0011] "Offensive comments" are comments that contain offensive, insulting, or disrespectful content towards other viewers or the streamer.

[0012] "Spam comments" are comments that contain irrelevant advertisements, links, offensive content or meaningless text.

[0013] "Prioritization" refers to the method of determining the order in which comments or information are presented to the broadcaster based on importance or relevance.

[0014] "Real-time" refers to data and information being processed and provided almost immediately, without delay.

[0015] "Terminal" refers to the device used by a Broadcaster to receive and display comments and summary information. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10]1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] DETAILED DESCRIPTION OF THE INVENTION The present invention will be described in detail below with reference to the accompanying drawings. The present invention relates to a system for analyzing, filtering, and summarizing comments from viewers in real time, and effectively presenting them to the broadcaster.

[0038] A natural language description of the program's operation

[0039] Server Roles

[0040] The server plays a central role in this system. When viewers send comments to the server, the server first receives them. Then, it analyzes these comments using natural language processing technology. This analysis includes sentiment analysis and understanding the content of the comments. The server then filters the comments based on the analysis results. It identifies offensive and spam comments and automatically filters them out. The server then summarizes the remaining comments. For example, if multiple viewers ask the same question, it combines them into a single unified message. Finally, the server sends the summarized comments to the broadcaster in real time.

[0041] Device Role

[0042] The terminal is used by the broadcaster to receive and display summary comments sent from the server. The broadcaster can smoothly communicate with viewers through this terminal. The terminal has an interface that presents the received summary information to the broadcaster in an easy-to-read format. When the broadcaster responds to a viewer's question, the response is also sent from the terminal to the server and is ultimately displayed to all viewers in real time.

[0043] User Roles

[0044] Users (viewers) post comments while watching a video broadcast. The comments are sent from the user's device to the server. The server analyzes and filters the user's comments, and important comments are summarized and sent to the broadcaster, so users can expect their questions to be quickly delivered to the broadcaster and answered.

[0045] Specific examples

[0046] For example, suppose a video streamer is live streaming a review of a new product. Viewer A comments, "What colors does the new product come in?", and viewer B comments, "Could you please give us a more detailed review?". At the same time, viewer C posts an offensive comment. The following process occurs:

[0047] Server Processing

[0048] 1. The server receives comments sent by viewers A, B, and C.

[0049] 2. The server uses natural language processing technology to analyze each comment and determines that comments from viewers A and B are positive, and that comments from viewer C are offensive.

[0050] 3. The server filters out viewer C's comments.

[0051] 4. The server summarizes the comments from viewers A and B and generates a summary that reads, "Viewers have asked questions about the color variations of new products and detailed reviews."

[0052] 5. The server sends this summary to the distributor's terminal.

[0053] Terminal handling

[0054] 1. The terminal receives the summary comment sent from the server.

[0055] 2. The device displays the summary comment received on the screen, and the broadcaster checks it.

[0056] 3. Based on the summary comments, the streamer will explain the color variations of the new product and provide a detailed review.

[0057] 4. The streamer's explanation is sent to the server via the device and displayed in real time on the viewer's chat screen.

[0058] This allows streamers to focus on the positive and important comments without being distracted by negative ones, and viewers appreciate that their questions are answered quickly.

[0059] The processing flow will be explained below.

[0060] Step 1:

[0061] The user creates a comment using their device and presses the send button.

[0062] Step 2:

[0063] The terminal sends this comment to the server.

[0064] Step 3:

[0065] The server receives the comment.

[0066] Step 4:

[0067] The server passes the comments to a natural language processing engine for content analysis, including sentiment analysis (positive, negative, neutral) and relevance detection of the comments.

[0068] Step 5:

[0069] The server filters comments based on the analysis results, automatically filtering out offensive, spam, and inappropriate comments.

[0070] Step 6:

[0071] The server summarizes the remaining comments, for example if the same question is asked by multiple viewers it will combine them into one unified message.

[0072] Step 7:

[0073] The server prioritizes the summarized comments based on importance.

[0074] Step 8:

[0075] The server sends prioritized summary comments to the broadcaster's terminal in real time.

[0076] Step 9:

[0077] The device displays the summary comments received on the screen, providing an interface that is intuitive for the broadcaster.

[0078] Step 10:

[0079] The streamer responds to the viewers' summary comments via the device. For example, he or she might begin explaining the "color variations of a new product" that is ranked highly.

[0080] Step 11:

[0081] The device sends the broadcaster's response to the server, which displays it on the viewer's chat screen in real time.

[0082] Step 12:

[0083] The user checks the response from the broadcaster, which gives the user satisfaction that their question was answered quickly.

[0084] Example 1

[0085] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0086] In conventional live streaming systems, streamers often become overwhelmed by the large number of comments from viewers, causing them to miss important questions and feedback. Furthermore, offensive or spam comments can increase the streamer's mental burden and hinder smooth communication. Furthermore, responding to numerous similar comments individually takes time and effort, making it difficult to respond quickly to all viewers.

[0087] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0088] In this invention, the server includes a natural language processing means for analyzing comments received from viewers, a filtering means for filtering out offensive and spam comments from the comments analyzed by the natural language processing means, and a summarizing means for summarizing the comments filtered by the filtering means. This makes it easier for broadcasters to focus on important comments and protects them from offensive and spam comments, enabling smooth broadcasting. In addition, by consolidating and summarizing a large number of similar comments, the broadcaster can quickly respond to viewers' questions and feedback.

[0089] "Natural language processing means" refers to technology for analyzing comments received from viewers and understanding and evaluating their content, including sentiment analysis, keyword extraction, and text classification.

[0090] "Filtering Methods" refers to methods for identifying and filtering out offensive or spam comments from comments analyzed by natural language processing methods, thereby enabling the Streamer to receive only appropriate comments.

[0091] "Summarization" refers to technology that consolidates multiple comments into a concise format and summarizes the main content, allowing streamers to efficiently grasp important information.

[0092] "Transmission Method" means the technology used to transmit the parsed, filtered, and summarized Comments to your device, including any network communications and data transfer protocols.

[0093] The "receiving means" refers to a technology for receiving the summary comment sent from the server on the distributor's terminal, which then displays the summarized comment on the distributor's terminal.

[0094] "Display means" refers to the interface displayed on the terminal so that the broadcaster can check the summary comments received, including screen displays, notification systems, etc.

[0095] The "response transmission means of the distributor" refers to a technology that allows the distributor to transmit a response to a viewer's comment to the server, thereby transmitting the distributor's response to the viewer.

[0096] "Means for displaying on the viewer's chat screen" refers to technology for displaying the streamer's responses on the viewer's chat screen in real time, allowing the viewer to check the streamer's responses in real time.

[0097] The following describes an embodiment of the present invention: This system analyzes, filters, and summarizes comments from viewers in real time, and effectively presents them to the broadcaster.

[0098] Server Roles

[0099] The server plays a central role in this system. Specifically, it receives comments sent by viewers and analyzes them using natural language processing technology. This analysis includes sentiment analysis and understanding of the comment content. The server filters out offensive and spam comments based on the analysis results. It has the ability to automatically filter out offensive and spam comments. The server then summarizes the remaining comments and sends them to the streamer in real time. This can be done, for example, using the Google Cloud Natural Language API.

[0100] Device Role

[0101] The terminal is used by the streamer to receive and display summary comments sent from the server. The streamer can smoothly communicate with viewers through this terminal. The terminal has an interface that presents the received summary information to the streamer in an easy-to-read format. In addition, when the streamer responds to viewers' questions, the response is also sent from the terminal to the server and is ultimately displayed to all viewers in real time. It is possible to use a dedicated application for streamers or streaming software (e.g., OBS Studio).

[0102] User Roles

[0103] Users (viewers) post comments while watching a video broadcast. The comments are sent from the user's device to the server. The server analyzes and filters the user's comments, and important comments are summarized and sent to the broadcaster. This allows users to expect their questions to be quickly delivered to the broadcaster and answered.

[0104] Specific examples

[0105] For example, suppose a video streamer is livestreaming a review of a new product. Viewer A comments, "What colors does the new product come in?", and viewer B comments, "Could you please give us a more detailed review?". At the same time, viewer C posts an offensive comment. The following process occurs:

[0106] Server Processing

[0107] 1. The server receives comments sent by viewers A, B, and C.

[0108] 2. The server analyzes each comment using natural language processing technology (e.g., Google Cloud Natural Language API) and determines the comments from viewers A and B as positive, and the comment from viewer C as offensive.

[0109] 3. The server filters out viewer C's comments.

[0110] 4. The server summarizes the comments from viewers A and B and generates a summary that reads, "Viewers have asked questions about the color variations of new products and detailed reviews."

[0111] 5. The server sends this summary to the distributor's terminal.

[0112] Terminal handling

[0113] 1. The terminal receives the summary comment sent from the server.

[0114] 2. The device displays the summary comment received on the screen, and the broadcaster checks it.

[0115] 3. Based on the summary comments, the streamer will explain the color variations of the new product and provide a detailed review.

[0116] 4. The streamer's explanation is sent to the server via the device and displayed in real time on the viewer's chat screen.

[0117] This allows streamers to focus on the positive and important comments without being distracted by negative ones, and viewers appreciate that their questions are answered quickly.

[0118] Example prompts to input to the generative AI model

[0119] Please explain in detail the process for creating a system that analyzes comments posted during video streaming, summarizes them, and provides them to the streamer.

[0120] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0121] Step 1:

[0122] Comments received

[0123] The server receives comments from viewers in real time, specifically text data sent from users' devices via network communication.

[0124] Input: Comments submitted by the user (e.g., "What are the color variations of the new product?").

[0125] Output: Received comment data.

[0126] Step 2:

[0127] Comment Analysis

[0128] The server analyzes the received comments using natural language processing technology (e.g., Google Cloud Natural Language API), including sentiment analysis and content classification.

[0129] Input: Received comment data.

[0130] Output: Parsed comment data (e.g. sentiment score, classification tags for comment content).

[0131] Step 3:

[0132] Comment Filtering

[0133] The server filters out offensive and spam comments based on the analysis results, and the filtered comments are removed.

[0134] Input: Parsed comment data.

[0135] Output: Filtered comment data (offensive and spam comments filtered out).

[0136] Step 4:

[0137] Comment Summary

[0138] The server summarizes the filtered comments, and if multiple viewers post similar questions, it merges them into a single message.

[0139] Input: Filtered comment data.

[0140] Output: Summarized comment data (e.g., "Viewers have asked about new product color variations and detailed reviews").

[0141] Step 5:

[0142] Submit summary comment

[0143] The server sends the summarized comments to the broadcaster's terminal via network communication.

[0144] Input: Summarized comment data.

[0145] Output: The summary comment submitted.

[0146] Step 6:

[0147] Summary comments received

[0148] The terminal receives the summary comment sent from the server.

[0149] Input: Summary comment data sent from the server.

[0150] Output: Received summary comment data.

[0151] Step 7:

[0152] Summary comment display

[0153] The device displays the received summary comments on the screen, allowing the broadcaster to view the comments in an easy-to-read interface.

[0154] Input: Received summary comment data.

[0155] Output: Summary comments displayed on the terminal screen.

[0156] Step 8:

[0157] Streamer's response

[0158] The broadcaster responds to viewers' questions based on the summary comments, and the responses are sent to the server via the terminal.

[0159] Input: The broadcaster's response (e.g., "The new product comes in three colors: red, blue, and green.").

[0160] Output: The publisher's response data sent to the server.

[0161] Step 9:

[0162] Response Display

[0163] The server receives the broadcaster's response and displays it in real time on the viewer's chat screen.

[0164] Input: Response data sent by the distributor.

[0165] Output: The streamer's response displayed in the viewer's chat screen.

[0166] This allows for effective management of large volumes of comments from viewers and allows streamers to respond quickly to appropriate comments.

[0167] (Application example 1)

[0168] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0169] The problem this invention aims to solve is how to enable content distributors to efficiently process large volumes of comments from viewers and quickly receive important feedback in content distribution services. In particular, it is necessary to provide a means to provide distributors with effective and immediate information when summarizing similar comments from viewers, while filtering out offensive comments, spam comments, and irrelevant comments. It is also necessary to create an environment where distributors can focus on important information by further enhancing summarization using a generative AI model.

[0170] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0171] In this invention, the server includes natural language processing means for analyzing comments received from viewers, filtering means for filtering out offensive comments and spam comments from the comments analyzed by the natural language processing means, summarizing means for summarizing the comments filtered by the filtering means, summarization enhancement means for further summarizing the comments using a generative AI model, and transmission means for transmitting the comments summarized by the summarization means and summary enhancement means to the broadcaster. This enables the broadcaster to quickly identify only important comments from a large number of comments and communicate effectively with viewers.

[0172] "Natural language processing means" refers to technology used to analyze comments received from viewers, and includes grammatical analysis and sentiment analysis.

[0173] "Filtering measures" are measures for filtering out offensive comments, spam comments, and irrelevant comments from the analyzed comments.

[0174] The "summarizing means" is a means for summarizing the comments filtered by the filtering means.

[0175] A "summarization enhancement means" is a means for further summarizing comments that have been summarized using a generative AI model.

[0176] The "transmission means" is a means for transmitting the comments summarized by the summarization means and the summary enhancement means to the distributor.

[0177] A "generative AI model" is a model that uses artificial intelligence technology to summarize comments and extract important information based on generated prompt sentences.

[0178] DETAILED DESCRIPTION OF THE INVENTION The following describes an embodiment of the present invention.

[0179] System Program

[0180] This invention provides a system for content distribution services that analyzes, filters, and summarizes comments from viewers in real time and effectively presents them to distributors. This system is mainly composed of three entities: a server, a terminal, and a user.

[0181] Server Roles

[0182] The server plays a central role in this system. When comments from viewers are sent to the server, the server first receives them. The server then analyzes these comments using natural language processing technology. The analysis includes sentiment analysis and understanding of the comment content. Specifically, it uses the Python language and libraries such as TensorFlow and spaCy. The server then filters the comments based on the analysis results. The filtering automatically excludes offensive comments, spam comments, and irrelevant comments.

[0183] The server then processes the filtered comments to summarize them. First, it uses basic summarization tools to summarize the comments, and then it further enhances the summarization using a generative AI model. This generative AI model uses the open-source AI system GPT-3 (or similar technology). It combines multiple comments into a unified message using the example prompts described below. Finally, the summarized comments are sent to the broadcaster's device.

[0184] Device Role

[0185] The device is used by the streamer to receive and display the summary comments sent from the server. Devices include smartphones, smart glasses, and head-mounted displays. These devices have an interface that presents the received summary information to the streamer in an easy-to-read format, and the visual display is achieved using front-end technologies such as React.js. When the streamer responds to viewer questions, the response is also sent from the device to the server and is ultimately displayed to all viewers in real time. In this way, the streamer can smoothly communicate with viewers in real time while focusing on important comments.

[0186] User Roles

[0187] Users (viewers) post comments while watching content broadcasts. The comments are sent from the user's device to the server. The server analyzes and filters the user's comments, and important comments are summarized and sent to the broadcaster, so users can expect their questions to be quickly delivered to the broadcaster and answered.

[0188] Specific examples

[0189] For example, when a video streamer is live streaming a review of a new product, multiple comments are posted by viewers. Viewer A comments, "What colors does the new product come in?", and viewer B comments, "Could you please give a more detailed review?". At the same time, if viewer C posts an offensive comment, the process proceeds as follows:

[0190] First, the server receives comments sent by viewers A, B, and C and analyzes them using natural language processing technology. Viewer C's comment is determined to be offensive and is filtered out. Next, the server summarizes the comments from viewers A and B and generates a summary such as, "A viewer has asked about the color variations of a new product and a detailed review." This summary is further enhanced by a generative AI model and sent to the streamer's device.

[0191] An example prompt is:

[0192] Summarize the comments below:

[0193] What colors are available in the new products?

[0194] Can you give us a more detailed review?

[0195] This allows streamers to efficiently understand important comments and facilitate smooth communication with viewers.

[0196] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0197] Step 1:

[0198] The server receives comments from viewers. Comments sent by viewers reach the server from the user's device. The input here is the viewer's comments, and the output is the comment data stored on the server.

[0199] Step 2:

[0200] The server analyzes the received comments using natural language processing. The server uses libraries such as TensorFlow and spaCy to analyze the grammar and sentiment of the comments. The input is the received comment data, and the output is the analysis results.

[0201] Step 3:

[0202] The server performs filtering based on the analysis results. Filtering methods are used to filter out offensive comments, spam comments, and irrelevant comments. The input is the analysis results, and the output is the filtered comment data.

[0203] Step 4:

[0204] The server summarizes the filtered comments using a summarization tool. The server uses libraries such as NLTK and Gensim to summarize the comments and extract important information. The input is the filtered comment data, and the output is basic summary data.

[0205] Step 5:

[0206] The server uses a generative AI model to enhance the summary. A prompt sentence is input to the generative AI model (e.g., GPT-3) to obtain the enhanced summary. The input is the basic summary data and the prompt sentence, and the output is the enhanced summary data. An example of a prompt sentence:

[0207] Summarize the comments below:

[0208] What colors are available in the new products?

[0209] Can you give us a more detailed review?

[0210] Step 6:

[0211] The server transmits the enhanced summary data to the distributor's terminal. The transmission means is used to display the summary comment on the distributor's smartphone or smart glasses. The input is the enhanced summary data, and the output is the summary comment displayed on the distributor's terminal.

[0212] Step 7:

[0213] The terminal provides an interface for the broadcaster to respond to viewers based on the summary comments. The broadcaster uses the terminal to respond to viewers' comments in real time. The input here is the summary comments and the broadcaster's response, and the output is the response data sent to the server.

[0214] Step 8:

[0215] The server receives the broadcaster's response and returns feedback to the viewer in real time. The server receives the broadcaster's response and distributes it so that it can be displayed on the user's device. The input here is the broadcaster's response data, and the output is a response message that is displayed on the viewer's device.

[0216] In this way, the system effectively analyzes, filters, and summarizes viewer comments, providing important feedback to the streamer.

[0217] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0218] As an embodiment of the present invention, this document describes a system that analyzes, filters, and summarizes viewer comments in real time, and also uses an emotion engine to perform sentiment analysis, helping broadcasters communicate more effectively.

[0219] A natural language description of the program's operation

[0220] Server Roles

[0221] The server is the core of the system. When comments from viewers are sent to the server, the server first receives them. Next, it analyzes these comments using natural language processing technology and an emotion engine. During the analysis process, the server recognizes the sentiment of the comments (positive, negative, neutral) and understands their content. The server then filters the comments based on the results of the emotion engine. Negative, spam, and offensive comments are automatically filtered out. The server then summarizes the remaining comments. For example, if the same question is asked by multiple viewers, it combines them into a single unified message. Finally, the server sends the summarized comments and their sentiment information to the broadcaster in real time.

[0222] Device Role

[0223] The terminal is used by the broadcaster to receive and display summary comments sent from the server. The broadcaster can smoothly communicate with viewers through this terminal. The terminal has an interface that presents the received summary information and emotional information in an easy-to-read format. When the broadcaster responds to viewers' questions, the response is also sent from the terminal to the server and is ultimately displayed to all viewers in real time.

[0224] User Roles

[0225] Users (viewers) post comments while watching a video broadcast. The comments are sent from the user's device to the server. The server analyzes and filters the user's comments, and important comments are summarized and sent to the broadcaster, so users can expect their questions to be promptly delivered to the broadcaster and answered.

[0226] Specific examples

[0227] For example, suppose a video streamer is live streaming a review of a new product. Viewer A comments, "What colors does the new product come in?", and viewer B comments, "Could you please give us a more detailed review?". At the same time, viewer C posts an offensive comment. The following process occurs:

[0228] Server Processing

[0229] 1. The server receives comments sent by viewers A, B, and C.

[0230] 2. The server analyzes each comment using natural language processing technology and an emotion engine, and determines that comments from viewers A and B are positive, and that comments from viewer C are negative.

[0231] 3. The server filters out and excludes Viewer C's offensive comments.

[0232] 4. The server summarizes the positive comments from viewers A and B and generates a summary that reads, "Viewers have asked questions about new product color variations and detailed reviews."

[0233] 5. The server sends this summary and positive emotion information to the distributor's device.

[0234] Terminal handling

[0235] 1. The terminal receives the summary comment sent from the server.

[0236] 2. The device displays the summary comment and emotional information received on the screen, and the streamer checks them.

[0237] 3. Based on the summary comments, the streamer will explain the color variations of the new product and provide a detailed review.

[0238] 4. The streamer's explanation is sent to the server via the device and displayed in real time on the viewer's chat screen.

[0239] This allows streamers to focus on positive and important comments without being distracted by negative comments. Viewers also enjoy the satisfaction of receiving prompt answers to their questions. This system efficiently supports communication between viewers and streamers, increasing satisfaction for both parties.

[0240] The processing flow will be explained below.

[0241] Step 1:

[0242] The user creates a comment using their device and presses the send button.

[0243] Step 2:

[0244] The terminal transmits the comment sent by the user to the server.

[0245] Step 3:

[0246] The server receives the comment.

[0247] Step 4:

[0248] The server passes the received comments to a natural language processing engine for linguistic analysis, which involves understanding the meaning, context, and relevance of the text.

[0249] Step 5:

[0250] The server uses the natural language processing engine to analyze the comments and then uses an emotion engine to determine their sentiment, categorizing them into positive, negative, or neutral.

[0251] Step 6:

[0252] The server filters comments based on the results of the sentiment engine, automatically filtering out negative sentiment, spam, and offensive comments.

[0253] Step 7:

[0254] The server summarizes the filtered comments, for example, if the same question is asked repeatedly, it will combine them into one unified message.

[0255] Step 8:

[0256] The server prioritizes the summarized comments based on their importance, placing important or popular questions at the top.

[0257] Step 9:

[0258] The server sends prioritized summary comments and their emotional information to the broadcaster's terminal in real time.

[0259] Step 10:

[0260] The device receives the summary comments sent from the server and displays them on the screen. The interface is designed to be intuitive for streamers.

[0261] Step 11:

[0262] The broadcaster responds to the viewer's summary comments via the terminal. For example, the broadcaster responds by providing detailed information about "color variations of a new product."

[0263] Step 12:

[0264] The device sends the broadcaster's response to the server, which displays it on the viewer's chat screen in real time.

[0265] Step 13:

[0266] The user checks the response from the broadcaster, which gives the user a sense of satisfaction as it confirms that their question has been answered promptly.

[0267] Example 2

[0268] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0269] Video streaming often receives a large number of comments from viewers, some of which may be offensive, spam, or irrelevant. This can make it difficult for streamers to find useful comments, hindering smooth communication with viewers. Furthermore, when multiple viewers have similar questions or opinions, streamers are required to efficiently summarize and respond to them. Therefore, a system is needed that can efficiently analyze, filter, and summarize viewer comments, allowing streamers to understand viewer sentiment in real time.

[0270] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0271] In this invention, the server includes natural language processing means for analyzing comments received from viewers, filtering means for filtering out offensive comments and spam comments from the comments analyzed by the natural language processing means, analysis means for performing sentiment analysis on the comments filtered by the filtering means, summarization means for summarizing the comments whose sentiments have been determined by the analysis means, transmission means for transmitting the comments summarized by the summarization means to the distributor, distribution means for receiving responses from the distributor and transmitting them to the viewers, and real-time processing means for processing the viewer comments and the distributor's responses in real time. This allows the viewer comments to be analyzed, filtered, and summarized efficiently, enabling the distributor to focus on important comments and respond quickly.

[0272] The "natural language processing means" is a means that uses techniques such as tokenization of comments, part-of-speech tagging, and dependency analysis to analyze comments received from viewers.

[0273] The "filtering means" is a means for filtering out offensive comments, spam comments, and irrelevant comments from the comments analyzed by the natural language processing means.

[0274] The "analysis means" is a means for performing sentiment analysis on the comments filtered by the filtering means using a sentiment engine, and classifying the comments into positive, negative, or neutral.

[0275] The "summarization means" is a means for summarizing the comments whose sentiments have been determined by the analysis means, and for integrating comments on similar themes submitted by multiple viewers into a single message.

[0276] The "transmission means" is a means for transmitting the comment summarized by the summarization means to the distributor.

[0277] The "distribution means" is a means for receiving responses from the distributor and transmitting them to the viewers.

[0278] The "real-time processing means" is a means for processing viewer comments and distributor responses in real time.

[0279] MODE FOR CARRYING OUT THE INVENTION

[0280] The present invention relates to a system for analyzing, filtering, summarizing, and analyzing sentiment in real time for comments posted by viewers during video streaming. Detailed embodiments of this system will be described below.

[0281] System Configuration

[0282] This system mainly consists of three components: the server, the terminal, and the user. The role and function of each component will be explained in order.

[0283] server

[0284] The server plays a central role in the system. It receives comments sent by viewers and performs the following steps:

[0285] 1. Natural language processing tools

[0286] The server uses natural language processing (NLP) techniques to analyze comments received from viewers. Specific software available is SpaCy or NLTK. The server uses these techniques to tokenize comments, tag them as parts of speech, and analyze dependencies.

[0287] 2. Filtering Methods

[0288] The server analyzes the comments using natural language processing and filters out offensive, spam, and irrelevant comments. Blacklists and keyword matching are used for filtering. This filtering prevents unnecessary information from reaching the broadcaster.

[0289] 3. Analysis tools

[0290] The server performs sentiment analysis on the filtered comments using a sentiment engine. Specifically, it uses sentiment analysis tools such as VADER and TextBlob to classify the sentiment of the comments as positive, negative, or neutral.

[0291] 4. Summary tools

[0292] The server summarizes the comments after sentiment analysis. If multiple viewers have similar questions or opinions, they are merged into a single unified message. For this purpose, a text summarization algorithm (e.g., BERTSUM, TextRank) is used.

[0293] 5. Transmission Method

[0294] A summary of the comments and their sentiment information is sent to the streamer, allowing the streamer to respond quickly to important comments.

[0295] 6. Distribution Method

[0296] It is a means for receiving the broadcaster's response and sending it to the viewers. This is also done in real time, and the broadcaster's response is visible to all viewers.

[0297] 7. Real-time processing methods

[0298] It processes viewer comments and streamer responses in real time, allowing the entire system to operate smoothly and enabling real-time two-way communication.

[0299] Terminal

[0300] The terminal is a device that allows the broadcaster to receive and display summary comments sent from the server. The terminal can be a general-purpose terminal such as a tablet, smartphone, or PC. The terminal has the following functions:

[0301] 1. Display summary comments

[0302] The summary comments and emotional information sent from the server are displayed in an easy-to-read format, and the broadcaster can review them and respond appropriately.

[0303] 2. Sending a Response

[0304] The responses entered by the streamer are sent from the device to the server, where they are displayed in real time to all viewers.

[0305] User

[0306] Users can post comments while watching a video broadcast. The comments are sent from the user's device to a server, where they are analyzed, filtered, summarized, and analyzed for sentiment. This allows users to expect their comments to be promptly delivered to the broadcaster and answered.

[0307] Specific examples

[0308] For example, consider a case where a video streamer is live-streaming a review of a new product. Viewer A comments, "What colors does the new product come in?", and viewer B comments, "Could you please provide a more detailed review?". At the same time, viewer C posts an offensive comment. The following process occurs:

[0309] Prompt Sentence Examples

[0310] "Analyze viewer comments about new product reviews to determine positive, negative, and neutral sentiment."

[0311] "If multiple viewers ask the same question, summarize it and combine it into one message."

[0312] In this way, the present invention is a system that can support smoother communication by efficiently analyzing, filtering, and summarizing viewer comments and providing distributors with quick and accurate information.

[0313] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0314] Step 1:

[0315] The user enters a comment in the comment input field for the live broadcast, and when the user presses the send button, the comment is sent from the user's device to the server.

[0316] Input: Comments entered by the user

[0317] Output: Comment sent to the server

[0318] Step 2:

[0319] The server receives comments sent by users.

[0320] Input: User comment

[0321] Output: Comments received

[0322] Step 3:

[0323] The server analyzes the received comments using natural language processing (NLP) libraries such as SpaCy and NLTK. Specific steps include tokenizing the comments, tagging them by part of speech, and analyzing dependencies.

[0324] Input: Received comments

[0325] Output: Analysis results (tokenized comments, part of speech information, dependencies)

[0326] Step 4:

[0327] The server filters the comments analyzed by natural language processing using filtering methods, such as blacklists and keyword matching, to filter out offensive, spam, and irrelevant comments.

[0328] Input: Analysis results (tokenized comments, part of speech information, dependencies)

[0329] Output: Filtered comments

[0330] Step 5:

[0331] The server then performs sentiment analysis on the filtered comments using tools such as VADER and TextBlob to determine whether the comment is positive, negative, or neutral.

[0332] Input: Filtered comments

[0333] Output: Sentiment analysis results (positive, negative, neutral)

[0334] Step 6:

[0335] The server uses a summarization tool to summarize comments based on the results of sentiment analysis. If the same question or opinion is posted by multiple viewers, it combines them into a single unified message. For summarization, it uses text summarization algorithms such as BERTSUM and TextRank.

[0336] Input: Sentiment analysis results

[0337] Output: Summarized comments

[0338] Step 7:

[0339] The server transmits the summarized comments and their emotional information to the distributor's terminal.

[0340] Input: Summarized comments and sentiment information

[0341] Output: Summary comments and sentiment information sent to the streamer's device

[0342] Step 8:

[0343] The terminal receives and displays the summary comment and emotion information sent from the server, and the broadcaster responds based on this.

[0344] Input: Summary comments and sentiment information sent from the server

[0345] Output: Summary comments and sentiment information displayed on the screen

[0346] Step 9:

[0347] The broadcaster answers the viewer's questions based on the summary comment and emotional information, enters the answer into the terminal, and presses the send button.

[0348] Input: Streamer's answer

[0349] Output: Answer sent from the terminal to the server

[0350] Step 10:

[0351] The server receives the broadcaster's response and distributes it to all viewers in real time. The broadcaster's response is displayed on the viewers' chat screens.

[0352] Input: Answer sent by the broadcaster

[0353] Output: The streamer's response displayed in the viewer's chat screen

[0354] These specific processing steps allow viewers' comments to be processed efficiently, and smooth communication with the broadcaster is realized.

[0355] (Application example 2)

[0356] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0357] In live streaming and content distribution services, a large number of viewer comments are often posted in a short period of time, making it difficult for the broadcaster to review all of these comments and respond appropriately. In particular, when offensive or spam comments are included, this increases the broadcaster's mental burden and hinders efficient communication. It also makes it difficult for viewers to quickly send important questions or interesting topics to the broadcaster, potentially reducing viewer satisfaction. A system that can solve these issues is needed.

[0358] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes natural language processing means for analyzing comments received from viewers, filtering means for filtering out offensive comments and spam comments from the comments analyzed by the natural language processing means, summarizing means for summarizing the comments filtered by the filtering means, transmission means for transmitting the comments summarized by the summarizing means and their emotional information to the broadcaster, and an emotion analysis engine for emotionally analyzing the viewer comments and recognizing the emotional information (positive, negative, neutral). This allows the broadcaster to focus on the target comments and respond efficiently, and important questions and feedback from viewers are conveyed to the broadcaster quickly and accurately, improving satisfaction.

[0359] "Natural language processing means" is a means for analyzing comments received from viewers, and is a technology that converts text data into a format that is easy for machines to understand and analyzes the meaning and context.

[0360] The "filtering means" is a means for filtering out offensive comments and spam comments from the comments analyzed by the natural language processing means.

[0361] The "summarizing means" is a means for concisely summarizing the comments filtered by the filtering means and conveying the summary to the distributor.

[0362] The "transmission means" is a means for transmitting the comments summarized by the summarization means and their emotional information to the distributor in real time.

[0363] An "emotion analysis engine" is an engine that analyzes viewers' comments and identifies their emotional information (positive, negative, neutral).

[0364] "Emotional information" is information indicating the emotions (positive, negative, neutral) contained in the viewer's comments.

[0365] As an embodiment of this invention, we will explain a system that analyzes, filters, and summarizes viewer comments in real time and performs sentiment analysis using an emotion engine. This system supports effective communication for broadcasters. The system is realized mainly by three components: a server, a terminal, and a user.

[0366] The server uses "natural language processing" to analyze comments received from viewers. Specifically, it uses Python 3 and utilizes the TextBlob and spacy libraries to mechanically analyze the comment text data. It then uses "filtering" on the analyzed comments. This removes offensive and spam comments, leaving only information that is easy for users to view.

[0367] The filtered comments are organized by a "summarization means." The same questions or similar feedback are combined into one, and finally transmitted in real time by a "transmission means" for transmission to the streamer. The server also uses an "emotion analysis engine" to recognize emotional information (positive, negative, neutral) contained in the viewer's comments and transmits this information to the streamer.

[0368] The terminal is used by the broadcaster to receive summary comments and emotional information sent from the server and display them on the screen in an easy-to-read format. Based on this information, the broadcaster can smoothly communicate with the viewers. For example, a device such as a smartphone or a head-mounted display (HMD) can be used.

[0369] Users (viewers) post comments while watching a video broadcast. The comments are sent from the user's device to the server, where they are analyzed, filtered, and summarized, and important comments are sent to the broadcaster. This allows users to expect their questions to be promptly received by the broadcaster and answered.

[0370] As a concrete example, consider a case where a video streamer is live-streaming a review of a new product. Viewer A comments, "What colors does the new product come in?", and viewer B comments, "Could you please give us a more detailed review?". At the same time, viewer C posts an offensive comment, and the process proceeds as follows:

[0371] 1. The server receives comments sent by viewers A, B, and C.

[0372] 2. The server analyzes each comment using natural language processing technology and an emotion engine, and determines that comments from viewers A and B are positive, and that comments from viewer C are negative.

[0373] 3. The server filters out and excludes Viewer C's offensive comments.

[0374] 4. The server summarizes the positive comments from viewers A and B and generates a summary that reads, "Viewers have asked questions about new product color variations and detailed reviews."

[0375] 5. The server sends this summary and positive emotion information to the distributor's device.

[0376] 6. Based on the summary comments, the streamer will explain the color variations of the new product and provide a detailed review.

[0377] 7. The streamer's explanation is sent to the server via the device and displayed in real time on the viewer's chat screen.

[0378] An example of a prompt to be input to the generative AI model is, "I'm asking about the color variations of a new product. I'd also like a detailed review. Please think of an explanation for this." This will help the generative AI model provide specific answers and supplementary information.

[0379] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0380] Step 1:

[0381] Users (viewers) post comments while watching the video broadcast. The comments are sent from the user's device to the server. The input is the viewer's comments, and the output is the comments sent to the server.

[0382] Step 2:

[0383] The server analyzes comments received from viewers using natural language processing. Specifically, it uses Python 3's TextBlob and spacy libraries to analyze the comment text data and determine its linguistic structure and sentiment. The input is the text data of the comments sent by viewers, and the output is the analyzed linguistic structure and sentiment information of the comments.

[0384] Step 3:

[0385] The server uses filtering techniques to filter out offensive and spam comments from the analyzed comments. Specifically, it identifies and filters out offensive and spam comments using pre-defined keywords and specific language patterns. The input is the analyzed comment text data and sentiment information, and the output is the filtered comment data.

[0386] Step 4:

[0387] The server summarizes the filtered comments using a summarization means. Specifically, it aggregates similar questions and feedbacks into a concise message. The input is the filtered comment data, and the output is the summarized comment data.

[0388] Step 5:

[0389] The server uses a sentiment analysis engine to recognize the emotional information (positive, negative, neutral) contained in viewers' comments. Specifically, it uses a machine learning algorithm to classify the emotions of the comments. The input is summarized comment data, and the output is comment data with emotional information added.

[0390] Step 6:

[0391] The server sends the summarized comments and emotional information to the broadcaster's terminal. The input is the comment data with the emotional information added, and the output is the comment information displayed on the broadcaster's terminal.

[0392] Step 7:

[0393] The terminal receives the summary comment and emotion information sent from the server and displays them on the screen. The broadcaster communicates with the viewer based on this information. The input is the summary comment and emotion information sent from the server, and the output is the comment information displayed on the screen.

[0394] Step 8:

[0395] The broadcaster responds to the viewer's comments. The response is sent from the terminal to the server and displayed in real time on the viewer's chat screen. The input is the broadcaster's response, and the output is a response message that is displayed on the viewer's chat screen.

[0396] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0397] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0398] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0399] [Second embodiment]

[0400] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0401] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0402] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0403] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0404] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0405] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0406] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0407] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0408] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0409] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0410] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0411] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0412] DETAILED DESCRIPTION OF THE INVENTION The present invention will be described in detail below with reference to the accompanying drawings. The present invention relates to a system for analyzing, filtering, and summarizing comments from viewers in real time, and effectively presenting them to the broadcaster.

[0413] A natural language description of the program's operation

[0414] Server Roles

[0415] The server plays a central role in this system. When viewers send comments to the server, the server first receives them. Then, it analyzes these comments using natural language processing technology. This analysis includes sentiment analysis and understanding the content of the comments. The server then filters the comments based on the analysis results. It identifies offensive and spam comments and automatically filters them out. The server then summarizes the remaining comments. For example, if multiple viewers ask the same question, it combines them into a single unified message. Finally, the server sends the summarized comments to the broadcaster in real time.

[0416] Device Role

[0417] The terminal is used by the broadcaster to receive and display summary comments sent from the server. The broadcaster can smoothly communicate with viewers through this terminal. The terminal has an interface that presents the received summary information to the broadcaster in an easy-to-read format. When the broadcaster responds to a viewer's question, the response is also sent from the terminal to the server and is ultimately displayed to all viewers in real time.

[0418] User Roles

[0419] Users (viewers) post comments while watching a video broadcast. The comments are sent from the user's device to the server. The server analyzes and filters the user's comments, and important comments are summarized and sent to the broadcaster, so users can expect their questions to be quickly delivered to the broadcaster and answered.

[0420] Specific examples

[0421] For example, suppose a video streamer is live streaming a review of a new product. Viewer A comments, "What colors does the new product come in?", and viewer B comments, "Could you please give us a more detailed review?". At the same time, viewer C posts an offensive comment. The following process occurs:

[0422] Server Processing

[0423] 1. The server receives comments sent by viewers A, B, and C.

[0424] 2. The server uses natural language processing technology to analyze each comment and determines that comments from viewers A and B are positive, and that comments from viewer C are offensive.

[0425] 3. The server filters out viewer C's comments.

[0426] 4. The server summarizes the comments from viewers A and B and generates a summary that reads, "Viewers have asked questions about the color variations of new products and detailed reviews."

[0427] 5. The server sends this summary to the distributor's terminal.

[0428] Terminal handling

[0429] 1. The terminal receives the summary comment sent from the server.

[0430] 2. The device displays the summary comment received on the screen, and the broadcaster checks it.

[0431] 3. Based on the summary comments, the streamer will explain the color variations of the new product and provide a detailed review.

[0432] 4. The streamer's explanation is sent to the server via the device and displayed in real time on the viewer's chat screen.

[0433] This allows streamers to focus on the positive and important comments without being distracted by negative ones, and viewers appreciate that their questions are answered quickly.

[0434] The processing flow will be explained below.

[0435] Step 1:

[0436] The user creates a comment using their device and presses the send button.

[0437] Step 2:

[0438] The terminal sends this comment to the server.

[0439] Step 3:

[0440] The server receives the comment.

[0441] Step 4:

[0442] The server passes the comments to a natural language processing engine for content analysis, including sentiment analysis (positive, negative, neutral) and relevance detection of the comments.

[0443] Step 5:

[0444] The server filters comments based on the analysis results, automatically filtering out offensive, spam, and inappropriate comments.

[0445] Step 6:

[0446] The server summarizes the remaining comments, for example if the same question is asked by multiple viewers it will combine them into one unified message.

[0447] Step 7:

[0448] The server prioritizes the summarized comments based on importance.

[0449] Step 8:

[0450] The server sends prioritized summary comments to the broadcaster's terminal in real time.

[0451] Step 9:

[0452] The device displays the summary comments received on the screen, providing an interface that is intuitive for the broadcaster.

[0453] Step 10:

[0454] The streamer responds to the viewers' summary comments via the device. For example, he or she might begin explaining the "color variations of a new product" that is ranked highly.

[0455] Step 11:

[0456] The device sends the broadcaster's response to the server, which displays it on the viewer's chat screen in real time.

[0457] Step 12:

[0458] The user checks the response from the broadcaster, which gives the user satisfaction that their question was answered quickly.

[0459] Example 1

[0460] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0461] In conventional live streaming systems, streamers often become overwhelmed by the large number of comments from viewers, causing them to miss important questions and feedback. Furthermore, offensive or spam comments can increase the streamer's mental burden and hinder smooth communication. Furthermore, responding to numerous similar comments individually takes time and effort, making it difficult to respond quickly to all viewers.

[0462] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0463] In this invention, the server includes a natural language processing means for analyzing comments received from viewers, a filtering means for filtering out offensive and spam comments from the comments analyzed by the natural language processing means, and a summarizing means for summarizing the comments filtered by the filtering means. This makes it easier for broadcasters to focus on important comments and protects them from offensive and spam comments, enabling smooth broadcasting. In addition, by consolidating and summarizing a large number of similar comments, the broadcaster can quickly respond to viewers' questions and feedback.

[0464] "Natural language processing means" refers to technology for analyzing comments received from viewers and understanding and evaluating their content, including sentiment analysis, keyword extraction, and text classification.

[0465] "Filtering Methods" refers to methods for identifying and filtering out offensive or spam comments from comments analyzed by natural language processing methods, thereby enabling the Streamer to receive only appropriate comments.

[0466] "Summarization" refers to technology that consolidates multiple comments into a concise format and summarizes the main content, allowing streamers to efficiently grasp important information.

[0467] "Transmission Method" means the technology used to transmit the parsed, filtered, and summarized Comments to your device, including any network communications and data transfer protocols.

[0468] The "receiving means" refers to a technology for receiving the summary comment sent from the server on the distributor's terminal, which then displays the summarized comment on the distributor's terminal.

[0469] "Display means" refers to the interface displayed on the terminal so that the broadcaster can check the summary comments received, including screen displays, notification systems, etc.

[0470] The "response transmission means of the distributor" refers to a technology that allows the distributor to transmit a response to a viewer's comment to the server, thereby transmitting the distributor's response to the viewer.

[0471] "Means for displaying on the viewer's chat screen" refers to technology for displaying the streamer's responses on the viewer's chat screen in real time, allowing the viewer to check the streamer's responses in real time.

[0472] The following describes an embodiment of the present invention: This system analyzes, filters, and summarizes comments from viewers in real time, and effectively presents them to the broadcaster.

[0473] Server Roles

[0474] The server plays a central role in this system. Specifically, it receives comments sent by viewers and analyzes them using natural language processing technology. This analysis includes sentiment analysis and understanding of the comment content. The server filters out offensive and spam comments based on the analysis results. It has the ability to automatically filter out offensive and spam comments. The server then summarizes the remaining comments and sends them to the streamer in real time. This can be done, for example, using the Google Cloud Natural Language API.

[0475] Device Role

[0476] The terminal is used by the streamer to receive and display summary comments sent from the server. The streamer can smoothly communicate with viewers through this terminal. The terminal has an interface that presents the received summary information to the streamer in an easy-to-read format. In addition, when the streamer responds to viewers' questions, the response is also sent from the terminal to the server and is ultimately displayed to all viewers in real time. It is possible to use a dedicated application for streamers or streaming software (e.g., OBS Studio).

[0477] User Roles

[0478] Users (viewers) post comments while watching a video broadcast. The comments are sent from the user's device to the server. The server analyzes and filters the user's comments, and important comments are summarized and sent to the broadcaster. This allows users to expect their questions to be quickly delivered to the broadcaster and answered.

[0479] Specific examples

[0480] For example, suppose a video streamer is livestreaming a review of a new product. Viewer A comments, "What colors does the new product come in?", and viewer B comments, "Could you please give us a more detailed review?". At the same time, viewer C posts an offensive comment. The following process occurs:

[0481] Server Processing

[0482] 1. The server receives comments sent by viewers A, B, and C.

[0483] 2. The server analyzes each comment using natural language processing technology (e.g., Google Cloud Natural Language API) and determines the comments from viewers A and B as positive, and the comment from viewer C as offensive.

[0484] 3. The server filters out viewer C's comments.

[0485] 4. The server summarizes the comments from viewers A and B and generates a summary that reads, "Viewers have asked questions about the color variations of new products and detailed reviews."

[0486] 5. The server sends this summary to the distributor's terminal.

[0487] Terminal handling

[0488] 1. The terminal receives the summary comment sent from the server.

[0489] 2. The device displays the summary comment received on the screen, and the broadcaster checks it.

[0490] 3. Based on the summary comments, the streamer will explain the color variations of the new product and provide a detailed review.

[0491] 4. The streamer's explanation is sent to the server via the device and displayed in real time on the viewer's chat screen.

[0492] This allows streamers to focus on the positive and important comments without being distracted by negative ones, and viewers appreciate that their questions are answered quickly.

[0493] Example prompts to input to the generative AI model

[0494] Please explain in detail the process for creating a system that analyzes comments posted during video streaming, summarizes them, and provides them to the streamer.

[0495] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0496] Step 1:

[0497] Comments received

[0498] The server receives comments from viewers in real time, specifically text data sent from users' devices via network communication.

[0499] Input: Comments submitted by the user (e.g., "What are the color variations of the new product?").

[0500] Output: Received comment data.

[0501] Step 2:

[0502] Comment Analysis

[0503] The server analyzes the received comments using natural language processing technology (e.g., Google Cloud Natural Language API), including sentiment analysis and content classification.

[0504] Input: Received comment data.

[0505] Output: Parsed comment data (e.g. sentiment score, classification tags for comment content).

[0506] Step 3:

[0507] Comment Filtering

[0508] The server filters out offensive and spam comments based on the analysis results, and the filtered comments are removed.

[0509] Input: Parsed comment data.

[0510] Output: Filtered comment data (offensive and spam comments filtered out).

[0511] Step 4:

[0512] Comment Summary

[0513] The server summarizes the filtered comments, and if multiple viewers post similar questions, it merges them into a single message.

[0514] Input: Filtered comment data.

[0515] Output: Summarized comment data (e.g., "Viewers have asked about new product color variations and detailed reviews").

[0516] Step 5:

[0517] Submit summary comment

[0518] The server sends the summarized comments to the broadcaster's terminal via network communication.

[0519] Input: Summarized comment data.

[0520] Output: The summary comment submitted.

[0521] Step 6:

[0522] Summary comments received

[0523] The terminal receives the summary comment sent from the server.

[0524] Input: Summary comment data sent from the server.

[0525] Output: Received summary comment data.

[0526] Step 7:

[0527] Summary comment display

[0528] The device displays the received summary comments on the screen, allowing the broadcaster to view the comments in an easy-to-read interface.

[0529] Input: Received summary comment data.

[0530] Output: Summary comments displayed on the terminal screen.

[0531] Step 8:

[0532] Streamer's response

[0533] The broadcaster responds to viewers' questions based on the summary comments, and the responses are sent to the server via the terminal.

[0534] Input: The broadcaster's response (e.g., "The new product comes in three colors: red, blue, and green.").

[0535] Output: The publisher's response data sent to the server.

[0536] Step 9:

[0537] Response Display

[0538] The server receives the broadcaster's response and displays it in real time on the viewer's chat screen.

[0539] Input: Response data sent by the distributor.

[0540] Output: The streamer's response displayed in the viewer's chat screen.

[0541] This allows for effective management of large volumes of comments from viewers and allows streamers to respond quickly to appropriate comments.

[0542] (Application example 1)

[0543] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0544] The problem this invention aims to solve is how to enable content distributors to efficiently process large volumes of comments from viewers and quickly receive important feedback in content distribution services. In particular, it is necessary to provide a means to provide distributors with effective and immediate information when summarizing similar comments from viewers, while filtering out offensive comments, spam comments, and irrelevant comments. It is also necessary to create an environment where distributors can focus on important information by further enhancing summarization using a generative AI model.

[0545] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0546] In this invention, the server includes natural language processing means for analyzing comments received from viewers, filtering means for filtering out offensive comments and spam comments from the comments analyzed by the natural language processing means, summarizing means for summarizing the comments filtered by the filtering means, summarization enhancement means for further summarizing the comments using a generative AI model, and transmission means for transmitting the comments summarized by the summarization means and summary enhancement means to the broadcaster. This enables the broadcaster to quickly identify only important comments from a large number of comments and communicate effectively with viewers.

[0547] "Natural language processing means" refers to technology used to analyze comments received from viewers, and includes grammatical analysis and sentiment analysis.

[0548] "Filtering measures" are measures for filtering out offensive comments, spam comments, and irrelevant comments from the analyzed comments.

[0549] The "summarizing means" is a means for summarizing the comments filtered by the filtering means.

[0550] A "summarization enhancement means" is a means for further summarizing comments that have been summarized using a generative AI model.

[0551] The "transmission means" is a means for transmitting the comments summarized by the summarization means and the summary enhancement means to the distributor.

[0552] A "generative AI model" is a model that uses artificial intelligence technology to summarize comments and extract important information based on generated prompt sentences.

[0553] DETAILED DESCRIPTION OF THE INVENTION The following describes an embodiment of the present invention.

[0554] System Program

[0555] This invention provides a system for content distribution services that analyzes, filters, and summarizes comments from viewers in real time and effectively presents them to distributors. This system is mainly composed of three entities: a server, a terminal, and a user.

[0556] Server Roles

[0557] The server plays a central role in this system. When comments from viewers are sent to the server, the server first receives them. The server then analyzes these comments using natural language processing technology. The analysis includes sentiment analysis and understanding of the comment content. Specifically, it uses the Python language and libraries such as TensorFlow and spaCy. The server then filters the comments based on the analysis results. The filtering automatically excludes offensive comments, spam comments, and irrelevant comments.

[0558] The server then processes the filtered comments to summarize them. First, it uses basic summarization tools to summarize the comments, and then it further enhances the summarization using a generative AI model. This generative AI model uses the open-source AI system GPT-3 (or similar technology). It combines multiple comments into a unified message using the example prompts described below. Finally, the summarized comments are sent to the broadcaster's device.

[0559] Device Role

[0560] The device is used by the streamer to receive and display the summary comments sent from the server. Devices include smartphones, smart glasses, and head-mounted displays. These devices have an interface that presents the received summary information to the streamer in an easy-to-read format, and the visual display is achieved using front-end technologies such as React.js. When the streamer responds to viewer questions, the response is also sent from the device to the server and is ultimately displayed to all viewers in real time. In this way, the streamer can smoothly communicate with viewers in real time while focusing on important comments.

[0561] User Roles

[0562] Users (viewers) post comments while watching content broadcasts. The comments are sent from the user's device to the server. The server analyzes and filters the user's comments, and important comments are summarized and sent to the broadcaster, so users can expect their questions to be quickly delivered to the broadcaster and answered.

[0563] Specific examples

[0564] For example, when a video streamer is live streaming a review of a new product, multiple comments are posted by viewers. Viewer A comments, "What colors does the new product come in?", and viewer B comments, "Could you please give a more detailed review?". At the same time, if viewer C posts an offensive comment, the process proceeds as follows:

[0565] First, the server receives comments sent by viewers A, B, and C and analyzes them using natural language processing technology. Viewer C's comment is determined to be offensive and is filtered out. Next, the server summarizes the comments from viewers A and B and generates a summary such as, "A viewer has asked about the color variations of a new product and a detailed review." This summary is further enhanced by a generative AI model and sent to the streamer's device.

[0566] An example prompt is:

[0567] Summarize the comments below:

[0568] What colors are available in the new products?

[0569] Can you give us a more detailed review?

[0570] This allows streamers to efficiently understand important comments and facilitate smooth communication with viewers.

[0571] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0572] Step 1:

[0573] The server receives comments from viewers. Comments sent by viewers reach the server from the user's device. The input here is the viewer's comments, and the output is the comment data stored on the server.

[0574] Step 2:

[0575] The server analyzes the received comments using natural language processing. The server uses libraries such as TensorFlow and spaCy to analyze the grammar and sentiment of the comments. The input is the received comment data, and the output is the analysis results.

[0576] Step 3:

[0577] The server performs filtering based on the analysis results. Filtering methods are used to filter out offensive comments, spam comments, and irrelevant comments. The input is the analysis results, and the output is the filtered comment data.

[0578] Step 4:

[0579] The server summarizes the filtered comments using a summarization tool. The server uses libraries such as NLTK and Gensim to summarize the comments and extract important information. The input is the filtered comment data, and the output is basic summary data.

[0580] Step 5:

[0581] The server uses a generative AI model to enhance the summary. A prompt sentence is input to the generative AI model (e.g., GPT-3) to obtain the enhanced summary. The input is the basic summary data and the prompt sentence, and the output is the enhanced summary data. An example of a prompt sentence:

[0582] Summarize the comments below:

[0583] What colors are available in the new products?

[0584] Can you give us a more detailed review?

[0585] Step 6:

[0586] The server transmits the enhanced summary data to the distributor's terminal. The transmission means is used to display the summary comment on the distributor's smartphone or smart glasses. The input is the enhanced summary data, and the output is the summary comment displayed on the distributor's terminal.

[0587] Step 7:

[0588] The terminal provides an interface for the broadcaster to respond to viewers based on the summary comments. The broadcaster uses the terminal to respond to viewers' comments in real time. The input here is the summary comments and the broadcaster's response, and the output is the response data sent to the server.

[0589] Step 8:

[0590] The server receives the broadcaster's response and returns feedback to the viewer in real time. The server receives the broadcaster's response and distributes it so that it can be displayed on the user's device. The input here is the broadcaster's response data, and the output is a response message that is displayed on the viewer's device.

[0591] In this way, the system effectively analyzes, filters, and summarizes viewer comments, providing important feedback to the streamer.

[0592] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0593] As an embodiment of the present invention, this document describes a system that analyzes, filters, and summarizes viewer comments in real time, and also uses an emotion engine to perform sentiment analysis, helping broadcasters communicate more effectively.

[0594] A natural language description of the program's operation

[0595] Server Roles

[0596] The server is the core of the system. When comments from viewers are sent to the server, the server first receives them. Next, it analyzes these comments using natural language processing technology and an emotion engine. During the analysis process, the server recognizes the sentiment of the comments (positive, negative, neutral) and understands their content. The server then filters the comments based on the results of the emotion engine. Negative, spam, and offensive comments are automatically filtered out. The server then summarizes the remaining comments. For example, if the same question is asked by multiple viewers, it combines them into a single unified message. Finally, the server sends the summarized comments and their sentiment information to the broadcaster in real time.

[0597] Device Role

[0598] The terminal is used by the broadcaster to receive and display summary comments sent from the server. The broadcaster can smoothly communicate with viewers through this terminal. The terminal has an interface that presents the received summary information and emotional information in an easy-to-read format. When the broadcaster responds to viewers' questions, the response is also sent from the terminal to the server and is ultimately displayed to all viewers in real time.

[0599] User Roles

[0600] Users (viewers) post comments while watching a video broadcast. The comments are sent from the user's device to the server. The server analyzes and filters the user's comments, and important comments are summarized and sent to the broadcaster, so users can expect their questions to be promptly delivered to the broadcaster and answered.

[0601] Specific examples

[0602] For example, suppose a video streamer is live streaming a review of a new product. Viewer A comments, "What colors does the new product come in?", and viewer B comments, "Could you please give us a more detailed review?". At the same time, viewer C posts an offensive comment. The following process occurs:

[0603] Server Processing

[0604] 1. The server receives comments sent by viewers A, B, and C.

[0605] 2. The server analyzes each comment using natural language processing technology and an emotion engine, and determines that comments from viewers A and B are positive, and that comments from viewer C are negative.

[0606] 3. The server filters out and excludes Viewer C's offensive comments.

[0607] 4. The server summarizes the positive comments from viewers A and B and generates a summary that reads, "Viewers have asked questions about new product color variations and detailed reviews."

[0608] 5. The server sends this summary and positive emotion information to the distributor's device.

[0609] Terminal handling

[0610] 1. The terminal receives the summary comment sent from the server.

[0611] 2. The device displays the summary comment and emotional information received on the screen, and the streamer checks them.

[0612] 3. Based on the summary comments, the streamer will explain the color variations of the new product and provide a detailed review.

[0613] 4. The streamer's explanation is sent to the server via the device and displayed in real time on the viewer's chat screen.

[0614] This allows streamers to focus on positive and important comments without being distracted by negative comments. Viewers also enjoy the satisfaction of receiving prompt answers to their questions. This system efficiently supports communication between viewers and streamers, increasing satisfaction for both parties.

[0615] The processing flow will be explained below.

[0616] Step 1:

[0617] The user creates a comment using their device and presses the send button.

[0618] Step 2:

[0619] The terminal transmits the comment sent by the user to the server.

[0620] Step 3:

[0621] The server receives the comment.

[0622] Step 4:

[0623] The server passes the received comments to a natural language processing engine for linguistic analysis, which involves understanding the meaning, context, and relevance of the text.

[0624] Step 5:

[0625] The server uses the natural language processing engine to analyze the comments and then uses an emotion engine to determine their sentiment, categorizing them into positive, negative, or neutral.

[0626] Step 6:

[0627] The server filters comments based on the results of the sentiment engine, automatically filtering out negative sentiment, spam, and offensive comments.

[0628] Step 7:

[0629] The server summarizes the filtered comments, for example, if the same question is asked repeatedly, it will combine them into one unified message.

[0630] Step 8:

[0631] The server prioritizes the summarized comments based on their importance, placing important or popular questions at the top.

[0632] Step 9:

[0633] The server sends prioritized summary comments and their emotional information to the broadcaster's terminal in real time.

[0634] Step 10:

[0635] The device receives the summary comments sent from the server and displays them on the screen. The interface is designed to be intuitive for streamers.

[0636] Step 11:

[0637] The broadcaster responds to the viewer's summary comments via the terminal. For example, the broadcaster responds by providing detailed information about "color variations of a new product."

[0638] Step 12:

[0639] The device sends the broadcaster's response to the server, which displays it on the viewer's chat screen in real time.

[0640] Step 13:

[0641] The user checks the response from the broadcaster, which gives the user a sense of satisfaction as it confirms that their question has been answered promptly.

[0642] Example 2

[0643] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0644] Video streaming often receives a large number of comments from viewers, some of which may be offensive, spam, or irrelevant. This can make it difficult for streamers to find useful comments, hindering smooth communication with viewers. Furthermore, when multiple viewers have similar questions or opinions, streamers are required to efficiently summarize and respond to them. Therefore, a system is needed that can efficiently analyze, filter, and summarize viewer comments, allowing streamers to understand viewer sentiment in real time.

[0645] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0646] In this invention, the server includes natural language processing means for analyzing comments received from viewers, filtering means for filtering out offensive comments and spam comments from the comments analyzed by the natural language processing means, analysis means for performing sentiment analysis on the comments filtered by the filtering means, summarization means for summarizing the comments whose sentiments have been determined by the analysis means, transmission means for transmitting the comments summarized by the summarization means to the distributor, distribution means for receiving responses from the distributor and transmitting them to the viewers, and real-time processing means for processing the viewer comments and the distributor's responses in real time. This allows the viewer comments to be analyzed, filtered, and summarized efficiently, enabling the distributor to focus on important comments and respond quickly.

[0647] The "natural language processing means" is a means that uses techniques such as tokenization of comments, part-of-speech tagging, and dependency analysis to analyze comments received from viewers.

[0648] The "filtering means" is a means for filtering out offensive comments, spam comments, and irrelevant comments from the comments analyzed by the natural language processing means.

[0649] The "analysis means" is a means for performing sentiment analysis on the comments filtered by the filtering means using a sentiment engine, and classifying the comments into positive, negative, or neutral.

[0650] The "summarization means" is a means for summarizing the comments whose sentiments have been determined by the analysis means, and for integrating comments on similar themes submitted by multiple viewers into a single message.

[0651] The "transmission means" is a means for transmitting the comment summarized by the summarization means to the distributor.

[0652] The "distribution means" is a means for receiving responses from the distributor and transmitting them to the viewers.

[0653] The "real-time processing means" is a means for processing viewer comments and distributor responses in real time.

[0654] MODE FOR CARRYING OUT THE INVENTION

[0655] The present invention relates to a system for analyzing, filtering, summarizing, and analyzing sentiment in real time for comments posted by viewers during video streaming. Detailed embodiments of this system will be described below.

[0656] System Configuration

[0657] This system mainly consists of three components: the server, the terminal, and the user. The role and function of each component will be explained in order.

[0658] server

[0659] The server plays a central role in the system. It receives comments sent by viewers and performs the following steps:

[0660] 1. Natural language processing tools

[0661] The server uses natural language processing (NLP) techniques to analyze comments received from viewers. Specific software available is SpaCy or NLTK. The server uses these techniques to tokenize comments, tag them as parts of speech, and analyze dependencies.

[0662] 2. Filtering Methods

[0663] The server analyzes the comments using natural language processing and filters out offensive, spam, and irrelevant comments. Blacklists and keyword matching are used for filtering. This filtering prevents unnecessary information from reaching the broadcaster.

[0664] 3. Analysis tools

[0665] The server performs sentiment analysis on the filtered comments using a sentiment engine. Specifically, it uses sentiment analysis tools such as VADER and TextBlob to classify the sentiment of the comments as positive, negative, or neutral.

[0666] 4. Summary tools

[0667] The server summarizes the comments after sentiment analysis. If multiple viewers have similar questions or opinions, they are merged into a single unified message. For this purpose, a text summarization algorithm (e.g., BERTSUM, TextRank) is used.

[0668] 5. Transmission Method

[0669] A summary of the comments and their sentiment information is sent to the streamer, allowing the streamer to respond quickly to important comments.

[0670] 6. Distribution Method

[0671] It is a means for receiving the broadcaster's response and sending it to the viewers. This is also done in real time, and the broadcaster's response is visible to all viewers.

[0672] 7. Real-time processing methods

[0673] It processes viewer comments and streamer responses in real time, allowing the entire system to operate smoothly and enabling real-time two-way communication.

[0674] Terminal

[0675] The terminal is a device that allows the broadcaster to receive and display summary comments sent from the server. The terminal can be a general-purpose terminal such as a tablet, smartphone, or PC. The terminal has the following functions:

[0676] 1. Display summary comments

[0677] The summary comments and emotional information sent from the server are displayed in an easy-to-read format, and the broadcaster can review them and respond appropriately.

[0678] 2. Sending a Response

[0679] The responses entered by the streamer are sent from the device to the server, where they are displayed in real time to all viewers.

[0680] User

[0681] Users can post comments while watching a video broadcast. The comments are sent from the user's device to a server, where they are analyzed, filtered, summarized, and analyzed for sentiment. This allows users to expect their comments to be promptly delivered to the broadcaster and answered.

[0682] Specific examples

[0683] For example, consider a case where a video streamer is live-streaming a review of a new product. Viewer A comments, "What colors does the new product come in?", and viewer B comments, "Could you please provide a more detailed review?". At the same time, viewer C posts an offensive comment. The following process occurs:

[0684] Prompt Sentence Examples

[0685] "Analyze viewer comments about new product reviews to determine positive, negative, and neutral sentiment."

[0686] "If multiple viewers ask the same question, summarize it and combine it into one message."

[0687] In this way, the present invention is a system that can support smoother communication by efficiently analyzing, filtering, and summarizing viewer comments and providing distributors with quick and accurate information.

[0688] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0689] Step 1:

[0690] The user enters a comment in the comment input field for the live broadcast, and when the user presses the send button, the comment is sent from the user's device to the server.

[0691] Input: Comments entered by the user

[0692] Output: Comment sent to the server

[0693] Step 2:

[0694] The server receives comments sent by users.

[0695] Input: User comment

[0696] Output: Comments received

[0697] Step 3:

[0698] The server analyzes the received comments using natural language processing (NLP) libraries such as SpaCy and NLTK. Specific steps include tokenizing the comments, tagging them by part of speech, and analyzing dependencies.

[0699] Input: Received comments

[0700] Output: Analysis results (tokenized comments, part of speech information, dependencies)

[0701] Step 4:

[0702] The server filters the comments analyzed by natural language processing using filtering methods, such as blacklists and keyword matching, to filter out offensive, spam, and irrelevant comments.

[0703] Input: Analysis results (tokenized comments, part of speech information, dependencies)

[0704] Output: Filtered comments

[0705] Step 5:

[0706] The server then performs sentiment analysis on the filtered comments using tools such as VADER and TextBlob to determine whether the comment is positive, negative, or neutral.

[0707] Input: Filtered comments

[0708] Output: Sentiment analysis results (positive, negative, neutral)

[0709] Step 6:

[0710] The server uses a summarization tool to summarize comments based on the results of sentiment analysis. If the same question or opinion is posted by multiple viewers, it combines them into a single unified message. For summarization, it uses text summarization algorithms such as BERTSUM and TextRank.

[0711] Input: Sentiment analysis results

[0712] Output: Summarized comments

[0713] Step 7:

[0714] The server transmits the summarized comments and their emotional information to the distributor's terminal.

[0715] Input: Summarized comments and sentiment information

[0716] Output: Summary comments and sentiment information sent to the streamer's device

[0717] Step 8:

[0718] The terminal receives and displays the summary comment and emotion information sent from the server, and the broadcaster responds based on this.

[0719] Input: Summary comments and sentiment information sent from the server

[0720] Output: Summary comments and sentiment information displayed on the screen

[0721] Step 9:

[0722] The broadcaster answers the viewer's questions based on the summary comment and emotional information, enters the answer into the terminal, and presses the send button.

[0723] Input: Streamer's answer

[0724] Output: Answer sent from the terminal to the server

[0725] Step 10:

[0726] The server receives the broadcaster's response and distributes it to all viewers in real time. The broadcaster's response is displayed on the viewers' chat screens.

[0727] Input: Answer sent by the broadcaster

[0728] Output: The streamer's response displayed in the viewer's chat screen

[0729] These specific processing steps allow viewers' comments to be processed efficiently, and smooth communication with the broadcaster is realized.

[0730] (Application example 2)

[0731] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0732] In live streaming and content distribution services, a large number of viewer comments are often posted in a short period of time, making it difficult for the broadcaster to review all of these comments and respond appropriately. In particular, when offensive or spam comments are included, this increases the broadcaster's mental burden and hinders efficient communication. It also makes it difficult for viewers to quickly send important questions or interesting topics to the broadcaster, potentially reducing viewer satisfaction. A system that can solve these issues is needed.

[0733] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes natural language processing means for analyzing comments received from viewers, filtering means for filtering out offensive comments and spam comments from the comments analyzed by the natural language processing means, summarizing means for summarizing the comments filtered by the filtering means, transmission means for transmitting the comments summarized by the summarizing means and their emotional information to the broadcaster, and an emotion analysis engine for emotionally analyzing the viewer comments and recognizing the emotional information (positive, negative, neutral). This allows the broadcaster to focus on the target comments and respond efficiently, and important questions and feedback from viewers are conveyed to the broadcaster quickly and accurately, improving satisfaction.

[0734] "Natural language processing means" is a means for analyzing comments received from viewers, and is a technology that converts text data into a format that is easy for machines to understand and analyzes the meaning and context.

[0735] The "filtering means" is a means for filtering out offensive comments and spam comments from the comments analyzed by the natural language processing means.

[0736] The "summarizing means" is a means for concisely summarizing the comments filtered by the filtering means and conveying the summary to the distributor.

[0737] The "transmission means" is a means for transmitting the comments summarized by the summarization means and their emotional information to the distributor in real time.

[0738] An "emotion analysis engine" is an engine that analyzes viewers' comments and identifies their emotional information (positive, negative, neutral).

[0739] "Emotional information" is information indicating the emotions (positive, negative, neutral) contained in the viewer's comments.

[0740] As an embodiment of this invention, we will explain a system that analyzes, filters, and summarizes viewer comments in real time and performs sentiment analysis using an emotion engine. This system supports effective communication for broadcasters. The system is realized mainly by three components: a server, a terminal, and a user.

[0741] The server uses "natural language processing" to analyze comments received from viewers. Specifically, it uses Python 3 and utilizes the TextBlob and spacy libraries to mechanically analyze the comment text data. It then uses "filtering" on the analyzed comments. This removes offensive and spam comments, leaving only information that is easy for users to view.

[0742] The filtered comments are organized by a "summarization means." The same questions or similar feedback are combined into one, and finally transmitted in real time by a "transmission means" for transmission to the streamer. The server also uses an "emotion analysis engine" to recognize emotional information (positive, negative, neutral) contained in the viewer's comments and transmits this information to the streamer.

[0743] The terminal is used by the broadcaster to receive summary comments and emotional information sent from the server and display them on the screen in an easy-to-read format. Based on this information, the broadcaster can smoothly communicate with the viewers. For example, a device such as a smartphone or a head-mounted display (HMD) can be used.

[0744] Users (viewers) post comments while watching a video broadcast. The comments are sent from the user's device to the server, where they are analyzed, filtered, and summarized, and important comments are sent to the broadcaster. This allows users to expect their questions to be promptly received by the broadcaster and answered.

[0745] As a concrete example, consider a case where a video streamer is live-streaming a review of a new product. Viewer A comments, "What colors does the new product come in?", and viewer B comments, "Could you please give us a more detailed review?". At the same time, viewer C posts an offensive comment, and the process proceeds as follows:

[0746] 1. The server receives comments sent by viewers A, B, and C.

[0747] 2. The server analyzes each comment using natural language processing technology and an emotion engine, and determines that comments from viewers A and B are positive, and that comments from viewer C are negative.

[0748] 3. The server filters out and excludes Viewer C's offensive comments.

[0749] 4. The server summarizes the positive comments from viewers A and B and generates a summary that reads, "Viewers have asked questions about new product color variations and detailed reviews."

[0750] 5. The server sends this summary and positive emotion information to the distributor's device.

[0751] 6. Based on the summary comments, the streamer will explain the color variations of the new product and provide a detailed review.

[0752] 7. The streamer's explanation is sent to the server via the device and displayed in real time on the viewer's chat screen.

[0753] An example of a prompt to be input to the generative AI model is, "I'm asking about the color variations of a new product. I'd also like a detailed review. Please think of an explanation for this." This will help the generative AI model provide specific answers and supplementary information.

[0754] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0755] Step 1:

[0756] Users (viewers) post comments while watching the video broadcast. The comments are sent from the user's device to the server. The input is the viewer's comments, and the output is the comments sent to the server.

[0757] Step 2:

[0758] The server analyzes comments received from viewers using natural language processing. Specifically, it uses Python 3's TextBlob and spacy libraries to analyze the comment text data and determine its linguistic structure and sentiment. The input is the text data of the comments sent by viewers, and the output is the analyzed linguistic structure and sentiment information of the comments.

[0759] Step 3:

[0760] The server uses filtering techniques to filter out offensive and spam comments from the analyzed comments. Specifically, it identifies and filters out offensive and spam comments using pre-defined keywords and specific language patterns. The input is the analyzed comment text data and sentiment information, and the output is the filtered comment data.

[0761] Step 4:

[0762] The server summarizes the filtered comments using a summarization means. Specifically, it aggregates similar questions and feedbacks into a concise message. The input is the filtered comment data, and the output is the summarized comment data.

[0763] Step 5:

[0764] The server uses a sentiment analysis engine to recognize the emotional information (positive, negative, neutral) contained in viewers' comments. Specifically, it uses a machine learning algorithm to classify the emotions of the comments. The input is summarized comment data, and the output is comment data with emotional information added.

[0765] Step 6:

[0766] The server sends the summarized comments and emotional information to the broadcaster's terminal. The input is the comment data with the emotional information added, and the output is the comment information displayed on the broadcaster's terminal.

[0767] Step 7:

[0768] The terminal receives the summary comment and emotion information sent from the server and displays them on the screen. The broadcaster communicates with the viewer based on this information. The input is the summary comment and emotion information sent from the server, and the output is the comment information displayed on the screen.

[0769] Step 8:

[0770] The broadcaster responds to the viewer's comments. The response is sent from the terminal to the server and displayed in real time on the viewer's chat screen. The input is the broadcaster's response, and the output is a response message that is displayed on the viewer's chat screen.

[0771] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0772] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0773] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0774] [Third embodiment]

[0775] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0776] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0777] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0778] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0779] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0780] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0781] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0782] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0783] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0784] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0785] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0786] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0787] DETAILED DESCRIPTION OF THE INVENTION The present invention will be described in detail below with reference to the accompanying drawings. The present invention relates to a system for analyzing, filtering, and summarizing comments from viewers in real time, and effectively presenting them to the broadcaster.

[0788] A natural language description of the program's operation

[0789] Server Roles

[0790] The server plays a central role in this system. When viewers send comments to the server, the server first receives them. Then, it analyzes these comments using natural language processing technology. This analysis includes sentiment analysis and understanding the content of the comments. The server then filters the comments based on the analysis results. It identifies offensive and spam comments and automatically filters them out. The server then summarizes the remaining comments. For example, if multiple viewers ask the same question, it combines them into a single unified message. Finally, the server sends the summarized comments to the broadcaster in real time.

[0791] Device Role

[0792] The terminal is used by the broadcaster to receive and display summary comments sent from the server. The broadcaster can smoothly communicate with viewers through this terminal. The terminal has an interface that presents the received summary information to the broadcaster in an easy-to-read format. When the broadcaster responds to a viewer's question, the response is also sent from the terminal to the server and is ultimately displayed to all viewers in real time.

[0793] User Roles

[0794] Users (viewers) post comments while watching a video broadcast. The comments are sent from the user's device to the server. The server analyzes and filters the user's comments, and important comments are summarized and sent to the broadcaster, so users can expect their questions to be quickly delivered to the broadcaster and answered.

[0795] Specific examples

[0796] For example, suppose a video streamer is live streaming a review of a new product. Viewer A comments, "What colors does the new product come in?", and viewer B comments, "Could you please give us a more detailed review?". At the same time, viewer C posts an offensive comment. The following process occurs:

[0797] Server Processing

[0798] 1. The server receives comments sent by viewers A, B, and C.

[0799] 2. The server uses natural language processing technology to analyze each comment and determines that comments from viewers A and B are positive, and that comments from viewer C are offensive.

[0800] 3. The server filters out viewer C's comments.

[0801] 4. The server summarizes the comments from viewers A and B and generates a summary that reads, "Viewers have asked questions about the color variations of new products and detailed reviews."

[0802] 5. The server sends this summary to the distributor's terminal.

[0803] Terminal handling

[0804] 1. The terminal receives the summary comment sent from the server.

[0805] 2. The device displays the summary comment received on the screen, and the broadcaster checks it.

[0806] 3. Based on the summary comments, the streamer will explain the color variations of the new product and provide a detailed review.

[0807] 4. The streamer's explanation is sent to the server via the device and displayed in real time on the viewer's chat screen.

[0808] This allows streamers to focus on the positive and important comments without being distracted by negative ones, and viewers appreciate that their questions are answered quickly.

[0809] The processing flow will be explained below.

[0810] Step 1:

[0811] The user creates a comment using their device and presses the send button.

[0812] Step 2:

[0813] The terminal sends this comment to the server.

[0814] Step 3:

[0815] The server receives the comment.

[0816] Step 4:

[0817] The server passes the comments to a natural language processing engine for content analysis, including sentiment analysis (positive, negative, neutral) and relevance detection of the comments.

[0818] Step 5:

[0819] The server filters comments based on the analysis results, automatically filtering out offensive, spam, and inappropriate comments.

[0820] Step 6:

[0821] The server summarizes the remaining comments, for example if the same question is asked by multiple viewers it will combine them into one unified message.

[0822] Step 7:

[0823] The server prioritizes the summarized comments based on importance.

[0824] Step 8:

[0825] The server sends prioritized summary comments to the broadcaster's terminal in real time.

[0826] Step 9:

[0827] The device displays the summary comments received on the screen, providing an interface that is intuitive for the broadcaster.

[0828] Step 10:

[0829] The streamer responds to the viewers' summary comments via the device. For example, he or she might begin explaining the "color variations of a new product" that is ranked highly.

[0830] Step 11:

[0831] The device sends the broadcaster's response to the server, which displays it on the viewer's chat screen in real time.

[0832] Step 12:

[0833] The user checks the response from the broadcaster, which gives the user satisfaction that their question was answered quickly.

[0834] Example 1

[0835] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0836] In conventional live streaming systems, streamers often become overwhelmed by the large number of comments from viewers, causing them to miss important questions and feedback. Furthermore, offensive or spam comments can increase the streamer's mental burden and hinder smooth communication. Furthermore, responding to numerous similar comments individually takes time and effort, making it difficult to respond quickly to all viewers.

[0837] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0838] In this invention, the server includes a natural language processing means for analyzing comments received from viewers, a filtering means for filtering out offensive and spam comments from the comments analyzed by the natural language processing means, and a summarizing means for summarizing the comments filtered by the filtering means. This makes it easier for broadcasters to focus on important comments and protects them from offensive and spam comments, enabling smooth broadcasting. In addition, by consolidating and summarizing a large number of similar comments, the broadcaster can quickly respond to viewers' questions and feedback.

[0839] "Natural language processing means" refers to technology for analyzing comments received from viewers and understanding and evaluating their content, including sentiment analysis, keyword extraction, and text classification.

[0840] "Filtering Methods" refers to methods for identifying and filtering out offensive or spam comments from comments analyzed by natural language processing methods, thereby enabling the Streamer to receive only appropriate comments.

[0841] "Summarization" refers to technology that consolidates multiple comments into a concise format and summarizes the main content, allowing streamers to efficiently grasp important information.

[0842] "Transmission Method" means the technology used to transmit the parsed, filtered, and summarized Comments to your device, including any network communications and data transfer protocols.

[0843] The "receiving means" refers to a technology for receiving the summary comment sent from the server on the distributor's terminal, which then displays the summarized comment on the distributor's terminal.

[0844] "Display means" refers to the interface displayed on the terminal so that the broadcaster can check the summary comments received, including screen displays, notification systems, etc.

[0845] The "response transmission means of the distributor" refers to a technology that allows the distributor to transmit a response to a viewer's comment to the server, thereby transmitting the distributor's response to the viewer.

[0846] "Means for displaying on the viewer's chat screen" refers to technology for displaying the streamer's responses on the viewer's chat screen in real time, allowing the viewer to check the streamer's responses in real time.

[0847] The following describes an embodiment of the present invention: This system analyzes, filters, and summarizes comments from viewers in real time, and effectively presents them to the broadcaster.

[0848] Server Roles

[0849] The server plays a central role in this system. Specifically, it receives comments sent by viewers and analyzes them using natural language processing technology. This analysis includes sentiment analysis and understanding of the comment content. The server filters out offensive and spam comments based on the analysis results. It has the ability to automatically filter out offensive and spam comments. The server then summarizes the remaining comments and sends them to the streamer in real time. This can be done, for example, using the Google Cloud Natural Language API.

[0850] Device Role

[0851] The terminal is used by the streamer to receive and display summary comments sent from the server. The streamer can smoothly communicate with viewers through this terminal. The terminal has an interface that presents the received summary information to the streamer in an easy-to-read format. In addition, when the streamer responds to viewers' questions, the response is also sent from the terminal to the server and is ultimately displayed to all viewers in real time. It is possible to use a dedicated application for streamers or streaming software (e.g., OBS Studio).

[0852] User Roles

[0853] Users (viewers) post comments while watching a video broadcast. The comments are sent from the user's device to the server. The server analyzes and filters the user's comments, and important comments are summarized and sent to the broadcaster. This allows users to expect their questions to be quickly delivered to the broadcaster and answered.

[0854] Specific examples

[0855] For example, suppose a video streamer is livestreaming a review of a new product. Viewer A comments, "What colors does the new product come in?", and viewer B comments, "Could you please give us a more detailed review?". At the same time, viewer C posts an offensive comment. The following process occurs:

[0856] Server Processing

[0857] 1. The server receives comments sent by viewers A, B, and C.

[0858] 2. The server analyzes each comment using natural language processing technology (e.g., Google Cloud Natural Language API) and determines the comments from viewers A and B as positive, and the comment from viewer C as offensive.

[0859] 3. The server filters out viewer C's comments.

[0860] 4. The server summarizes the comments from viewers A and B and generates a summary that reads, "Viewers have asked questions about the color variations of new products and detailed reviews."

[0861] 5. The server sends this summary to the distributor's terminal.

[0862] Terminal handling

[0863] 1. The terminal receives the summary comment sent from the server.

[0864] 2. The device displays the summary comment received on the screen, and the broadcaster checks it.

[0865] 3. Based on the summary comments, the streamer will explain the color variations of the new product and provide a detailed review.

[0866] 4. The streamer's explanation is sent to the server via the device and displayed in real time on the viewer's chat screen.

[0867] This allows streamers to focus on the positive and important comments without being distracted by negative ones, and viewers appreciate that their questions are answered quickly.

[0868] Example prompts to input to the generative AI model

[0869] Please explain in detail the process for creating a system that analyzes comments posted during video streaming, summarizes them, and provides them to the streamer.

[0870] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0871] Step 1:

[0872] Comments received

[0873] The server receives comments from viewers in real time, specifically text data sent from users' devices via network communication.

[0874] Input: Comments submitted by the user (e.g., "What are the color variations of the new product?").

[0875] Output: Received comment data.

[0876] Step 2:

[0877] Comment Analysis

[0878] The server analyzes the received comments using natural language processing technology (e.g., Google Cloud Natural Language API), including sentiment analysis and content classification.

[0879] Input: Received comment data.

[0880] Output: Parsed comment data (e.g. sentiment score, classification tags for comment content).

[0881] Step 3:

[0882] Comment Filtering

[0883] The server filters out offensive and spam comments based on the analysis results, and the filtered comments are removed.

[0884] Input: Parsed comment data.

[0885] Output: Filtered comment data (offensive and spam comments filtered out).

[0886] Step 4:

[0887] Comment Summary

[0888] The server summarizes the filtered comments, and if multiple viewers post similar questions, it merges them into a single message.

[0889] Input: Filtered comment data.

[0890] Output: Summarized comment data (e.g., "Viewers have asked about new product color variations and detailed reviews").

[0891] Step 5:

[0892] Submit summary comment

[0893] The server sends the summarized comments to the broadcaster's terminal via network communication.

[0894] Input: Summarized comment data.

[0895] Output: The summary comment submitted.

[0896] Step 6:

[0897] Summary comments received

[0898] The terminal receives the summary comment sent from the server.

[0899] Input: Summary comment data sent from the server.

[0900] Output: Received summary comment data.

[0901] Step 7:

[0902] Summary comment display

[0903] The device displays the received summary comments on the screen, allowing the broadcaster to view the comments in an easy-to-read interface.

[0904] Input: Received summary comment data.

[0905] Output: Summary comments displayed on the terminal screen.

[0906] Step 8:

[0907] Streamer's response

[0908] The broadcaster responds to viewers' questions based on the summary comments, and the responses are sent to the server via the terminal.

[0909] Input: The broadcaster's response (e.g., "The new product comes in three colors: red, blue, and green.").

[0910] Output: The publisher's response data sent to the server.

[0911] Step 9:

[0912] Response Display

[0913] The server receives the broadcaster's response and displays it in real time on the viewer's chat screen.

[0914] Input: Response data sent by the distributor.

[0915] Output: The streamer's response displayed in the viewer's chat screen.

[0916] This allows for effective management of large volumes of comments from viewers and allows streamers to respond quickly to appropriate comments.

[0917] (Application example 1)

[0918] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0919] The problem this invention aims to solve is how to enable content distributors to efficiently process large volumes of comments from viewers and quickly receive important feedback in content distribution services. In particular, it is necessary to provide a means to provide distributors with effective and immediate information when summarizing similar comments from viewers, while filtering out offensive comments, spam comments, and irrelevant comments. It is also necessary to create an environment where distributors can focus on important information by further enhancing summarization using a generative AI model.

[0920] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0921] In this invention, the server includes natural language processing means for analyzing comments received from viewers, filtering means for filtering out offensive comments and spam comments from the comments analyzed by the natural language processing means, summarizing means for summarizing the comments filtered by the filtering means, summarization enhancement means for further summarizing the comments using a generative AI model, and transmission means for transmitting the comments summarized by the summarization means and summary enhancement means to the broadcaster. This enables the broadcaster to quickly identify only important comments from a large number of comments and communicate effectively with viewers.

[0922] "Natural language processing means" refers to technology used to analyze comments received from viewers, and includes grammatical analysis and sentiment analysis.

[0923] "Filtering measures" are measures for filtering out offensive comments, spam comments, and irrelevant comments from the analyzed comments.

[0924] The "summarizing means" is a means for summarizing the comments filtered by the filtering means.

[0925] A "summarization enhancement means" is a means for further summarizing comments that have been summarized using a generative AI model.

[0926] The "transmission means" is a means for transmitting the comments summarized by the summarization means and the summary enhancement means to the distributor.

[0927] A "generative AI model" is a model that uses artificial intelligence technology to summarize comments and extract important information based on generated prompt sentences.

[0928] DETAILED DESCRIPTION OF THE INVENTION The following describes an embodiment of the present invention.

[0929] System Program

[0930] This invention provides a system for content distribution services that analyzes, filters, and summarizes comments from viewers in real time and effectively presents them to distributors. This system is mainly composed of three entities: a server, a terminal, and a user.

[0931] Server Roles

[0932] The server plays a central role in this system. When comments from viewers are sent to the server, the server first receives them. The server then analyzes these comments using natural language processing technology. The analysis includes sentiment analysis and understanding of the comment content. Specifically, it uses the Python language and libraries such as TensorFlow and spaCy. The server then filters the comments based on the analysis results. The filtering automatically excludes offensive comments, spam comments, and irrelevant comments.

[0933] The server then processes the filtered comments to summarize them. First, it uses basic summarization tools to summarize the comments, and then it further enhances the summarization using a generative AI model. This generative AI model uses the open-source AI system GPT-3 (or similar technology). It combines multiple comments into a unified message using the example prompts described below. Finally, the summarized comments are sent to the broadcaster's device.

[0934] Device Role

[0935] The device is used by the streamer to receive and display the summary comments sent from the server. Devices include smartphones, smart glasses, and head-mounted displays. These devices have an interface that presents the received summary information to the streamer in an easy-to-read format, and the visual display is achieved using front-end technologies such as React.js. When the streamer responds to viewer questions, the response is also sent from the device to the server and is ultimately displayed to all viewers in real time. In this way, the streamer can smoothly communicate with viewers in real time while focusing on important comments.

[0936] User Roles

[0937] Users (viewers) post comments while watching content broadcasts. The comments are sent from the user's device to the server. The server analyzes and filters the user's comments, and important comments are summarized and sent to the broadcaster, so users can expect their questions to be quickly delivered to the broadcaster and answered.

[0938] Specific examples

[0939] For example, when a video streamer is live streaming a review of a new product, multiple comments are posted by viewers. Viewer A comments, "What colors does the new product come in?", and viewer B comments, "Could you please give a more detailed review?". At the same time, if viewer C posts an offensive comment, the process proceeds as follows:

[0940] First, the server receives comments sent by viewers A, B, and C and analyzes them using natural language processing technology. Viewer C's comment is determined to be offensive and is filtered out. Next, the server summarizes the comments from viewers A and B and generates a summary such as, "A viewer has asked about the color variations of a new product and a detailed review." This summary is further enhanced by a generative AI model and sent to the streamer's device.

[0941] An example prompt is:

[0942] Summarize the comments below:

[0943] What colors are available in the new products?

[0944] Can you give us a more detailed review?

[0945] This allows streamers to efficiently understand important comments and facilitate smooth communication with viewers.

[0946] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0947] Step 1:

[0948] The server receives comments from viewers. Comments sent by viewers reach the server from the user's device. The input here is the viewer's comments, and the output is the comment data stored on the server.

[0949] Step 2:

[0950] The server analyzes the received comments using natural language processing. The server uses libraries such as TensorFlow and spaCy to analyze the grammar and sentiment of the comments. The input is the received comment data, and the output is the analysis results.

[0951] Step 3:

[0952] The server performs filtering based on the analysis results. Filtering methods are used to filter out offensive comments, spam comments, and irrelevant comments. The input is the analysis results, and the output is the filtered comment data.

[0953] Step 4:

[0954] The server summarizes the filtered comments using a summarization tool. The server uses libraries such as NLTK and Gensim to summarize the comments and extract important information. The input is the filtered comment data, and the output is basic summary data.

[0955] Step 5:

[0956] The server uses a generative AI model to enhance the summary. A prompt sentence is input to the generative AI model (e.g., GPT-3) to obtain the enhanced summary. The input is the basic summary data and the prompt sentence, and the output is the enhanced summary data. An example of a prompt sentence:

[0957] Summarize the comments below:

[0958] What colors are available in the new products?

[0959] Can you give us a more detailed review?

[0960] Step 6:

[0961] The server transmits the enhanced summary data to the distributor's terminal. The transmission means is used to display the summary comment on the distributor's smartphone or smart glasses. The input is the enhanced summary data, and the output is the summary comment displayed on the distributor's terminal.

[0962] Step 7:

[0963] The terminal provides an interface for the broadcaster to respond to viewers based on the summary comments. The broadcaster uses the terminal to respond to viewers' comments in real time. The input here is the summary comments and the broadcaster's response, and the output is the response data sent to the server.

[0964] Step 8:

[0965] The server receives the broadcaster's response and returns feedback to the viewer in real time. The server receives the broadcaster's response and distributes it so that it can be displayed on the user's device. The input here is the broadcaster's response data, and the output is a response message that is displayed on the viewer's device.

[0966] In this way, the system effectively analyzes, filters, and summarizes viewer comments, providing important feedback to the streamer.

[0967] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0968] As an embodiment of the present invention, this document describes a system that analyzes, filters, and summarizes viewer comments in real time, and also uses an emotion engine to perform sentiment analysis, helping broadcasters communicate more effectively.

[0969] A natural language description of the program's operation

[0970] Server Roles

[0971] The server is the core of the system. When comments from viewers are sent to the server, the server first receives them. Next, it analyzes these comments using natural language processing technology and an emotion engine. During the analysis process, the server recognizes the sentiment of the comments (positive, negative, neutral) and understands their content. The server then filters the comments based on the results of the emotion engine. Negative, spam, and offensive comments are automatically filtered out. The server then summarizes the remaining comments. For example, if the same question is asked by multiple viewers, it combines them into a single unified message. Finally, the server sends the summarized comments and their sentiment information to the broadcaster in real time.

[0972] Device Role

[0973] The terminal is used by the broadcaster to receive and display summary comments sent from the server. The broadcaster can smoothly communicate with viewers through this terminal. The terminal has an interface that presents the received summary information and emotional information in an easy-to-read format. When the broadcaster responds to viewers' questions, the response is also sent from the terminal to the server and is ultimately displayed to all viewers in real time.

[0974] User Roles

[0975] Users (viewers) post comments while watching a video broadcast. The comments are sent from the user's device to the server. The server analyzes and filters the user's comments, and important comments are summarized and sent to the broadcaster, so users can expect their questions to be promptly delivered to the broadcaster and answered.

[0976] Specific examples

[0977] For example, suppose a video streamer is live streaming a review of a new product. Viewer A comments, "What colors does the new product come in?", and viewer B comments, "Could you please give us a more detailed review?". At the same time, viewer C posts an offensive comment. The following process occurs:

[0978] Server Processing

[0979] 1. The server receives comments sent by viewers A, B, and C.

[0980] 2. The server analyzes each comment using natural language processing technology and an emotion engine, and determines that comments from viewers A and B are positive, and that comments from viewer C are negative.

[0981] 3. The server filters out and excludes Viewer C's offensive comments.

[0982] 4. The server summarizes the positive comments from viewers A and B and generates a summary that reads, "Viewers have asked questions about new product color variations and detailed reviews."

[0983] 5. The server sends this summary and positive emotion information to the distributor's device.

[0984] Terminal handling

[0985] 1. The terminal receives the summary comment sent from the server.

[0986] 2. The device displays the summary comment and emotional information received on the screen, and the streamer checks them.

[0987] 3. Based on the summary comments, the streamer will explain the color variations of the new product and provide a detailed review.

[0988] 4. The streamer's explanation is sent to the server via the device and displayed in real time on the viewer's chat screen.

[0989] This allows streamers to focus on positive and important comments without being distracted by negative comments. Viewers also enjoy the satisfaction of receiving prompt answers to their questions. This system efficiently supports communication between viewers and streamers, increasing satisfaction for both parties.

[0990] The processing flow will be explained below.

[0991] Step 1:

[0992] The user creates a comment using their device and presses the send button.

[0993] Step 2:

[0994] The terminal transmits the comment sent by the user to the server.

[0995] Step 3:

[0996] The server receives the comment.

[0997] Step 4:

[0998] The server passes the received comments to a natural language processing engine for linguistic analysis, which involves understanding the meaning, context, and relevance of the text.

[0999] Step 5:

[1000] The server uses the natural language processing engine to analyze the comments and then uses an emotion engine to determine their sentiment, categorizing them into positive, negative, or neutral.

[1001] Step 6:

[1002] The server filters comments based on the results of the sentiment engine, automatically filtering out negative sentiment, spam, and offensive comments.

[1003] Step 7:

[1004] The server summarizes the filtered comments, for example, if the same question is asked repeatedly, it will combine them into one unified message.

[1005] Step 8:

[1006] The server prioritizes the summarized comments based on their importance, placing important or popular questions at the top.

[1007] Step 9:

[1008] The server sends prioritized summary comments and their emotional information to the broadcaster's terminal in real time.

[1009] Step 10:

[1010] The device receives the summary comments sent from the server and displays them on the screen. The interface is designed to be intuitive for streamers.

[1011] Step 11:

[1012] The broadcaster responds to the viewer's summary comments via the terminal. For example, the broadcaster responds by providing detailed information about "color variations of a new product."

[1013] Step 12:

[1014] The device sends the broadcaster's response to the server, which displays it on the viewer's chat screen in real time.

[1015] Step 13:

[1016] The user checks the response from the broadcaster, which gives the user a sense of satisfaction as it confirms that their question has been answered promptly.

[1017] Example 2

[1018] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1019] Video streaming often receives a large number of comments from viewers, some of which may be offensive, spam, or irrelevant. This can make it difficult for streamers to find useful comments, hindering smooth communication with viewers. Furthermore, when multiple viewers have similar questions or opinions, streamers are required to efficiently summarize and respond to them. Therefore, a system is needed that can efficiently analyze, filter, and summarize viewer comments, allowing streamers to understand viewer sentiment in real time.

[1020] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1021] In this invention, the server includes natural language processing means for analyzing comments received from viewers, filtering means for filtering out offensive comments and spam comments from the comments analyzed by the natural language processing means, analysis means for performing sentiment analysis on the comments filtered by the filtering means, summarization means for summarizing the comments whose sentiments have been determined by the analysis means, transmission means for transmitting the comments summarized by the summarization means to the distributor, distribution means for receiving responses from the distributor and transmitting them to the viewers, and real-time processing means for processing the viewer comments and the distributor's responses in real time. This allows the viewer comments to be analyzed, filtered, and summarized efficiently, enabling the distributor to focus on important comments and respond quickly.

[1022] The "natural language processing means" is a means that uses techniques such as tokenization of comments, part-of-speech tagging, and dependency analysis to analyze comments received from viewers.

[1023] The "filtering means" is a means for filtering out offensive comments, spam comments, and irrelevant comments from the comments analyzed by the natural language processing means.

[1024] The "analysis means" is a means for performing sentiment analysis on the comments filtered by the filtering means using a sentiment engine, and classifying the comments into positive, negative, or neutral.

[1025] The "summarization means" is a means for summarizing the comments whose sentiments have been determined by the analysis means, and for integrating comments on similar themes submitted by multiple viewers into a single message.

[1026] The "transmission means" is a means for transmitting the comment summarized by the summarization means to the distributor.

[1027] The "distribution means" is a means for receiving responses from the distributor and transmitting them to the viewers.

[1028] The "real-time processing means" is a means for processing viewer comments and distributor responses in real time.

[1029] MODE FOR CARRYING OUT THE INVENTION

[1030] The present invention relates to a system for analyzing, filtering, summarizing, and analyzing sentiment in real time for comments posted by viewers during video streaming. Detailed embodiments of this system will be described below.

[1031] System Configuration

[1032] This system mainly consists of three components: the server, the terminal, and the user. The role and function of each component will be explained in order.

[1033] server

[1034] The server plays a central role in the system. It receives comments sent by viewers and performs the following steps:

[1035] 1. Natural language processing tools

[1036] The server uses natural language processing (NLP) techniques to analyze comments received from viewers. Specific software available is SpaCy or NLTK. The server uses these techniques to tokenize comments, tag them as parts of speech, and analyze dependencies.

[1037] 2. Filtering Methods

[1038] The server analyzes the comments using natural language processing and filters out offensive, spam, and irrelevant comments. Blacklists and keyword matching are used for filtering. This filtering prevents unnecessary information from reaching the broadcaster.

[1039] 3. Analysis tools

[1040] The server performs sentiment analysis on the filtered comments using a sentiment engine. Specifically, it uses sentiment analysis tools such as VADER and TextBlob to classify the sentiment of the comments as positive, negative, or neutral.

[1041] 4. Summary tools

[1042] The server summarizes the comments after sentiment analysis. If multiple viewers have similar questions or opinions, they are merged into a single unified message. For this purpose, a text summarization algorithm (e.g., BERTSUM, TextRank) is used.

[1043] 5. Transmission Method

[1044] A summary of the comments and their sentiment information is sent to the streamer, allowing the streamer to respond quickly to important comments.

[1045] 6. Distribution Method

[1046] It is a means for receiving the broadcaster's response and sending it to the viewers. This is also done in real time, and the broadcaster's response is visible to all viewers.

[1047] 7. Real-time processing methods

[1048] It processes viewer comments and streamer responses in real time, allowing the entire system to operate smoothly and enabling real-time two-way communication.

[1049] Terminal

[1050] The terminal is a device that allows the broadcaster to receive and display summary comments sent from the server. The terminal can be a general-purpose terminal such as a tablet, smartphone, or PC. The terminal has the following functions:

[1051] 1. Display summary comments

[1052] The summary comments and emotional information sent from the server are displayed in an easy-to-read format, and the broadcaster can review them and respond appropriately.

[1053] 2. Sending a Response

[1054] The responses entered by the streamer are sent from the device to the server, where they are displayed in real time to all viewers.

[1055] User

[1056] Users can post comments while watching a video broadcast. The comments are sent from the user's device to a server, where they are analyzed, filtered, summarized, and analyzed for sentiment. This allows users to expect their comments to be promptly delivered to the broadcaster and answered.

[1057] Specific examples

[1058] For example, consider a case where a video streamer is live-streaming a review of a new product. Viewer A comments, "What colors does the new product come in?", and viewer B comments, "Could you please provide a more detailed review?". At the same time, viewer C posts an offensive comment. The following process occurs:

[1059] Prompt Sentence Examples

[1060] "Analyze viewer comments about new product reviews to determine positive, negative, and neutral sentiment."

[1061] "If multiple viewers ask the same question, summarize it and combine it into one message."

[1062] In this way, the present invention is a system that can support smoother communication by efficiently analyzing, filtering, and summarizing viewer comments and providing distributors with quick and accurate information.

[1063] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1064] Step 1:

[1065] The user enters a comment in the comment input field for the live broadcast, and when the user presses the send button, the comment is sent from the user's device to the server.

[1066] Input: Comments entered by the user

[1067] Output: Comment sent to the server

[1068] Step 2:

[1069] The server receives comments sent by users.

[1070] Input: User comment

[1071] Output: Comments received

[1072] Step 3:

[1073] The server analyzes the received comments using natural language processing (NLP) libraries such as SpaCy and NLTK. Specific steps include tokenizing the comments, tagging them by part of speech, and analyzing dependencies.

[1074] Input: Received comments

[1075] Output: Analysis results (tokenized comments, part of speech information, dependencies)

[1076] Step 4:

[1077] The server filters the comments analyzed by natural language processing using filtering methods, such as blacklists and keyword matching, to filter out offensive, spam, and irrelevant comments.

[1078] Input: Analysis results (tokenized comments, part of speech information, dependencies)

[1079] Output: Filtered comments

[1080] Step 5:

[1081] The server then performs sentiment analysis on the filtered comments using tools such as VADER and TextBlob to determine whether the comment is positive, negative, or neutral.

[1082] Input: Filtered comments

[1083] Output: Sentiment analysis results (positive, negative, neutral)

[1084] Step 6:

[1085] The server uses a summarization tool to summarize comments based on the results of sentiment analysis. If the same question or opinion is posted by multiple viewers, it combines them into a single unified message. For summarization, it uses text summarization algorithms such as BERTSUM and TextRank.

[1086] Input: Sentiment analysis results

[1087] Output: Summarized comments

[1088] Step 7:

[1089] The server transmits the summarized comments and their emotional information to the distributor's terminal.

[1090] Input: Summarized comments and sentiment information

[1091] Output: Summary comments and sentiment information sent to the streamer's device

[1092] Step 8:

[1093] The terminal receives and displays the summary comment and emotion information sent from the server, and the broadcaster responds based on this.

[1094] Input: Summary comments and sentiment information sent from the server

[1095] Output: Summary comments and sentiment information displayed on the screen

[1096] Step 9:

[1097] The broadcaster answers the viewer's questions based on the summary comment and emotional information, enters the answer into the terminal, and presses the send button.

[1098] Input: Streamer's answer

[1099] Output: Answer sent from the terminal to the server

[1100] Step 10:

[1101] The server receives the broadcaster's response and distributes it to all viewers in real time. The broadcaster's response is displayed on the viewers' chat screens.

[1102] Input: Answer sent by the broadcaster

[1103] Output: The streamer's response displayed in the viewer's chat screen

[1104] These specific processing steps allow viewers' comments to be processed efficiently, and smooth communication with the broadcaster is realized.

[1105] (Application example 2)

[1106] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1107] In live streaming and content distribution services, a large number of viewer comments are often posted in a short period of time, making it difficult for the broadcaster to review all of these comments and respond appropriately. In particular, when offensive or spam comments are included, this increases the broadcaster's mental burden and hinders efficient communication. It also makes it difficult for viewers to quickly send important questions or interesting topics to the broadcaster, potentially reducing viewer satisfaction. A system that can solve these issues is needed.

[1108] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes natural language processing means for analyzing comments received from viewers, filtering means for filtering out offensive comments and spam comments from the comments analyzed by the natural language processing means, summarizing means for summarizing the comments filtered by the filtering means, transmission means for transmitting the comments summarized by the summarizing means and their emotional information to the broadcaster, and an emotion analysis engine for emotionally analyzing the viewer comments and recognizing the emotional information (positive, negative, neutral). This allows the broadcaster to focus on the target comments and respond efficiently, and important questions and feedback from viewers are conveyed to the broadcaster quickly and accurately, improving satisfaction.

[1109] "Natural language processing means" is a means for analyzing comments received from viewers, and is a technology that converts text data into a format that is easy for machines to understand and analyzes the meaning and context.

[1110] The "filtering means" is a means for filtering out offensive comments and spam comments from the comments analyzed by the natural language processing means.

[1111] The "summarizing means" is a means for concisely summarizing the comments filtered by the filtering means and conveying the summary to the distributor.

[1112] The "transmission means" is a means for transmitting the comments summarized by the summarization means and their emotional information to the distributor in real time.

[1113] An "emotion analysis engine" is an engine that analyzes viewers' comments and identifies their emotional information (positive, negative, neutral).

[1114] "Emotional information" is information indicating the emotions (positive, negative, neutral) contained in the viewer's comments.

[1115] As an embodiment of this invention, we will explain a system that analyzes, filters, and summarizes viewer comments in real time and performs sentiment analysis using an emotion engine. This system supports effective communication for broadcasters. The system is realized mainly by three components: a server, a terminal, and a user.

[1116] The server uses "natural language processing" to analyze comments received from viewers. Specifically, it uses Python 3 and utilizes the TextBlob and spacy libraries to mechanically analyze the comment text data. It then uses "filtering" on the analyzed comments. This removes offensive and spam comments, leaving only information that is easy for users to view.

[1117] The filtered comments are organized by a "summarization means." The same questions or similar feedback are combined into one, and finally transmitted in real time by a "transmission means" for transmission to the streamer. The server also uses an "emotion analysis engine" to recognize emotional information (positive, negative, neutral) contained in the viewer's comments and transmits this information to the streamer.

[1118] The terminal is used by the broadcaster to receive summary comments and emotional information sent from the server and display them on the screen in an easy-to-read format. Based on this information, the broadcaster can smoothly communicate with the viewers. For example, a device such as a smartphone or a head-mounted display (HMD) can be used.

[1119] Users (viewers) post comments while watching a video broadcast. The comments are sent from the user's device to the server, where they are analyzed, filtered, and summarized, and important comments are sent to the broadcaster. This allows users to expect their questions to be promptly received by the broadcaster and answered.

[1120] As a concrete example, consider a case where a video streamer is live-streaming a review of a new product. Viewer A comments, "What colors does the new product come in?", and viewer B comments, "Could you please give us a more detailed review?". At the same time, viewer C posts an offensive comment, and the process proceeds as follows:

[1121] 1. The server receives comments sent by viewers A, B, and C.

[1122] 2. The server analyzes each comment using natural language processing technology and an emotion engine, and determines that comments from viewers A and B are positive, and that comments from viewer C are negative.

[1123] 3. The server filters out and excludes Viewer C's offensive comments.

[1124] 4. The server summarizes the positive comments from viewers A and B and generates a summary that reads, "Viewers have asked questions about new product color variations and detailed reviews."

[1125] 5. The server sends this summary and positive emotion information to the distributor's device.

[1126] 6. Based on the summary comments, the streamer will explain the color variations of the new product and provide a detailed review.

[1127] 7. The streamer's explanation is sent to the server via the device and displayed in real time on the viewer's chat screen.

[1128] An example of a prompt to be input to the generative AI model is, "I'm asking about the color variations of a new product. I'd also like a detailed review. Please think of an explanation for this." This will help the generative AI model provide specific answers and supplementary information.

[1129] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1130] Step 1:

[1131] Users (viewers) post comments while watching the video broadcast. The comments are sent from the user's device to the server. The input is the viewer's comments, and the output is the comments sent to the server.

[1132] Step 2:

[1133] The server analyzes comments received from viewers using natural language processing. Specifically, it uses Python 3's TextBlob and spacy libraries to analyze the comment text data and determine its linguistic structure and sentiment. The input is the text data of the comments sent by viewers, and the output is the analyzed linguistic structure and sentiment information of the comments.

[1134] Step 3:

[1135] The server uses filtering techniques to filter out offensive and spam comments from the analyzed comments. Specifically, it identifies and filters out offensive and spam comments using pre-defined keywords and specific language patterns. The input is the analyzed comment text data and sentiment information, and the output is the filtered comment data.

[1136] Step 4:

[1137] The server summarizes the filtered comments using a summarization means. Specifically, it aggregates similar questions and feedbacks into a concise message. The input is the filtered comment data, and the output is the summarized comment data.

[1138] Step 5:

[1139] The server uses a sentiment analysis engine to recognize the emotional information (positive, negative, neutral) contained in viewers' comments. Specifically, it uses a machine learning algorithm to classify the emotions of the comments. The input is summarized comment data, and the output is comment data with emotional information added.

[1140] Step 6:

[1141] The server sends the summarized comments and emotional information to the broadcaster's terminal. The input is the comment data with the emotional information added, and the output is the comment information displayed on the broadcaster's terminal.

[1142] Step 7:

[1143] The terminal receives the summary comment and emotion information sent from the server and displays them on the screen. The broadcaster communicates with the viewer based on this information. The input is the summary comment and emotion information sent from the server, and the output is the comment information displayed on the screen.

[1144] Step 8:

[1145] The broadcaster responds to the viewer's comments. The response is sent from the terminal to the server and displayed in real time on the viewer's chat screen. The input is the broadcaster's response, and the output is a response message that is displayed on the viewer's chat screen.

[1146] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1147] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1148] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1149] [Fourth embodiment]

[1150] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1151] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1152] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1153] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1154] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1155] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1156] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1157] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1158] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1159] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1160] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1161] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1162] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1163] DETAILED DESCRIPTION OF THE INVENTION The present invention will be described in detail below with reference to the accompanying drawings. The present invention relates to a system for analyzing, filtering, and summarizing comments from viewers in real time, and effectively presenting them to the broadcaster.

[1164] A natural language description of the program's operation

[1165] Server Roles

[1166] The server plays a central role in this system. When viewers send comments to the server, the server first receives them. Then, it analyzes these comments using natural language processing technology. This analysis includes sentiment analysis and understanding the content of the comments. The server then filters the comments based on the analysis results. It identifies offensive and spam comments and automatically filters them out. The server then summarizes the remaining comments. For example, if multiple viewers ask the same question, it combines them into a single unified message. Finally, the server sends the summarized comments to the broadcaster in real time.

[1167] Device Role

[1168] The terminal is used by the broadcaster to receive and display summary comments sent from the server. The broadcaster can smoothly communicate with viewers through this terminal. The terminal has an interface that presents the received summary information to the broadcaster in an easy-to-read format. When the broadcaster responds to a viewer's question, the response is also sent from the terminal to the server and is ultimately displayed to all viewers in real time.

[1169] User Roles

[1170] Users (viewers) post comments while watching a video broadcast. The comments are sent from the user's device to the server. The server analyzes and filters the user's comments, and important comments are summarized and sent to the broadcaster, so users can expect their questions to be quickly delivered to the broadcaster and answered.

[1171] Specific examples

[1172] For example, suppose a video streamer is live streaming a review of a new product. Viewer A comments, "What colors does the new product come in?", and viewer B comments, "Could you please give us a more detailed review?". At the same time, viewer C posts an offensive comment. The following process occurs:

[1173] Server Processing

[1174] 1. The server receives comments sent by viewers A, B, and C.

[1175] 2. The server uses natural language processing technology to analyze each comment and determines that comments from viewers A and B are positive, and that comments from viewer C are offensive.

[1176] 3. The server filters out viewer C's comments.

[1177] 4. The server summarizes the comments from viewers A and B and generates a summary that reads, "Viewers have asked questions about the color variations of new products and detailed reviews."

[1178] 5. The server sends this summary to the distributor's terminal.

[1179] Terminal handling

[1180] 1. The terminal receives the summary comment sent from the server.

[1181] 2. The device displays the summary comment received on the screen, and the broadcaster checks it.

[1182] 3. Based on the summary comments, the streamer will explain the color variations of the new product and provide a detailed review.

[1183] 4. The streamer's explanation is sent to the server via the device and displayed in real time on the viewer's chat screen.

[1184] This allows streamers to focus on the positive and important comments without being distracted by negative ones, and viewers appreciate that their questions are answered quickly.

[1185] The processing flow will be explained below.

[1186] Step 1:

[1187] The user creates a comment using their device and presses the send button.

[1188] Step 2:

[1189] The terminal sends this comment to the server.

[1190] Step 3:

[1191] The server receives the comment.

[1192] Step 4:

[1193] The server passes the comments to a natural language processing engine for content analysis, including sentiment analysis (positive, negative, neutral) and relevance detection of the comments.

[1194] Step 5:

[1195] The server filters comments based on the analysis results, automatically filtering out offensive, spam, and inappropriate comments.

[1196] Step 6:

[1197] The server summarizes the remaining comments, for example if the same question is asked by multiple viewers it will combine them into one unified message.

[1198] Step 7:

[1199] The server prioritizes the summarized comments based on importance.

[1200] Step 8:

[1201] The server sends prioritized summary comments to the broadcaster's terminal in real time.

[1202] Step 9:

[1203] The device displays the summary comments received on the screen, providing an interface that is intuitive for the broadcaster.

[1204] Step 10:

[1205] The streamer responds to the viewers' summary comments via the device. For example, he or she might begin explaining the "color variations of a new product" that is ranked highly.

[1206] Step 11:

[1207] The device sends the broadcaster's response to the server, which displays it on the viewer's chat screen in real time.

[1208] Step 12:

[1209] The user checks the response from the broadcaster, which gives the user satisfaction that their question was answered quickly.

[1210] Example 1

[1211] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1212] In conventional live streaming systems, streamers often become overwhelmed by the large number of comments from viewers, causing them to miss important questions and feedback. Furthermore, offensive or spam comments can increase the streamer's mental burden and hinder smooth communication. Furthermore, responding to numerous similar comments individually takes time and effort, making it difficult to respond quickly to all viewers.

[1213] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1214] In this invention, the server includes a natural language processing means for analyzing comments received from viewers, a filtering means for filtering out offensive and spam comments from the comments analyzed by the natural language processing means, and a summarizing means for summarizing the comments filtered by the filtering means. This makes it easier for broadcasters to focus on important comments and protects them from offensive and spam comments, enabling smooth broadcasting. In addition, by consolidating and summarizing a large number of similar comments, the broadcaster can quickly respond to viewers' questions and feedback.

[1215] "Natural language processing means" refers to technology for analyzing comments received from viewers and understanding and evaluating their content, including sentiment analysis, keyword extraction, and text classification.

[1216] "Filtering Methods" refers to methods for identifying and filtering out offensive or spam comments from comments analyzed by natural language processing methods, thereby enabling the Streamer to receive only appropriate comments.

[1217] "Summarization" refers to technology that consolidates multiple comments into a concise format and summarizes the main content, allowing streamers to efficiently grasp important information.

[1218] "Transmission Method" means the technology used to transmit the parsed, filtered, and summarized Comments to your device, including any network communications and data transfer protocols.

[1219] The "receiving means" refers to a technology for receiving the summary comment sent from the server on the distributor's terminal, which then displays the summarized comment on the distributor's terminal.

[1220] "Display means" refers to the interface displayed on the terminal so that the broadcaster can check the summary comments received, including screen displays, notification systems, etc.

[1221] The "response transmission means of the distributor" refers to a technology that allows the distributor to transmit a response to a viewer's comment to the server, thereby transmitting the distributor's response to the viewer.

[1222] "Means for displaying on the viewer's chat screen" refers to technology for displaying the streamer's responses on the viewer's chat screen in real time, allowing the viewer to check the streamer's responses in real time.

[1223] The following describes an embodiment of the present invention: This system analyzes, filters, and summarizes comments from viewers in real time, and effectively presents them to the broadcaster.

[1224] Server Roles

[1225] The server plays a central role in this system. Specifically, it receives comments sent by viewers and analyzes them using natural language processing technology. This analysis includes sentiment analysis and understanding of the comment content. The server filters out offensive and spam comments based on the analysis results. It has the ability to automatically filter out offensive and spam comments. The server then summarizes the remaining comments and sends them to the streamer in real time. This can be done, for example, using the Google Cloud Natural Language API.

[1226] Device Role

[1227] The terminal is used by the streamer to receive and display summary comments sent from the server. The streamer can smoothly communicate with viewers through this terminal. The terminal has an interface that presents the received summary information to the streamer in an easy-to-read format. In addition, when the streamer responds to viewers' questions, the response is also sent from the terminal to the server and is ultimately displayed to all viewers in real time. It is possible to use a dedicated application for streamers or streaming software (e.g., OBS Studio).

[1228] User Roles

[1229] Users (viewers) post comments while watching a video broadcast. The comments are sent from the user's device to the server. The server analyzes and filters the user's comments, and important comments are summarized and sent to the broadcaster. This allows users to expect their questions to be quickly delivered to the broadcaster and answered.

[1230] Specific examples

[1231] For example, suppose a video streamer is livestreaming a review of a new product. Viewer A comments, "What colors does the new product come in?", and viewer B comments, "Could you please give us a more detailed review?". At the same time, viewer C posts an offensive comment. The following process occurs:

[1232] Server Processing

[1233] 1. The server receives comments sent by viewers A, B, and C.

[1234] 2. The server analyzes each comment using natural language processing technology (e.g., Google Cloud Natural Language API) and determines the comments from viewers A and B as positive, and the comment from viewer C as offensive.

[1235] 3. The server filters out viewer C's comments.

[1236] 4. The server summarizes the comments from viewers A and B and generates a summary that reads, "Viewers have asked questions about the color variations of new products and detailed reviews."

[1237] 5. The server sends this summary to the distributor's terminal.

[1238] Terminal handling

[1239] 1. The terminal receives the summary comment sent from the server.

[1240] 2. The device displays the summary comment received on the screen, and the broadcaster checks it.

[1241] 3. Based on the summary comments, the streamer will explain the color variations of the new product and provide a detailed review.

[1242] 4. The streamer's explanation is sent to the server via the device and displayed in real time on the viewer's chat screen.

[1243] This allows streamers to focus on the positive and important comments without being distracted by negative ones, and viewers appreciate that their questions are answered quickly.

[1244] Example prompts to input to the generative AI model

[1245] Please explain in detail the process for creating a system that analyzes comments posted during video streaming, summarizes them, and provides them to the streamer.

[1246] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1247] Step 1:

[1248] Comments received

[1249] The server receives comments from viewers in real time, specifically text data sent from users' devices via network communication.

[1250] Input: Comments submitted by the user (e.g., "What are the color variations of the new product?").

[1251] Output: Received comment data.

[1252] Step 2:

[1253] Comment Analysis

[1254] The server analyzes the received comments using natural language processing technology (e.g., Google Cloud Natural Language API), including sentiment analysis and content classification.

[1255] Input: Received comment data.

[1256] Output: Parsed comment data (e.g. sentiment score, classification tags for comment content).

[1257] Step 3:

[1258] Comment Filtering

[1259] The server filters out offensive and spam comments based on the analysis results, and the filtered comments are removed.

[1260] Input: Parsed comment data.

[1261] Output: Filtered comment data (offensive and spam comments filtered out).

[1262] Step 4:

[1263] Comment Summary

[1264] The server summarizes the filtered comments, and if multiple viewers post similar questions, it merges them into a single message.

[1265] Input: Filtered comment data.

[1266] Output: Summarized comment data (e.g., "Viewers have asked about new product color variations and detailed reviews").

[1267] Step 5:

[1268] Submit summary comment

[1269] The server sends the summarized comments to the broadcaster's terminal via network communication.

[1270] Input: Summarized comment data.

[1271] Output: The summary comment submitted.

[1272] Step 6:

[1273] Summary comments received

[1274] The terminal receives the summary comment sent from the server.

[1275] Input: Summary comment data sent from the server.

[1276] Output: Received summary comment data.

[1277] Step 7:

[1278] Summary comment display

[1279] The device displays the received summary comments on the screen, allowing the broadcaster to view the comments in an easy-to-read interface.

[1280] Input: Received summary comment data.

[1281] Output: Summary comments displayed on the terminal screen.

[1282] Step 8:

[1283] Streamer's response

[1284] The broadcaster responds to viewers' questions based on the summary comments, and the responses are sent to the server via the terminal.

[1285] Input: The broadcaster's response (e.g., "The new product comes in three colors: red, blue, and green.").

[1286] Output: The publisher's response data sent to the server.

[1287] Step 9:

[1288] Response Display

[1289] The server receives the broadcaster's response and displays it in real time on the viewer's chat screen.

[1290] Input: Response data sent by the distributor.

[1291] Output: The streamer's response displayed in the viewer's chat screen.

[1292] This allows for effective management of large volumes of comments from viewers and allows streamers to respond quickly to appropriate comments.

[1293] (Application example 1)

[1294] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1295] The problem this invention aims to solve is how to enable content distributors to efficiently process large volumes of comments from viewers and quickly receive important feedback in content distribution services. In particular, it is necessary to provide a means to provide distributors with effective and immediate information when summarizing similar comments from viewers, while filtering out offensive comments, spam comments, and irrelevant comments. It is also necessary to create an environment where distributors can focus on important information by further enhancing summarization using a generative AI model.

[1296] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1297] In this invention, the server includes natural language processing means for analyzing comments received from viewers, filtering means for filtering out offensive comments and spam comments from the comments analyzed by the natural language processing means, summarizing means for summarizing the comments filtered by the filtering means, summarization enhancement means for further summarizing the comments using a generative AI model, and transmission means for transmitting the comments summarized by the summarization means and summary enhancement means to the broadcaster. This enables the broadcaster to quickly identify only important comments from a large number of comments and communicate effectively with viewers.

[1298] "Natural language processing means" refers to technology used to analyze comments received from viewers, and includes grammatical analysis and sentiment analysis.

[1299] "Filtering measures" are measures for filtering out offensive comments, spam comments, and irrelevant comments from the analyzed comments.

[1300] The "summarizing means" is a means for summarizing the comments filtered by the filtering means.

[1301] A "summarization enhancement means" is a means for further summarizing comments that have been summarized using a generative AI model.

[1302] The "transmission means" is a means for transmitting the comments summarized by the summarization means and the summary enhancement means to the distributor.

[1303] A "generative AI model" is a model that uses artificial intelligence technology to summarize comments and extract important information based on generated prompt sentences.

[1304] DETAILED DESCRIPTION OF THE INVENTION The following describes an embodiment of the present invention.

[1305] System Program

[1306] This invention provides a system for content distribution services that analyzes, filters, and summarizes comments from viewers in real time and effectively presents them to distributors. This system is mainly composed of three entities: a server, a terminal, and a user.

[1307] Server Roles

[1308] The server plays a central role in this system. When comments from viewers are sent to the server, the server first receives them. The server then analyzes these comments using natural language processing technology. The analysis includes sentiment analysis and understanding of the comment content. Specifically, it uses the Python language and libraries such as TensorFlow and spaCy. The server then filters the comments based on the analysis results. The filtering automatically excludes offensive comments, spam comments, and irrelevant comments.

[1309] The server then processes the filtered comments to summarize them. First, it uses basic summarization tools to summarize the comments, and then it further enhances the summarization using a generative AI model. This generative AI model uses the open-source AI system GPT-3 (or similar technology). It combines multiple comments into a unified message using the example prompts described below. Finally, the summarized comments are sent to the broadcaster's device.

[1310] Device Role

[1311] The device is used by the streamer to receive and display the summary comments sent from the server. Devices include smartphones, smart glasses, and head-mounted displays. These devices have an interface that presents the received summary information to the streamer in an easy-to-read format, and the visual display is achieved using front-end technologies such as React.js. When the streamer responds to viewer questions, the response is also sent from the device to the server and is ultimately displayed to all viewers in real time. In this way, the streamer can smoothly communicate with viewers in real time while focusing on important comments.

[1312] User Roles

[1313] Users (viewers) post comments while watching content broadcasts. The comments are sent from the user's device to the server. The server analyzes and filters the user's comments, and important comments are summarized and sent to the broadcaster, so users can expect their questions to be quickly delivered to the broadcaster and answered.

[1314] Specific examples

[1315] For example, when a video streamer is live streaming a review of a new product, multiple comments are posted by viewers. Viewer A comments, "What colors does the new product come in?", and viewer B comments, "Could you please give a more detailed review?". At the same time, if viewer C posts an offensive comment, the process proceeds as follows:

[1316] First, the server receives comments sent by viewers A, B, and C and analyzes them using natural language processing technology. Viewer C's comment is determined to be offensive and is filtered out. Next, the server summarizes the comments from viewers A and B and generates a summary such as, "A viewer has asked about the color variations of a new product and a detailed review." This summary is further enhanced by a generative AI model and sent to the streamer's device.

[1317] An example prompt is:

[1318] Summarize the comments below:

[1319] What colors are available in the new products?

[1320] Can you give us a more detailed review?

[1321] This allows streamers to efficiently understand important comments and facilitate smooth communication with viewers.

[1322] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1323] Step 1:

[1324] The server receives comments from viewers. Comments sent by viewers reach the server from the user's device. The input here is the viewer's comments, and the output is the comment data stored on the server.

[1325] Step 2:

[1326] The server analyzes the received comments using natural language processing. The server uses libraries such as TensorFlow and spaCy to analyze the grammar and sentiment of the comments. The input is the received comment data, and the output is the analysis results.

[1327] Step 3:

[1328] The server performs filtering based on the analysis results. Filtering methods are used to filter out offensive comments, spam comments, and irrelevant comments. The input is the analysis results, and the output is the filtered comment data.

[1329] Step 4:

[1330] The server summarizes the filtered comments using a summarization tool. The server uses libraries such as NLTK and Gensim to summarize the comments and extract important information. The input is the filtered comment data, and the output is basic summary data.

[1331] Step 5:

[1332] The server uses a generative AI model to enhance the summary. A prompt sentence is input to the generative AI model (e.g., GPT-3) to obtain the enhanced summary. The input is the basic summary data and the prompt sentence, and the output is the enhanced summary data. An example of a prompt sentence:

[1333] Summarize the comments below:

[1334] What colors are available in the new products?

[1335] Can you give us a more detailed review?

[1336] Step 6:

[1337] The server transmits the enhanced summary data to the distributor's terminal. The transmission means is used to display the summary comment on the distributor's smartphone or smart glasses. The input is the enhanced summary data, and the output is the summary comment displayed on the distributor's terminal.

[1338] Step 7:

[1339] The terminal provides an interface for the broadcaster to respond to viewers based on the summary comments. The broadcaster uses the terminal to respond to viewers' comments in real time. The input here is the summary comments and the broadcaster's response, and the output is the response data sent to the server.

[1340] Step 8:

[1341] The server receives the broadcaster's response and returns feedback to the viewer in real time. The server receives the broadcaster's response and distributes it so that it can be displayed on the user's device. The input here is the broadcaster's response data, and the output is a response message that is displayed on the viewer's device.

[1342] In this way, the system effectively analyzes, filters, and summarizes viewer comments, providing important feedback to the streamer.

[1343] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1344] As an embodiment of the present invention, this document describes a system that analyzes, filters, and summarizes viewer comments in real time, and also uses an emotion engine to perform sentiment analysis, helping broadcasters communicate more effectively.

[1345] A natural language description of the program's operation

[1346] Server Roles

[1347] The server is the core of the system. When comments from viewers are sent to the server, the server first receives them. Next, it analyzes these comments using natural language processing technology and an emotion engine. During the analysis process, the server recognizes the sentiment of the comments (positive, negative, neutral) and understands their content. The server then filters the comments based on the results of the emotion engine. Negative, spam, and offensive comments are automatically filtered out. The server then summarizes the remaining comments. For example, if the same question is asked by multiple viewers, it combines them into a single unified message. Finally, the server sends the summarized comments and their sentiment information to the broadcaster in real time.

[1348] Device Role

[1349] The terminal is used by the broadcaster to receive and display summary comments sent from the server. The broadcaster can smoothly communicate with viewers through this terminal. The terminal has an interface that presents the received summary information and emotional information in an easy-to-read format. When the broadcaster responds to viewers' questions, the response is also sent from the terminal to the server and is ultimately displayed to all viewers in real time.

[1350] User Roles

[1351] Users (viewers) post comments while watching a video broadcast. The comments are sent from the user's device to the server. The server analyzes and filters the user's comments, and important comments are summarized and sent to the broadcaster, so users can expect their questions to be promptly delivered to the broadcaster and answered.

[1352] Specific examples

[1353] For example, suppose a video streamer is live streaming a review of a new product. Viewer A comments, "What colors does the new product come in?", and viewer B comments, "Could you please give us a more detailed review?". At the same time, viewer C posts an offensive comment. The following process occurs:

[1354] Server Processing

[1355] 1. The server receives comments sent by viewers A, B, and C.

[1356] 2. The server analyzes each comment using natural language processing technology and an emotion engine, and determines that comments from viewers A and B are positive, and that comments from viewer C are negative.

[1357] 3. The server filters out and excludes Viewer C's offensive comments.

[1358] 4. The server summarizes the positive comments from viewers A and B and generates a summary that reads, "Viewers have asked questions about new product color variations and detailed reviews."

[1359] 5. The server sends this summary and positive emotion information to the distributor's device.

[1360] Terminal handling

[1361] 1. The terminal receives the summary comment sent from the server.

[1362] 2. The device displays the summary comment and emotional information received on the screen, and the streamer checks them.

[1363] 3. Based on the summary comments, the streamer will explain the color variations of the new product and provide a detailed review.

[1364] 4. The streamer's explanation is sent to the server via the device and displayed in real time on the viewer's chat screen.

[1365] This allows streamers to focus on positive and important comments without being distracted by negative comments. Viewers also enjoy the satisfaction of receiving prompt answers to their questions. This system efficiently supports communication between viewers and streamers, increasing satisfaction for both parties.

[1366] The processing flow will be explained below.

[1367] Step 1:

[1368] The user creates a comment using their device and presses the send button.

[1369] Step 2:

[1370] The terminal transmits the comment sent by the user to the server.

[1371] Step 3:

[1372] The server receives the comment.

[1373] Step 4:

[1374] The server passes the received comments to a natural language processing engine for linguistic analysis, which involves understanding the meaning, context, and relevance of the text.

[1375] Step 5:

[1376] The server uses the natural language processing engine to analyze the comments and then uses an emotion engine to determine their sentiment, categorizing them into positive, negative, or neutral.

[1377] Step 6:

[1378] The server filters comments based on the results of the sentiment engine, automatically filtering out negative sentiment, spam, and offensive comments.

[1379] Step 7:

[1380] The server summarizes the filtered comments, for example, if the same question is asked repeatedly, it will combine them into one unified message.

[1381] Step 8:

[1382] The server prioritizes the summarized comments based on their importance, placing important or popular questions at the top.

[1383] Step 9:

[1384] The server sends prioritized summary comments and their emotional information to the broadcaster's terminal in real time.

[1385] Step 10:

[1386] The device receives the summary comments sent from the server and displays them on the screen. The interface is designed to be intuitive for streamers.

[1387] Step 11:

[1388] The broadcaster responds to the viewer's summary comments via the terminal. For example, the broadcaster responds by providing detailed information about "color variations of a new product."

[1389] Step 12:

[1390] The device sends the broadcaster's response to the server, which displays it on the viewer's chat screen in real time.

[1391] Step 13:

[1392] The user checks the response from the broadcaster, which gives the user a sense of satisfaction as it confirms that their question has been answered promptly.

[1393] Example 2

[1394] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1395] Video streaming often receives a large number of comments from viewers, some of which may be offensive, spam, or irrelevant. This can make it difficult for streamers to find useful comments, hindering smooth communication with viewers. Furthermore, when multiple viewers have similar questions or opinions, streamers are required to efficiently summarize and respond to them. Therefore, a system is needed that can efficiently analyze, filter, and summarize viewer comments, allowing streamers to understand viewer sentiment in real time.

[1396] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1397] In this invention, the server includes natural language processing means for analyzing comments received from viewers, filtering means for filtering out offensive comments and spam comments from the comments analyzed by the natural language processing means, analysis means for performing sentiment analysis on the comments filtered by the filtering means, summarization means for summarizing the comments whose sentiments have been determined by the analysis means, transmission means for transmitting the comments summarized by the summarization means to the distributor, distribution means for receiving responses from the distributor and transmitting them to the viewers, and real-time processing means for processing the viewer comments and the distributor's responses in real time. This allows the viewer comments to be analyzed, filtered, and summarized efficiently, enabling the distributor to focus on important comments and respond quickly.

[1398] The "natural language processing means" is a means that uses techniques such as tokenization of comments, part-of-speech tagging, and dependency analysis to analyze comments received from viewers.

[1399] The "filtering means" is a means for filtering out offensive comments, spam comments, and irrelevant comments from the comments analyzed by the natural language processing means.

[1400] The "analysis means" is a means for performing sentiment analysis on the comments filtered by the filtering means using a sentiment engine, and classifying the comments into positive, negative, or neutral.

[1401] The "summarization means" is a means for summarizing the comments whose sentiments have been determined by the analysis means, and for integrating comments on similar themes submitted by multiple viewers into a single message.

[1402] The "transmission means" is a means for transmitting the comment summarized by the summarization means to the distributor.

[1403] The "distribution means" is a means for receiving responses from the distributor and transmitting them to the viewers.

[1404] The "real-time processing means" is a means for processing viewer comments and distributor responses in real time.

[1405] MODE FOR CARRYING OUT THE INVENTION

[1406] The present invention relates to a system for analyzing, filtering, summarizing, and analyzing sentiment in real time for comments posted by viewers during video streaming. Detailed embodiments of this system will be described below.

[1407] System Configuration

[1408] This system mainly consists of three components: the server, the terminal, and the user. The role and function of each component will be explained in order.

[1409] server

[1410] The server plays a central role in the system. It receives comments sent by viewers and performs the following steps:

[1411] 1. Natural language processing tools

[1412] The server uses natural language processing (NLP) techniques to analyze comments received from viewers. Specific software available is SpaCy or NLTK. The server uses these techniques to tokenize comments, tag them as parts of speech, and analyze dependencies.

[1413] 2. Filtering Methods

[1414] The server analyzes the comments using natural language processing and filters out offensive, spam, and irrelevant comments. Blacklists and keyword matching are used for filtering. This filtering prevents unnecessary information from reaching the broadcaster.

[1415] 3. Analysis tools

[1416] The server performs sentiment analysis on the filtered comments using a sentiment engine. Specifically, it uses sentiment analysis tools such as VADER and TextBlob to classify the sentiment of the comments as positive, negative, or neutral.

[1417] 4. Summary tools

[1418] The server summarizes the comments after sentiment analysis. If multiple viewers have similar questions or opinions, they are merged into a single unified message. For this purpose, a text summarization algorithm (e.g., BERTSUM, TextRank) is used.

[1419] 5. Transmission Method

[1420] A summary of the comments and their sentiment information is sent to the streamer, allowing the streamer to respond quickly to important comments.

[1421] 6. Distribution Method

[1422] It is a means for receiving the broadcaster's response and sending it to the viewers. This is also done in real time, and the broadcaster's response is visible to all viewers.

[1423] 7. Real-time processing methods

[1424] It processes viewer comments and streamer responses in real time, allowing the entire system to operate smoothly and enabling real-time two-way communication.

[1425] Terminal

[1426] The terminal is a device that allows the broadcaster to receive and display summary comments sent from the server. The terminal can be a general-purpose terminal such as a tablet, smartphone, or PC. The terminal has the following functions:

[1427] 1. Display summary comments

[1428] The summary comments and emotional information sent from the server are displayed in an easy-to-read format, and the broadcaster can review them and respond appropriately.

[1429] 2. Sending a Response

[1430] The responses entered by the streamer are sent from the device to the server, where they are displayed in real time to all viewers.

[1431] User

[1432] Users can post comments while watching a video broadcast. The comments are sent from the user's device to a server, where they are analyzed, filtered, summarized, and analyzed for sentiment. This allows users to expect their comments to be promptly delivered to the broadcaster and answered.

[1433] Specific examples

[1434] For example, consider a case where a video streamer is live-streaming a review of a new product. Viewer A comments, "What colors does the new product come in?", and viewer B comments, "Could you please provide a more detailed review?". At the same time, viewer C posts an offensive comment. The following process occurs:

[1435] Prompt Sentence Examples

[1436] "Analyze viewer comments about new product reviews to determine positive, negative, and neutral sentiment."

[1437] "If multiple viewers ask the same question, summarize it and combine it into one message."

[1438] In this way, the present invention is a system that can support smoother communication by efficiently analyzing, filtering, and summarizing viewer comments and providing distributors with quick and accurate information.

[1439] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1440] Step 1:

[1441] The user enters a comment in the comment input field for the live broadcast, and when the user presses the send button, the comment is sent from the user's device to the server.

[1442] Input: Comments entered by the user

[1443] Output: Comment sent to the server

[1444] Step 2:

[1445] The server receives comments sent by users.

[1446] Input: User comment

[1447] Output: Comments received

[1448] Step 3:

[1449] The server analyzes the received comments using natural language processing (NLP) libraries such as SpaCy and NLTK. Specific steps include tokenizing the comments, tagging them by part of speech, and analyzing dependencies.

[1450] Input: Received comments

[1451] Output: Analysis results (tokenized comments, part of speech information, dependencies)

[1452] Step 4:

[1453] The server filters the comments analyzed by natural language processing using filtering methods, such as blacklists and keyword matching, to filter out offensive, spam, and irrelevant comments.

[1454] Input: Analysis results (tokenized comments, part of speech information, dependencies)

[1455] Output: Filtered comments

[1456] Step 5:

[1457] The server then performs sentiment analysis on the filtered comments using tools such as VADER and TextBlob to determine whether the comment is positive, negative, or neutral.

[1458] Input: Filtered comments

[1459] Output: Sentiment analysis results (positive, negative, neutral)

[1460] Step 6:

[1461] The server uses a summarization tool to summarize comments based on the results of sentiment analysis. If the same question or opinion is posted by multiple viewers, it combines them into a single unified message. For summarization, it uses text summarization algorithms such as BERTSUM and TextRank.

[1462] Input: Sentiment analysis results

[1463] Output: Summarized comments

[1464] Step 7:

[1465] The server transmits the summarized comments and their emotional information to the distributor's terminal.

[1466] Input: Summarized comments and sentiment information

[1467] Output: Summary comments and sentiment information sent to the streamer's device

[1468] Step 8:

[1469] The terminal receives and displays the summary comment and emotion information sent from the server, and the broadcaster responds based on this.

[1470] Input: Summary comments and sentiment information sent from the server

[1471] Output: Summary comments and sentiment information displayed on the screen

[1472] Step 9:

[1473] The broadcaster answers the viewer's questions based on the summary comment and emotional information, enters the answer into the terminal, and presses the send button.

[1474] Input: Streamer's answer

[1475] Output: Answer sent from the terminal to the server

[1476] Step 10:

[1477] The server receives the broadcaster's response and distributes it to all viewers in real time. The broadcaster's response is displayed on the viewers' chat screens.

[1478] Input: Answer sent by the broadcaster

[1479] Output: The streamer's response displayed in the viewer's chat screen

[1480] These specific processing steps allow viewers' comments to be processed efficiently, and smooth communication with the broadcaster is realized.

[1481] (Application example 2)

[1482] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1483] In live streaming and content distribution services, a large number of viewer comments are often posted in a short period of time, making it difficult for the broadcaster to review all of these comments and respond appropriately. In particular, when offensive or spam comments are included, this increases the broadcaster's mental burden and hinders efficient communication. It also makes it difficult for viewers to quickly send important questions or interesting topics to the broadcaster, potentially reducing viewer satisfaction. A system that can solve these issues is needed.

[1484] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes natural language processing means for analyzing comments received from viewers, filtering means for filtering out offensive comments and spam comments from the comments analyzed by the natural language processing means, summarizing means for summarizing the comments filtered by the filtering means, transmission means for transmitting the comments summarized by the summarizing means and their emotional information to the broadcaster, and an emotion analysis engine for emotionally analyzing the viewer comments and recognizing the emotional information (positive, negative, neutral). This allows the broadcaster to focus on the target comments and respond efficiently, and important questions and feedback from viewers are conveyed to the broadcaster quickly and accurately, improving satisfaction.

[1485] "Natural language processing means" is a means for analyzing comments received from viewers, and is a technology that converts text data into a format that is easy for machines to understand and analyzes the meaning and context.

[1486] The "filtering means" is a means for filtering out offensive comments and spam comments from the comments analyzed by the natural language processing means.

[1487] The "summarizing means" is a means for concisely summarizing the comments filtered by the filtering means and conveying the summary to the distributor.

[1488] The "transmission means" is a means for transmitting the comments summarized by the summarization means and their emotional information to the distributor in real time.

[1489] An "emotion analysis engine" is an engine that analyzes viewers' comments and identifies their emotional information (positive, negative, neutral).

[1490] "Emotional information" is information indicating the emotions (positive, negative, neutral) contained in the viewer's comments.

[1491] As an embodiment of this invention, we will explain a system that analyzes, filters, and summarizes viewer comments in real time and performs sentiment analysis using an emotion engine. This system supports effective communication for broadcasters. The system is realized mainly by three components: a server, a terminal, and a user.

[1492] The server uses "natural language processing" to analyze comments received from viewers. Specifically, it uses Python 3 and utilizes the TextBlob and spacy libraries to mechanically analyze the comment text data. It then uses "filtering" on the analyzed comments. This removes offensive and spam comments, leaving only information that is easy for users to view.

[1493] The filtered comments are organized by a "summarization means." The same questions or similar feedback are combined into one, and finally transmitted in real time by a "transmission means" for transmission to the streamer. The server also uses an "emotion analysis engine" to recognize emotional information (positive, negative, neutral) contained in the viewer's comments and transmits this information to the streamer.

[1494] The terminal is used by the broadcaster to receive summary comments and emotional information sent from the server and display them on the screen in an easy-to-read format. Based on this information, the broadcaster can smoothly communicate with the viewers. For example, a device such as a smartphone or a head-mounted display (HMD) can be used.

[1495] Users (viewers) post comments while watching a video broadcast. The comments are sent from the user's device to the server, where they are analyzed, filtered, and summarized, and important comments are sent to the broadcaster. This allows users to expect their questions to be promptly received by the broadcaster and answered.

[1496] As a concrete example, consider a case where a video streamer is live-streaming a review of a new product. Viewer A comments, "What colors does the new product come in?", and viewer B comments, "Could you please give us a more detailed review?". At the same time, viewer C posts an offensive comment, and the process proceeds as follows:

[1497] 1. The server receives comments sent by viewers A, B, and C.

[1498] 2. The server analyzes each comment using natural language processing technology and an emotion engine, and determines that comments from viewers A and B are positive, and that comments from viewer C are negative.

[1499] 3. The server filters out and excludes Viewer C's offensive comments.

[1500] 4. The server summarizes the positive comments from viewers A and B and generates a summary that reads, "Viewers have asked questions about new product color variations and detailed reviews."

[1501] 5. The server sends this summary and positive emotion information to the distributor's device.

[1502] 6. Based on the summary comments, the streamer will explain the color variations of the new product and provide a detailed review.

[1503] 7. The streamer's explanation is sent to the server via the device and displayed in real time on the viewer's chat screen.

[1504] An example of a prompt to be input to the generative AI model is, "I'm asking about the color variations of a new product. I'd also like a detailed review. Please think of an explanation for this." This will help the generative AI model provide specific answers and supplementary information.

[1505] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1506] Step 1:

[1507] Users (viewers) post comments while watching the video broadcast. The comments are sent from the user's device to the server. The input is the viewer's comments, and the output is the comments sent to the server.

[1508] Step 2:

[1509] The server analyzes comments received from viewers using natural language processing. Specifically, it uses Python 3's TextBlob and spacy libraries to analyze the comment text data and determine its linguistic structure and sentiment. The input is the text data of the comments sent by viewers, and the output is the analyzed linguistic structure and sentiment information of the comments.

[1510] Step 3:

[1511] The server uses filtering techniques to filter out offensive and spam comments from the analyzed comments. Specifically, it identifies and filters out offensive and spam comments using pre-defined keywords and specific language patterns. The input is the analyzed comment text data and sentiment information, and the output is the filtered comment data.

[1512] Step 4:

[1513] The server summarizes the filtered comments using a summarization means. Specifically, it aggregates similar questions and feedbacks into a concise message. The input is the filtered comment data, and the output is the summarized comment data.

[1514] Step 5:

[1515] The server uses a sentiment analysis engine to recognize the emotional information (positive, negative, neutral) contained in viewers' comments. Specifically, it uses a machine learning algorithm to classify the emotions of the comments. The input is summarized comment data, and the output is comment data with emotional information added.

[1516] Step 6:

[1517] The server sends the summarized comments and emotional information to the broadcaster's terminal. The input is the comment data with the emotional information added, and the output is the comment information displayed on the broadcaster's terminal.

[1518] Step 7:

[1519] The terminal receives the summary comment and emotion information sent from the server and displays them on the screen. The broadcaster communicates with the viewer based on this information. The input is the summary comment and emotion information sent from the server, and the output is the comment information displayed on the screen.

[1520] Step 8:

[1521] The broadcaster responds to the viewer's comments. The response is sent from the terminal to the server and displayed in real time on the viewer's chat screen. The input is the broadcaster's response, and the output is a response message that is displayed on the viewer's chat screen.

[1522] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1523] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1524] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1525] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1526] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1527] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1528] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1529] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1530] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1531] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1532] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1533] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1534] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1535] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1536] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1537] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1538] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1539] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1540] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1541] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1542] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1543] The following is further disclosed regarding the above embodiment.

[1544] (Claim 1)

[1545] natural language processing means for analyzing comments received from viewers;

[1546] a filtering means for filtering out offensive comments and spam comments from the comments analyzed by the natural language processing means;

[1547] a summarizing means for summarizing the comments filtered by the filtering means;

[1548] a sending means for sending the summarized comment to a distributor;

[1549] A system including:

[1550] (Claim 2)

[1551] The system of claim 1 , wherein the filtering means further comprises means for filtering out irrelevant comments.

[1552] (Claim 3)

[1553] 10. The system of claim 1, wherein the summarizing means includes means for aggregating similar comments from multiple viewers.

[1554] (Claim 4)

[1555] 2. The system of claim 1, wherein the transmitting means includes means for transmitting the summarized comments to a broadcaster's terminal in real time.

[1556] (Claim 5)

[1557] 10. The system of claim 1, wherein the summarizing means includes means for prioritizing comments based on importance.

[1558] "Example 1"

[1559] (Claim 1)

[1560] natural language processing means for analyzing comments received from viewers;

[1561] a filtering means for filtering out offensive comments and spam comments from the comments analyzed by the natural language processing means;

[1562] a summarizing means for summarizing the comments filtered by the filtering means;

[1563] a sending means for sending the summarized comment to a distributor;

[1564] a receiving means for receiving and displaying the summary comment transmitted by the transmitting means;

[1565] a transmitting means for transmitting to a server a response from the distributor to the comment received by the receiving means;

[1566] a display means for receiving a response from the distributor via the server and displaying the response on the chat screen of the viewer in real time;

[1567] A system including:

[1568] (Claim 2)

[1569] The system of claim 1 , wherein the filtering means further comprises means for filtering out irrelevant comments.

[1570] (Claim 3)

[1571] 10. The system of claim 1, wherein the summarizing means includes means for aggregating similar comments from multiple viewers.

[1572] "Application Example 1"

[1573] New Claims

[1574] (Claim 1)

[1575] natural language processing means for analyzing comments received from viewers;

[1576] a filtering means for filtering out offensive comments and spam comments from the comments analyzed by the natural language processing means;

[1577] a summarizing means for summarizing the comments filtered by the filtering means;

[1578] a summarization enhancement means that uses a generative AI model to further summarize the comments;

[1579] a sending means for sending the comments summarized by the summarizing means and the summary enhancing means to a distributor;

[1580] A system including:

[1581] (Claim 2)

[1582] The system of claim 1 , wherein the filtering means further comprises means for filtering out irrelevant comments.

[1583] (Claim 3)

[1584] 10. The system of claim 1, wherein the summarizing means and the summarization enhancement means include means for aggregating similar comments from multiple viewers.

[1585] "Example 2: Combining Emotion Engines"

[1586] (Claim 1)

[1587] natural language processing means for analyzing comments received from viewers;

[1588] a filtering means for filtering out offensive comments and spam comments from the comments analyzed by the natural language processing means;

[1589] an analysis means for performing sentiment analysis on the comments filtered by the filtering means;

[1590] a summarizing means for summarizing the comments whose emotions have been determined by the analyzing means;

[1591] a sending means for sending the summarized comment to a distributor;

[1592] a distribution means for receiving responses from the distributor and transmitting them to the viewers;

[1593] A real-time processing means for processing comments from viewers and responses from broadcasters in real time;

[1594] A system including:

[1595] (Claim 2)

[1596] The system of claim 1 , wherein the filtering means further comprises means for filtering out irrelevant comments.

[1597] (Claim 3)

[1598] 10. The system of claim 1, wherein the summarizing means includes means for aggregating similar comments from multiple viewers.

[1599] "Application example 2 when combining emotion engines"

[1600] (Claim 1)

[1601] natural language processing means for analyzing comments received from viewers;

[1602] a filtering means for filtering out offensive comments and spam comments from the comments analyzed by the natural language processing means;

[1603] a summarizing means for summarizing the comments filtered by the filtering means;

[1604] a transmitting means for transmitting the comment summarized by the summarizing means and its emotional information to a distributor;

[1605] A sentiment analysis engine that analyzes viewers' comments and recognizes their emotional information (positive, negative, neutral),

[1606] A system including:

[1607] (Claim 2)

[1608] The system of claim 1 , wherein the filtering means further comprises means for filtering out irrelevant comments.

[1609] (Claim 3)

[1610] 10. The system of claim 1, wherein the summarizing means includes means for aggregating similar comments from multiple viewers. [Explanation of symbols]

[1611] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. natural language processing means for analyzing comments received from viewers; a filtering means for filtering out offensive comments and spam comments from the comments analyzed by the natural language processing means; a summarizing means for summarizing the comments filtered by the filtering means; a sending means for sending the summarized comment to a distributor; A system including:

2. The system of claim 1 , wherein the filtering means further comprises means for filtering out irrelevant comments.

3. 2. The system of claim 1, wherein the summarizing means includes means for aggregating similar comments from multiple viewers.

4. 2. The system of claim 1, wherein the transmitting means includes means for transmitting the summarized comments to a broadcaster's terminal in real time.

5. 2. The system of claim 1, wherein the summarizing means includes means for prioritizing comments based on importance.

Images