System

Abstract

A system is provided.SOLUTION: A system comprising: means for acquiring discussion contents of a local congress; means for converting the acquired discussion contents into text by voice recognition; means for summarizing the discussion contents converted into text by using natural language processing; means for adding an advertisement to a summary; means for distributing the summary to a user; means for posting an evaluation or a comment to the summary by the user; and means for storing the evaluation and the comment in a database.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, it has become increasingly important to quickly and accurately communicate the content of discussions in local assemblies to citizens and to create an environment in which citizens can easily provide feedback on the content of discussions. However, currently, it takes time to understand the content of discussions, and there are limited opportunities for citizens to express their opinions. Furthermore, there is a lack of infrastructure to effectively reflect citizen opinions. As a result, there is a lack of communication between local governments and citizens, and issues such as insufficient government transparency and citizen participation have arisen. [Means for solving the problem]

[0005] The present invention provides a system that acquires the content of discussions in local assemblies, converts them into text using speech recognition, and summarizes them using natural language processing. This allows citizens to quickly understand the content of the discussions. Furthermore, by adding advertisements to the summaries and distributing them to users, it stimulates local economic activity. Furthermore, it provides a function that allows users to post ratings and comments on the summaries, and stores the ratings and comments in a database. This system allows citizens to easily provide feedback on the content of the discussions, improving two-way communication between local governments and citizens and promoting government transparency and citizen participation. Furthermore, by including means for moderating user-submitted comments and means for managing advertising data and analyzing interaction data, it enables appropriate information management and effective advertising management.

[0006] "Local assembly discussion content" refers to a series of information such as discussions, debates, and resolutions that take place in local government assemblies.

[0007] "Means for acquiring" refers to a method or device for capturing discussion content, including audio and video data, from an external source into the system.

[0008] "Speech recognition" refers to the technology of analyzing voice data and converting it into appropriate text data.

[0009] "Text conversion means" refers to a process or device that converts acquired voice data into text using voice recognition technology.

[0010] "Natural language processing" refers to the field of computer science that analyzes text data and extracts semantically significant elements.

[0011] A "summarization tool" refers to a process or device that uses natural language processing to extract key points from text data and summarize them in a concise form.

[0012] "Means for adding advertising" refers to a process or device that incorporates advertising information for local businesses and services into the summary text.

[0013] The "means for delivering to the user" refers to a process or system that transmits the generated summary and advertisement to the user terminal.

[0014] "Means for posting ratings or comments" refers to the process or function by which a user inputs their thoughts or feedback on the discussion content and sends it to the system.

[0015] "Means for storing in a database" refers to a storage system for persistently storing rating and comment data and the process for managing it.

[0016] "Moderation measures" refers to algorithmic or human processes used to review user-submitted comments and filter out inappropriate content.

[0017] "Means for managing advertising data" refers to a process or system for organizing, storing, and distributing advertising information provided by local businesses.

[0018] "Means for analyzing interaction data" refers to the processes or technologies used to collect user behavior data and analyze statistics and trends. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] The present invention is a system that acquires the contents of discussions in local assemblies, summarizes them so that citizens can quickly understand them, and provides feedback on their opinions and evaluations. An example is shown below.

[0041] System Configuration

[0042] Server: Acquires parliamentary data, converts it into text using speech recognition, and summarizes it using natural language processing. It also adds advertisements to the summarized discussion content and delivers it to users.

[0043] Device: A device on which users can view summaries of discussions and enter ratings and comments. This includes smartphones, tablets, and PCs.

[0044] Users: Citizens who use the application to view summaries of local council discussions and rate and comment on them.

[0045] Processing content

[0046] 1. Acquiring discussion content

[0047] The server retrieves audio and video data through the API after each local council session, either in real time or periodically.

[0048] 2. Text conversion using voice recognition

[0049] The server passes the acquired voice data to a voice recognition engine and converts it into text data, which is used in the next step.

[0050] 3. Natural Language Processing Summarization

[0051] The server passes the converted data to a natural language processing engine, extracts key points, and generates a summary based on the extracted points. This summary is of an appropriate length to convey the content concisely.

[0052] 4. Adding Ads

[0053] The server adds advertising information provided by local businesses to the abstracts, which are appropriately inserted into each abstract and positioned to attract the user's attention.

[0054] 5. Summary distribution

[0055] The server delivers the generated summary and advertisement to registered users via push notification, which includes the title of the summary and a brief introduction.

[0056] 6. Accepting ratings and comments

[0057] Users tap the push notification on their device, open the app, and view the summary. They can then rate or comment using the "Like" button or comment field below the summary.

[0058] The terminal transmits the ratings and comments entered by the user to the server.

[0059] 7. Data Retention

[0060] The server stores the submitted ratings and comments in a database, which is later used for analysis and engagement evaluation.

[0061] 8. Moderation and Ad Management

[0062] The server moderates each new comment as it is posted, filtering out inappropriate content, and also manages advertising data and generates reports for advertisers.

[0063] Specific examples

[0064] For example, if a new park development plan is discussed at a local assembly on July 5th, the server will acquire the audio data after the assembly ends. Next, a speech recognition engine will convert the audio into text, and a natural language processing engine will extract key points and generate a summary. After that, the opinions and ratings of User A, who posted a comment saying, "I'm happy that a new park will be built!", will be moderated and saved in the database. An advertisement for a local cafe will be added to this summary and sent to User B via push notification.

[0065] This system will enable discussions in local assemblies to be communicated to citizens quickly and efficiently, creating an environment in which citizen feedback can be reflected in local administration.

[0066] The processing flow will be explained below.

[0067] Step 1:

[0068] After the local assembly session ends, the server uses the API to retrieve the audio data, which includes the contents of the assembly discussion and is later converted into text.

[0069] Step 2:

[0070] The server passes the acquired voice data to a voice recognition engine, which converts it into text data in real time or in batches. The voice recognition engine analyzes the content of the voice and outputs it as text information.

[0071] Step 3:

[0072] The server passes the text data output by the speech recognition engine to the natural language processing engine, which extracts important points from the text data and generates a summary. This summary is a concise summary of the content.

[0073] Step 4:

[0074] The server adds advertising information provided by local businesses to the generated abstract, which is then formatted to be displayed with the abstract and delivered to the user.

[0075] Step 5:

[0076] The server compiles the summary and advertisement and sends a push notification to registered users, which includes the title of the summary and a brief intro designed to grab the user's attention.

[0077] Step 6:

[0078] A user receives a push notification on their device and taps the notification to open the application, which displays a summary and an advertisement on the screen.

[0079] Step 7:

[0080] The user views the summary and taps the "Like" button. If necessary, the user can enter their opinion or feedback in the comment field and tap the submit button. This inputs the user's feedback.

[0081] Step 8:

[0082] The device sends the user's rating (likes) and comment data to the server, which is then recorded as user feedback.

[0083] Step 9:

[0084] The server stores the submitted rating and comment data in a database, which is later used for analysis and feedback evaluation.

[0085] Step 10:

[0086] The server moderates newly submitted comments to check for inappropriate content. Inappropriate comments are filtered out, and only appropriate comments are saved.

[0087] Step 11:

[0088] The server manages the advertising data and collects user interaction data, which is used to evaluate the effectiveness of the advertisements and generate reports as feedback to the advertisers.

[0089] Through these steps, this system provides a mechanism for quickly and efficiently communicating the contents of local assembly discussions to citizens and reflecting citizen feedback in the administration.

[0090] Example 1

[0091] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0092] While the content of local assembly discussions is important information for citizens, it is difficult to quickly and concisely grasp this information. Furthermore, there are limited ways for citizens to provide feedback on the content of discussions, resulting in a lack of two-way communication with local governments. Furthermore, local assembly information is not properly integrated with external advertising, limiting opportunities for local advertisers. To solve these issues, an appropriate system is needed.

[0093] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0094] In this invention, the server includes means for acquiring the contents of local assembly discussions, means for converting the acquired discussion contents into text using speech recognition, means for summarizing the converted text using natural language processing, means for adding advertisements to the summaries, means for distributing the summaries to users, means for users to post ratings or comments on the summaries, means for saving the ratings and comments in a database, means for moderating the content of the ratings and comments, and means for appropriately placing advertisements together with the summaries. This allows for quick and concise provision of local assembly information to citizens, smooth collection of feedback from citizens, and two-way communication. It also allows for effective provision of information to local advertisers.

[0095] A "local assembly" refers to the assembly of a local government, an institution that discusses and makes decisions on local public policies, budgets, etc.

[0096] "Discussion content" refers to all statements and debates made by members of parliament and representatives in local assemblies, as well as all related information.

[0097] A "means" refers to a device, technique, method, or process used to accomplish a particular purpose.

[0098] "Speech recognition" refers to the technology of converting speech into text data.

[0099] "Text conversion" refers to the conversion of audio information extracted from audio data or video data into readable text information.

[0100] "Natural language processing" refers to the technology that enables computers to understand, interpret, and generate human language.

[0101] A "summary" is a short, concise summary of the main content.

[0102] "Advertising" refers to the means of providing information to publicize a product, service, or business.

[0103] "Distribution" refers to the act of transmitting information so that specific users receive the information.

[0104] "Users" refer to people who access the system, consume the information provided, and provide feedback such as ratings and comments.

[0105] "Feedback" refers to reactions to the system and its offerings, such as ratings and comments provided by users.

[0106] "Evaluation" refers to the act of a user expressing their opinion or impression about the provided summary or content.

[0107] "Comment" refers to the act of a user inputting detailed opinions or impressions about the provided summary or content in words.

[0108] A "database" refers to an electronic system for systematically storing and managing information.

[0109] "Moderation" refers to the process of reviewing comments and feedback provided by users and determining whether they are appropriate.

[0110] "Advertiser" refers to an individual or organization that provides advertising and seeks to profit from such advertising.

[0111] "Interaction data" refers to information about user actions and reactions, which is a record of the interaction between a system and a user.

[0112] The present invention is a system that quickly and easily provides citizens with the content of discussions in local assemblies and collects their feedback. This system functions in cooperation with three parties: a server, a terminal, and a user. Specific embodiments are described below.

[0113] Server Functions and Processing

[0114] 1. Acquiring discussion content

[0115] The server retrieves audio and video data via an API each time a local council session ends. This data is collected in real time or periodically. The API can be publicly available from the local council or from a dedicated data provider.

[0116] 2. Text conversion using voice recognition

[0117] The server passes the acquired voice data to a voice recognition engine (e.g., a voice recognition cloud service) and converts it into text data, which is used for subsequent processing.

[0118] 3. Natural Language Processing Summarization

[0119] The server passes the converted discussion content to a natural language processing engine (e.g., a natural language processing library), extracts key points, and generates a summary. The summary is adjusted to an optimal length to convey the discussion content concisely.

[0120] 4. Adding Ads

[0121] The server adds advertisements from local vendors to the generated summary, and the advertisements are appropriately placed to attract the user's interest.

[0122] 5. Summary distribution

[0123] The server delivers the generated summary and advertisement to registered users via push notification (e.g., push notification cloud service).

[0124] 6. Accepting ratings and comments

[0125] The server accepts user ratings and comments on the summaries and stores them in a database, which is later used for analysis and engagement evaluation.

[0126] 7. Moderation and Ad Management

[0127] The server moderates content as new comments are posted, filtering out inappropriate content, and also manages advertising data and generates reports for advertisers.

[0128] Terminal functions and processing

[0129] 1. View the summary and enter your rating and comments

[0130] The device receives the push notification, and when the user taps the notification, the application opens and displays the summary. The user can then rate or comment on the summary using the "Like" button or comment field. The entered rating or comment is then sent to the server.

[0131] User operations

[0132] 1. Review and feedback on the summary

[0133] Users receive a push notification from their device and can review the summary. They can rate the summary and add comments, providing feedback on the local council discussion.

[0134] Specific examples

[0135] For example, if a new park development plan is discussed at a local assembly on July 5th, the server will obtain the audio data via an API after the assembly ends. Next, a speech recognition engine will convert the audio into text, and a natural language processing engine will extract key points and generate a summary. After that, User A's comment, "I'm happy that a new park will be built!", will be moderated and saved in the database. This summary, along with an advertisement for a local cafe, will be sent to User B via a push notification.

[0136] Prompt Sentence Examples

[0137] "A new park development plan was discussed at the local council meeting on July 5th. Please summarize the key points of the discussion."

[0138] This system will enable important information from local councils to be communicated to citizens quickly and efficiently, creating an environment in which citizen feedback can be reflected in local administration.

[0139] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0140] Step 1:

[0141] The server acquires the contents of discussions in local assemblies.

[0142] Specific operation: The server uses the API published by the local council to obtain audio and video data in real time or periodically.

[0143] Input: Local council audio and video data.

[0144] Output: Audio data file.

[0145] Step 2:

[0146] The server passes the acquired voice data to a voice recognition engine and converts it into text data.

[0147] Specific operation: The server sends the voice data to the voice recognition cloud service and obtains the corresponding text data.

[0148] Input: Audio data file.

[0149] Output: Text data.

[0150] Step 3:

[0151] The server passes the converted discussion content to a natural language processing engine, extracts important points, and generates a summary.

[0152] How it works: The server inputs text data into a natural language processing library and generates a summary of appropriate length. The summary is constructed by scoring important keywords and sentences and extracting the top content.

[0153] Input: Text data.

[0154] Output: Summary statement.

[0155] Step 4:

[0156] The server adds advertisements provided by local businesses to the summary.

[0157] What it does: The server extracts relevant ads from an ad database and inserts them into specific parts of the summary. Relevance is determined by taking into account keywords and themes that interest the user.

[0158] Input: Abstract, ad data.

[0159] Output: Summary text with ad added.

[0160] Step 5:

[0161] The server delivers the generated summary and advertisement to the user via push notification.

[0162] Specific operation: The server retrieves the device tokens of all users from the user database, and uses the push notification cloud service to generate and deliver notifications containing summary text and advertisements.

[0163] Input: Ad summary, user device token.

[0164] Output: Push notification.

[0165] Step 6:

[0166] The user receives a push notification on their device, opens the app, and views the summary.

[0167] What happens: The user taps the notification and sees the summary in the app, which includes an ad.

[0168] Enter: push notification.

[0169] Output: Summary and advertisement.

[0170] Step 7:

[0171] The user inputs an evaluation and comments on the summary.

[0172] What it does: Users use the "Like" button and comment field below the summary to enter their own rating or comment.

[0173] Input: User rating and comments.

[0174] Output: The input rating and comment data.

[0175] Step 8:

[0176] The terminal transmits the ratings and comments entered by the user to the server.

[0177] Specific operation: The entered ratings and comments are sent from the app to the server, where the data received is stored in a database.

[0178] Input: Rating and comment data.

[0179] Output: Send to server.

[0180] Step 9:

[0181] The server stores the submitted ratings and comments in a database.

[0182] Specific operation: The server stores the received rating and comment data in the corresponding location in the database.

[0183] Input: Rating and comment data.

[0184] Output: Database update.

[0185] Step 10:

[0186] The server moderates each new comment as it is posted, filtering out inappropriate content.

[0187] What it does: Moderation algorithms are applied to automatically identify and filter inappropriate keywords and content, with manual review where necessary.

[0188] Input: Rating and comment data.

[0189] Output: Moderated comment data.

[0190] Step 11:

[0191] The server manages the advertising data and generates reports for the advertisers.

[0192] Specific operation: The server collects and analyzes the logs of ad display, and automatically generates and sends reports to advertisers.

[0193] Input: Ad display log.

[0194] Output: Report to advertiser.

[0195] (Application example 1)

[0196] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0197] The content of discussions in local assemblies is difficult for many citizens to understand, and it is difficult to follow in real time. There is also a lack of mechanisms for citizens to quickly have their opinions reflected in local government. Furthermore, there are limited means to increase the effectiveness of advertising for local businesses. It is necessary to resolve these issues, increase citizen engagement, and promote transparency in local government and the sharing of discussion content.

[0198] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0199] In this invention, the server includes means for acquiring the contents of discussions in local assemblies, means for converting the acquired contents into text using speech recognition, means for summarizing the converted text using natural language processing, means for adding advertisements to the summaries, means for distributing the summaries to users, means for users to post ratings or comments on the summaries, means for storing the ratings and comments in a database, means for displaying the contents of discussions in local assemblies in a virtual space, and means for users to provide feedback in the virtual space. This allows citizens to quickly and efficiently understand the contents of discussions in local assemblies and provide feedback of their opinions in real time. It also enhances the advertising effectiveness of local businesses.

[0200] A "local assembly" is an institution in a local government that deliberates and decides on bills.

[0201] "Discussion content" refers to the various issues, proposals, and opinions that are deliberated in local assemblies.

[0202] "Speech recognition" is a technology that converts speech into text.

[0203] "Text conversion" refers to the process of converting audio or video data into text information.

[0204] "Natural language processing" is a technology that allows computers to understand, analyze, and generate human language.

[0205] "Summarization" refers to extracting important parts of text data and summarizing them concisely.

[0206] "Advertising" is promotional information intended to widely publicize a particular product or service.

[0207] "Distribution" refers to the transmission of information or data to multiple users.

[0208] "User" refers to citizens or individual users of the system.

[0209] "Evaluation" refers to a user expressing an opinion about a particular piece of information or service.

[0210] "Comments" refers to opinions or feedback provided by users.

[0211] A "database" is a collection of data that allows information to be managed efficiently and easily searched and updated.

[0212] "Virtual space" refers to a virtual three-dimensional space created using computer technology.

[0213] "Feedback" refers to reactions such as ratings and comments from users.

[0214] The system for implementing this invention consists of three main elements: a server, a terminal, and a user.

[0215] Server Features

[0216] The server efficiently retrieves the contents of local assembly discussions, converts them into text using speech recognition, then generates summaries using a natural language processing engine, adds advertisements to the summaries, and delivers them to users in the virtual space.

[0217] Specifically, the process is as follows: The server passes the voice data obtained from the local assembly to a voice recognition engine (for example, Google's voice recognition API) and converts it into text data. This text data is then used by OpenAI's natural language processing engine (such as GPT-3) to extract key points and generate a summary. Advertising information from local businesses is then appropriately inserted into the generated summary.

[0218] Device Features

[0219] The device functions as a device for users to view the summary and post ratings and comments. It can be a smartphone, tablet, or PC. When a user taps the push notification, an application opens and the summary is displayed. The user can then use the "Like!" button or comment field at the bottom of the summary to enter a rating or comment.

[0220] Virtual space functions

[0221] The virtual space provides a place where users can experience the contents of local assembly discussions in a virtual three-dimensional space. Specifically, there is a virtual assembly hall where the discussion contents are displayed, and users gathered there can view the discussion contents in real time and provide feedback. This can be achieved, for example, through a VR headset or a smartphone app.

[0222] example

[0223] For example, if a local assembly is debating a new city park plan, the server captures the audio data after the assembly. The audio data is converted into text using Google's speech recognition API, and a summary is generated using GPT-3. The summary is then supplemented with an advertisement for a local cafe and delivered to the user in the virtual space.

[0224] Users receive a push notification through their smartphone app and open it to view the summary. The user's comment, "I'm happy that a new park will be built!", is sent from the device to a server and stored in a database. This data will later be analyzed by the local government and used as part of their feedback.

[0225] Prompt Sentence Examples

[0226] Here is an example prompt for generating a summary:

[0227] "Summarize the following text: Today, the local assembly discussed the design of a new park. Council members exchanged opinions on the park's location, design, and budget, and debated points to improve the quality of life for residents."

[0228] This system allows citizens to quickly and efficiently understand the content of discussions in local assemblies and provide feedback on their opinions in real time. It can also increase the effectiveness of advertising for local businesses.

[0229] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0230] Step 1:

[0231] The server acquires audio data of the local assembly discussions. After each assembly, audio and video data are collected in real time or periodically through the API. In this step, the audio data is the input, and the output is the audio file stored on the server.

[0232] Step 2:

[0233] The server passes the acquired voice data to a voice recognition engine and converts it into text data. Specifically, it uses Google's voice recognition API to analyze the voice data and convert the discussion content into text. In this step, voice data is the input and text data is the output.

[0234] Step 3:

[0235] The server passes the converted data to a natural language processing engine, extracts key points, and generates a summary. Specifically, OpenAI's GPT-3 is used to generate a summary of the text data. In this step, the text data is the input and the summary is the output.

[0236] Step 4:

[0237] The server adds advertising information provided by local businesses to the summary. The advertising information is inserted appropriately in a position that is likely to attract the user's interest. In this step, the summary and advertising data are the input, and the summary with the added advertisement is the output.

[0238] Step 5:

[0239] The server delivers the generated summary and advertisement to registered users' devices via push notification. The push notification includes the title of the summary and a brief introduction. In this step, the summary with advertisement is the input, and the push notification to the user's device is the output.

[0240] Step 6:

[0241] The user taps the push notification on their device, opens the application, and reads the summary. The device displays the summary and provides an interface where the user can rate or comment using the "Like" button and comment field. In this step, receiving the push notification is the input, and the display of the summary and the interface for comment input are the outputs.

[0242] Step 7:

[0243] The terminal sends the ratings and comments entered by the user to the server. The ratings and comments entered by the user on the interface are generated as data and then sent. In this step, the user's ratings and comments are the input, and the data sent to the server is the output.

[0244] Step 8:

[0245] The server stores the submitted ratings and comments in a database. The stored data is later used for analysis and engagement evaluation. In this step, the rating and comment data are the input and the storage in the database is the output.

[0246] Step 9:

[0247] The server moderates each new comment as it is posted, filtering out inappropriate content, and also manages advertising data and generates reports for advertisers. In this step, user comments and advertising data are the input, and filtered comments and reports for advertisers are the output.

[0248] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0249] This invention combines an emotion engine with a system that acquires and summarizes the contents of local assembly discussions and distributes them to citizens, and collects and analyzes feedback from citizens. An example is shown below.

[0250] System Configuration

[0251] Server: This server has the function of acquiring parliamentary data, converting it into text using speech recognition, and summarizing it using natural language processing. It also adds advertisements to the summaries and delivers them to users. It also moderates user comments and analyzes their sentiment using an emotion engine.

[0252] Device: A device on which users can view summaries of discussions and enter ratings and comments. This includes smartphones, tablets, and PCs.

[0253] Users: Citizens who use the application to view summaries of local council discussions and provide ratings and comments.

[0254] Specific processing of the program

[0255] 1. Acquiring discussion content

[0256] After the local assembly session ends, the server uses an API to retrieve the audio data, which includes the contents of the assembly discussion and is then converted into text in a later process.

[0257] 2. Text conversion using voice recognition

[0258] The server passes the acquired voice data to a voice recognition engine, which converts it into text data in real time or in batches. The voice recognition engine analyzes the content of the voice and outputs it as text information.

[0259] 3. Natural Language Processing Summarization

[0260] The server passes the text data output by the speech recognition engine to the natural language processing engine, which extracts important points from the text data and generates a summary. This summary is a concise summary of the content.

[0261] 4. Adding Ads

[0262] The server adds advertising information provided by local businesses to the generated abstract, which is then formatted to be displayed with the abstract and delivered to the user.

[0263] 5. Summary distribution

[0264] The server compiles the summary and advertisement and sends a push notification to registered users, which includes the title of the summary and a brief intro designed to grab the user's attention.

[0265] 6. Accepting ratings and comments

[0266] A user receives a push notification on their device and taps the notification to open the application, which displays a summary and an advertisement on the screen.

[0267] The user views the summary and taps the "Like" button. If necessary, the user can enter their opinion or feedback in the comment field and tap the submit button. This inputs the user's feedback.

[0268] 7. Data transmission and storage

[0269] The device sends the user's rating (likes) and comment data to the server, which is then recorded as user feedback.

[0270] The server stores the submitted rating and comment data in a database, which is later used for analysis and feedback evaluation.

[0271] 8. Comment Moderation

[0272] The server moderates newly submitted comments to check for inappropriate content. Inappropriate comments are filtered out, and only appropriate comments are saved.

[0273] 9. Sentiment Analysis

[0274] The server passes the user's comment data to the emotion engine for emotion analysis. The emotion engine analyzes the content of the comment and identifies the emotion contained therein (e.g., joy, anger, sadness, etc.).

[0275] The server evaluates citizens' emotional reactions to the content of the discussion based on the emotional data analyzed by the emotion engine.

[0276] 10. Adjusting Ad Display

[0277] The server then adjusts the content of the advertisements based on the emotion data identified by the emotion engine. For example, if a user has a positive opinion, the server displays an advertisement that corresponds to that opinion, thereby personalizing the advertisements.

[0278] Specific examples

[0279] For example, if a new park development plan is discussed at a local assembly on July 5th, the server will collect the audio data after the assembly ends. The audio will then be converted into text using a speech recognition engine, and a natural language processing engine will extract key points to generate a summary. An advertisement for a local cafe will then be added to the summary, and the summary and advertisement will be sent to User A via push notification.

[0280] User A taps the notification, reads the summary in the application, and taps the "Like" button. He also posts a comment saying, "I'm so happy that a new park is opening!" The device sends the comment data to the server. The server stores the comment data in a database and simultaneously passes the comment to the emotion engine for analysis. The emotion engine identifies the emotion "joy," and the server uses this emotion data to adjust the display of more relevant ads to the same User A the next time.

[0281] Through this series of steps, the system not only quickly and efficiently conveys the contents of discussions in local assemblies to citizens and reflects citizen feedback in the administration, but also grasps their emotional reactions and further improves the accuracy of advertising.

[0282] The processing flow will be explained below.

[0283] Step 1:

[0284] The server uses an API to retrieve audio data after the local assembly session has ended. This audio data contains the contents of the assembly discussions and is stored for later processing to convert it into text.

[0285] Step 2:

[0286] The server passes the acquired voice data to a voice recognition engine, which converts it into text data in real time or in batches. The voice recognition engine analyzes the content of the voice and outputs the corresponding text information.

[0287] Step 3:

[0288] The server passes the text data output by the speech recognition engine to a natural language processing engine, which extracts important points from the text data and generates a concise summary of the discussion.

[0289] Step 4:

[0290] The server adds advertising information provided by local businesses to the generated abstract, which is then formatted to be displayed in the appropriate location along with the abstract.

[0291] Step 5:

[0292] The server compiles the summary and advertisement and sends a push notification to registered users, which includes the title of the summary and a brief intro designed to grab the user's attention.

[0293] Step 6:

[0294] A user receives a push notification on their device and taps the notification to open the application, which displays a summary and an advertisement on their device screen.

[0295] Step 7:

[0296] The user views the summary and taps the "Like" button. If necessary, the user can enter their opinion or feedback in the comment field and tap the submit button. This inputs the user's feedback.

[0297] Step 8:

[0298] The device sends the user's ratings (likes) and comment data to the server, which is then recorded as user feedback.

[0299] Step 9:

[0300] The server stores the submitted rating and comment data in a database, which is later used for analysis and engagement measurement.

[0301] Step 10:

[0302] The server moderates newly submitted comments to check for inappropriate content, filtering out inappropriate comments and keeping only appropriate comments.

[0303] Step 11:

[0304] The server passes the user's comment data to the emotion engine, which analyzes the content of the comment and identifies the emotions contained therein (e.g., joy, anger, sadness, etc.).

[0305] Step 12:

[0306] The server uses the emotion data identified by the emotion engine to assess citizens' emotional reactions to the content of the discussion, thereby understanding how citizens feel.

[0307] Step 13:

[0308] The server adjusts the content of the advertisements based on the emotional data. For example, if the user has a positive opinion, the server will display an advertisement that corresponds to that opinion, thereby personalizing the advertisements.

[0309] To give a concrete example, if a new park development plan is discussed at a local assembly on July 5th, the server will acquire the voice data after the assembly ends and convert it into text using a speech recognition engine. The server will then generate a summary using a natural language processing engine, add an advertisement for a local cafe to the summary, and send a push notification to User A. User A taps the notification to view the summary, then enters a comment such as "I'm so happy that a new park is being built!" and sends it. The server will pass this comment to the emotion engine, identify the emotion of joy, and adjust and deliver relevant advertisements to User A next time. By obtaining feedback using the emotion engine in this way, it is possible to understand citizens' reactions and deliver appropriate advertisements, thereby improving the effectiveness of the entire system.

[0310] Example 2

[0311] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0312] The content of discussions in local assemblies is important public information that should be shared with citizens. However, there is a need for a method to efficiently collect, summarize, and quickly provide this content to citizens. There is also a need for a method to improve government services by obtaining feedback from citizens and analyzing their emotional reactions. Current systems have difficulty meeting these requirements, and there are challenges, particularly with the additional time and effort required for sentiment analysis and advertising personalization.

[0313] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0314] In this invention, the server includes: means for acquiring the contents of discussions in local assemblies; means for converting the acquired contents of discussions into text using speech recognition; means for summarizing the converted text using natural language processing; means for adding advertisements to the summaries; means for distributing the summaries to users; means for users to post ratings or comments on the summaries; means for saving the ratings and comments in a database; means including a sentiment analysis engine for analyzing the saved rating and comment data and identifying sentiment; and means for adjusting the content of advertisements based on the results of the sentiment analysis. This makes it possible to quickly and efficiently provide the contents of discussions in local assemblies to citizens, and to deeply understand citizen feedback and reflect it in administrative services.

[0315] A local assembly is an administrative body in a specific region where representatives of local residents gather to deliberate and decide on local laws, budgets, policies, etc.

[0316] "Discussion content" refers to the totality of information and opinions that are taken up as topics in local assemblies and debated among participants.

[0317] A "means" is a method, process, or device used to achieve a particular purpose.

[0318] "Speech recognition" is a technology that analyzes voice data and converts it into text data.

[0319] "Text conversion" is the process of analyzing non-textual information such as audio data and image data and converting it into textual information.

[0320] "Natural language processing" is a field of technology in which computers understand, analyze, and generate human language.

[0321] "Summarization" refers to the process of extracting important information from long text data and summarizing it in a short, concise form, or the result of that process.

[0322] An "advertisement" is a message intended to inform the general public about a particular product, service, or information.

[0323] "Users" refers to local residents and the general public who use this system.

[0324] "Distribution" is the process of getting information or data to a specific recipient.

[0325] "Evaluation" is the act of a user giving positive or negative feedback on the content of a discussion or a summary.

[0326] A "comment" is an act in which a user writes down their own opinion or impression on the content of a discussion or a summary.

[0327] A "database" is a digital system for efficiently storing, retrieving, updating, and managing large amounts of data.

[0328] An "emotion analysis engine" is a technology that analyzes text data and identifies the emotions contained within it (e.g., joy, anger, sadness, etc.).

[0329] "Moderation" is the process of reviewing posted comments to check for inappropriate content.

[0330] This invention is a system that aims to improve administrative services by providing the contents of local assembly discussions to citizens quickly and efficiently and collecting and analyzing feedback from citizens. The components of the system and their specific operation are described below.

[0331] System Configuration

[0332] 1. Server

[0333] The server will play a central role in collecting and processing data from local council discussions. The server's main functions are as follows:

[0334] Acquiring discussion content

[0335] The server uses an API to retrieve audio data after a local assembly session ends. This audio data contains the contents of the assembly discussion and is later converted to text. Specifically, the server hits the API endpoint, sends parameters such as the date and session ID, and downloads the audio data. For example, the audio data is saved in MP3 format.

[0336] Voice Recognition

[0337] The server passes the acquired voice data to a speech recognition engine (e.g., Google Cloud Speech-to-Text) and converts the voice into text data. The server then puts the voice data into a queue for batch processing, and the speech recognition engine converts it into text. This text data is saved in the form of meeting minutes.

[0338] Natural Language Processing

[0339] The server passes the text data output from the speech recognition engine to a natural language processing engine (e.g., OpenAI GPT-4), extracts important points, and generates a summary. The server then prompts the engine with a prompt such as "Summarize the following text," and stores the generated summary within the system.

[0340] Adding Ads

[0341] The server adds advertising information provided by local businesses to the generated summary, searches for relevant advertisements (e.g., coupon information for local stores) in the advertisement database, and adds the advertisements to the end of the summary. The formatted summary and advertisements are saved in HTML or JSON format.

[0342] Summary distribution

[0343] The server uses a push notification service (e.g., Firebase Cloud Messaging) to deliver the summary and advertisement to the user. It obtains the registered user's device ID and sends a notification message containing the summary, advertisement title, and intro. The notification is displayed on the user's device.

[0344] Accepting ratings and comments

[0345] The server stores the ratings (likes) and comments that users make on summaries in a database, allowing us to use user feedback for future analysis.

[0346] Comment moderation

[0347] The server moderates newly submitted comments, using a text filtering library (e.g., CleanSpeak) to check for inappropriate content, rejecting comments that are deemed inappropriate, and storing only appropriate comments.

[0348] Sentiment analysis

[0349] The server passes the user's comment data to a sentiment analysis engine (e.g., IBM Watson Natural Language Understanding) to identify emotions from the content of the comment. The sentiment analysis engine analyzes emotions such as joy, anger, and sadness, and stores the analysis results in a database.

[0350] Ad display adjustment

[0351] The server adjusts the content of advertisements based on the analysis results of the emotion engine. For example, if the user expresses the emotion of "joy," it will display advertisements related to that emotion (such as ticket information for a new park opening event) as the next push notification or in-app advertisement.

[0352] 2. Terminal

[0353] The terminal is a device that allows users to view summaries of discussions and enter ratings and comments. This includes smartphones, tablets, and PCs.

[0354] 3. Users

[0355] Users are citizens who use the application to view summaries of local assembly discussions and provide ratings and comments.

[0356] Specific examples

[0357] For example, if a new park development plan is discussed at a local council meeting on July 5th, the specific process would be as follows:

[0358] 1. The server uses the API to obtain the parliamentary audio data.

[0359] 2. The server passes the acquired voice data to Google Cloud Speech-to-Text and converts it into text data.

[0360] 3. The server passes the text data to OpenAI GPT-4, which generates a summary with the prompt "Summarize the following text."

[0361] 4. The server adds advertisements for local stores to the generated summary.

[0362] 5. The server uses Firebase Cloud Messaging to send a push notification containing the summary and advertisement to User A.

[0363] 6. User A taps the notification, reads the summary in the app, taps "Like," and posts a comment saying, "I'm so happy that the new park is opening!"

[0364] 7. The device sends the comment data to the server, which stores it in a database.

[0365] 8. The server moderates comments and only appropriate comments are left.

[0366] 9. The server passes the comment data to the emotion engine to identify the emotion "joy."

[0367] 10. The server configures the next push notification to display an ad related to User A.

[0368] Prompt Sentence Examples

[0369] Here are some examples of prompts for generative AI models:

[0370] "A new park development plan was discussed at the local council meeting on July 5th. Please summarize the content of that discussion."

[0371] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0372] Step 1:

[0373] Acquiring discussion content

[0374] After the local council session ends, the server hits the API endpoint and sends parameters such as the date and session ID to retrieve the audio data. The input is the API endpoint and required parameters. The output is audio data in MP3 format. Specifically, an HTTP request is sent and the audio data is received as a response.

[0375] Step 2:

[0376] Speech recognition to text

[0377] The server places the acquired audio data into a batch processing queue and passes it to a speech recognition engine (e.g., Google Cloud Speech-to-Text). The input is audio data in MP3 format. The output is text data in minutes format. Specifically, the URL of the audio file is passed to the speech recognition engine, and the text data is received as the processing result. An API is called on the server and the results are saved.

[0378] Step 3:

[0379] Natural language processing summarization

[0380] The server passes the text data output from the speech recognition engine to a natural language processing engine (e.g., OpenAI GPT-4), and issues instructions using a prompt such as "Summarize the following text." The input is the text data of the minutes and the prompt. The output is a summary. Specifically, the text is sent to the natural language processing engine along with the prompt, and the generated summary is received.

[0381] Step 4:

[0382] Adding Ads

[0383] The server adds advertising information provided by local businesses to the generated summary. The input is the summary and an advertising database. The output is the summary with the advertising added. Specifically, it searches for relevant advertisements in the advertising database and adds the advertisements to the end of the summary. The formatted summary and advertisements are saved in HTML or JSON format.

[0384] Step 5:

[0385] Summary distribution

[0386] The server uses a push notification service (e.g., Firebase Cloud Messaging) to deliver the summary and advertisement to the user. The input is the summary, advertisement, and the registered user's device ID. The output is the result of sending the push notification. Specifically, a notification message containing the summary, advertisement title, and intro is created and sent via Firebase Cloud Messaging. The notification is then displayed on the user's device.

[0387] Step 6:

[0388] Accepting ratings and comments

[0389] The user taps the push notification, opens the application, and views the summary and advertisement. The input is the push notification and user actions (tap, comment). The output is rating data (likes) and comment data. Specifically, the user taps the "Like" button, or enters their opinion or feedback in the comment section and taps the send button. The feedback data is sent from the device to the server.

[0390] Step 7:

[0391] Data transmission and storage

[0392] The device sends user ratings (likes) and comment data to the server. The input is rating data (likes) and comment data. The output is the database update results. Specifically, the device receives ratings and comments, stores them in the appropriate fields, and saves them in the database. The saved data is used for later analysis.

[0393] Step 8:

[0394] Comment moderation

[0395] The server moderates newly submitted comments. The input is the comment data. The output is the result of only saving appropriate comments. Specifically, a text filtering library (e.g. CleanSpeak) is used to check for inappropriate content. Comments that are deemed inappropriate are rejected, and only appropriate comments are saved in the database.

[0396] Step 9:

[0397] Sentiment analysis

[0398] The server passes user comment data to a sentiment analysis engine (e.g., IBM Watson Natural Language Understanding) to identify emotions from the comments. The input is comment data. The output is the sentiment analysis results. Specifically, the comment data is sent to the sentiment analysis engine, which identifies emotions such as joy, anger, and sadness, receives the results, and stores them in a database.

[0399] Step 10:

[0400] Ad display adjustment

[0401] The server adjusts the ad content based on the emotion engine's analysis results. The input is the emotion analysis results. The output is the adjusted ad content. Specifically, based on the analysis results, if the user expresses the emotion of "joy," for example, the server sets the next push notification or in-app ad to display an ad related to that emotion (e.g., ticket information for a new park opening event).

[0402] (Application example 2)

[0403] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0404] While systems already exist for effectively communicating the content of local assembly discussions to citizens and efficiently collecting and analyzing their feedback, there is no established system for collecting customer feedback in brick-and-mortar stores and personalizing advertisements based on that feedback. As a result, brick-and-mortar stores are unable to quickly and appropriately reflect the diverse emotions and opinions of customers, and are unable to maximize the effectiveness of advertising. There is also a lack of automated systems for moderating inappropriate comments and personalizing advertisements based on sentiment analysis.

[0405] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0406] In this invention, the server includes means for acquiring the content of discussions in local assemblies, means for converting the acquired content of discussions into text using speech recognition, means for summarizing the converted text of discussions using natural language processing, means for adding advertisements to the summaries, means for distributing the summaries to users, means for users to post ratings or comments on the summaries, means for saving the ratings and comments in a database, means for collecting customer feedback from physical stores, means for moderating the collected feedback, means for sentiment analysis of the feedback data, and means for personalizing advertisements based on the sentiment analysis. This makes it possible to quickly and appropriately reflect customer feedback even in physical stores and personalize advertisements based on the results.

[0407] "Local assembly discussions" are records of the content of discussions held by assembly members and other stakeholders in local government assemblies.

[0408] "Speech recognition" is a technology that analyzes voice data and converts it into corresponding text data.

[0409] "Natural language processing" is a technology that analyzes text data and automatically understands its meaning and structure.

[0410] A "summary" is a short summary of the main points extracted from a long text.

[0411] "Adding advertisements" refers to the technique or act of inserting advertisements into abstracts or other content.

[0412] "Delivering to the user" refers to the technique or act of sending the created summary and related information to the user's terminal.

[0413] Posting a "rating or comment" means that a user inputs and sends their opinion or feedback on the content.

[0414] "Storing in a database" means recording the collected evaluations and comments in digital form so that they can be referenced and analyzed as needed.

[0415] "Brick-and-mortar customer feedback" refers to the opinions and evaluations that customers give about their experiences and products in brick-and-mortar stores.

[0416] "Moderation" refers to the process of examining the content of comments posted by users and removing or modifying inappropriate comments.

[0417] "Sentiment analysis" is a technology that identifies and analyzes emotional elements (e.g., joy, anger, sadness, etc.) from text data.

[0418] "Personalizing ads" means customizing advertising content based on the characteristics and behavior of individual users.

[0419] This invention is a system that acquires the contents of discussions in local assemblies, efficiently conveys them to citizens, and collects and analyzes feedback from citizens. We will specifically explain how this system can be applied to brick-and-mortar stores to collect customer feedback and personalize advertisements through sentiment analysis.

[0420] server

[0421] The server has the following main functions:

[0422] The contents of local assembly discussions are obtained via API.

[0423] The acquired voice data is passed to a voice recognition engine (e.g., Google Cloud Speech-to-Text) and converted into text data.

[0424] Summarize text data using a natural language processing engine (e.g., IBM Watson NLU).

[0425] Add an ad to the summary.

[0426] The summary and advertisement are delivered to the user's device via push notification.

[0427] Moderating customer feedback collected in physical stores.

[0428] Analyze the moderated feedback data with a sentiment engine (e.g., Microsoft Azure Text Analytics).

[0429] Personalize advertisements displayed on in-store digital signage based on sentiment analysis.

[0430] Terminal

[0431] The terminal has the following main features:

[0432] A device that allows users to view the summary and provide feedback. This includes tablets (e.g., iPad) and smartphones (e.g., iPhone).

[0433] An interface that displays a summary and allows users to post ratings or comments.

[0434] A feedback device installed in a physical store for customers to enter their feedback.

[0435] User

[0436] Specifically, the user performs the following operations:

[0437] Receive summary text via push notification on your smartphone or tablet.

[0438] Open the push notification, read the summary, click the "Like" button, or post a comment.

[0439] Enter feedback about your shopping experience and products using feedback devices in physical stores.

[0440] Specific examples of hardware and software used

[0441] Hardware

[0442] Tablet device (e.g. iPad)

[0443] Smartphone (e.g. iPhone)

[0444] Server (Cloud server, e.g. AWS EC2)

[0445] Digital Signage

[0446] software

[0447] Speech recognition engine (e.g. Google Cloud Speech-to-Text)

[0448] Natural language processing engine (e.g. IBM Watson NLU)

[0449] Sentiment engine (e.g. Microsoft Azure Text Analytics)

[0450] Database (e.g. MySQL)

[0451] Application (dedicated feedback collection app)

[0452] Specific examples of data processing procedures

[0453] For example, Customer A sends a comment to a physical store's feedback app saying, "The new cake was delicious!" The server receives the comment and converts it into text using a speech recognition engine. Next, a summary is generated using a natural language processing engine and an advertisement is added to the summary. This summary and advertisement are then sent to User A via a push notification. User A opens the notification, reads the summary, and clicks the "Like!" button to post feedback. The server receives this feedback and analyzes it using an emotion engine. The analysis results are transmitted to digital signage, and an advertisement corresponding to the emotion is displayed.

[0454] Prompt Sentence Examples

[0455] "Comment: "The new cake was absolutely delicious!"

[0456] Emotion: "Joy"

[0457] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0458] Step 1:

[0459] The server acquires the contents of the local assembly discussions. This acquisition is done by downloading audio data via an API. The acquired audio data is the input, and the audio data stored on the server is the output.

[0460] Step 2:

[0461] The server passes the acquired voice data to a voice recognition engine (e.g., Google Cloud Speech-to-Text) and converts it into text data. The voice recognition engine analyzes the voice data and generates corresponding text data. The input is voice data, and the output is text data.

[0462] Step 3:

[0463] The server passes the text data output from the speech recognition engine to a natural language processing engine (e.g., IBM Watson NLU), which summarizes the text data. The natural language processing engine extracts important points from the text data and generates a summary. The input is text data, and the output is a summary.

[0464] Step 4:

[0465] The server adds advertisements to the generated summary. The advertisements are stored in a database in advance, and the appropriate one is selected based on the content of the summary. The input is the summary and the advertisement database, and the output is the summary with the advertisement added.

[0466] Step 5:

[0467] The server delivers the summary with the ad added to the user's device. The delivery is done via push notification and sent to the application installed by the user. The input is the summary with the ad added, and the output is the push notification sent to the user's device.

[0468] Step 6:

[0469] A user receives a push notification on their device, taps the notification to open the application, and then views the summary and advertisement. Here, the input is the push notification, and the output is the user's viewing action.

[0470] Step 7:

[0471] Users rate or comment on summaries. Users click the "Like" button or enter their opinions in the comment field and submit them. The input is the user's rating or comment, and the output is the user's feedback data.

[0472] Step 8:

[0473] The terminal sends the user's rating and comment data to the server, which receives this data and stores it in a database. The input is the user's feedback data, and the output is the data stored in the database.

[0474] Step 9:

[0475] The server moderates newly submitted comment data, filtering comment content and eliminating inappropriate content. The input is user comment data, and the output is moderated comment data.

[0476] Step 10:

[0477] The server passes the moderated comment data to a sentiment engine (e.g., Microsoft Azure Text Analytics) for sentiment analysis. The sentiment engine analyzes the comment content and identifies the sentiment. The input is the moderated comment data, and the output is sentiment data.

[0478] Step 11:

[0479] The server personalizes the advertisements displayed on the digital signage in the store based on the emotional data identified by the emotion engine. This allows the optimal advertisement to be displayed according to the user's emotions. The input is emotional data, and the output is a personalized advertisement display.

[0480] Example prompt sentence:

[0481] text

[0482] "Comment: The new cake was delicious!

[0483] Emotion: Joy

[0484] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0485] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0486] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0487] [Second embodiment]

[0488] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0489] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0490] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0491] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0492] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0493] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0494] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0495] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0496] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0497] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0498] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0499] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0500] The present invention is a system that acquires the contents of discussions in local assemblies, summarizes them so that citizens can quickly understand them, and provides feedback on their opinions and evaluations. An example is shown below.

[0501] System Configuration

[0502] Server: Acquires parliamentary data, converts it into text using speech recognition, and summarizes it using natural language processing. It also adds advertisements to the summarized discussion content and delivers it to users.

[0503] Device: A device on which users can view summaries of discussions and enter ratings and comments. This includes smartphones, tablets, and PCs.

[0504] Users: Citizens who use the application to view summaries of local council discussions and rate and comment on them.

[0505] Processing content

[0506] 1. Acquiring discussion content

[0507] The server retrieves audio and video data through the API after each local council session, either in real time or periodically.

[0508] 2. Text conversion using voice recognition

[0509] The server passes the acquired voice data to a voice recognition engine and converts it into text data, which is used in the next step.

[0510] 3. Natural Language Processing Summarization

[0511] The server passes the converted data to a natural language processing engine, extracts key points, and generates a summary based on the extracted points. This summary is of an appropriate length to convey the content concisely.

[0512] 4. Adding Ads

[0513] The server adds advertising information provided by local businesses to the abstracts, which are appropriately inserted into each abstract and positioned to attract the user's attention.

[0514] 5. Summary distribution

[0515] The server delivers the generated summary and advertisement to registered users via push notification, which includes the title of the summary and a brief introduction.

[0516] 6. Accepting ratings and comments

[0517] Users tap the push notification on their device, open the app, and view the summary. They can then rate or comment using the "Like" button or comment field below the summary.

[0518] The terminal transmits the ratings and comments entered by the user to the server.

[0519] 7. Data Retention

[0520] The server stores the submitted ratings and comments in a database, which is later used for analysis and engagement evaluation.

[0521] 8. Moderation and Ad Management

[0522] The server moderates each new comment as it is posted, filtering out inappropriate content, and also manages advertising data and generates reports for advertisers.

[0523] Specific examples

[0524] For example, if a new park development plan is discussed at a local assembly on July 5th, the server will acquire the audio data after the assembly ends. Next, a speech recognition engine will convert the audio into text, and a natural language processing engine will extract key points and generate a summary. After that, the opinions and ratings of User A, who posted a comment saying, "I'm happy that a new park will be built!", will be moderated and saved in the database. An advertisement for a local cafe will be added to this summary and sent to User B via push notification.

[0525] This system will enable discussions in local assemblies to be communicated to citizens quickly and efficiently, creating an environment in which citizen feedback can be reflected in local administration.

[0526] The processing flow will be explained below.

[0527] Step 1:

[0528] After the local assembly session ends, the server uses the API to retrieve the audio data, which includes the contents of the assembly discussion and is later converted into text.

[0529] Step 2:

[0530] The server passes the acquired voice data to a voice recognition engine, which converts it into text data in real time or in batches. The voice recognition engine analyzes the content of the voice and outputs it as text information.

[0531] Step 3:

[0532] The server passes the text data output by the speech recognition engine to the natural language processing engine, which extracts important points from the text data and generates a summary. This summary is a concise summary of the content.

[0533] Step 4:

[0534] The server adds advertising information provided by local businesses to the generated abstract, which is then formatted to be displayed with the abstract and delivered to the user.

[0535] Step 5:

[0536] The server compiles the summary and advertisement and sends a push notification to registered users, which includes the title of the summary and a brief intro designed to grab the user's attention.

[0537] Step 6:

[0538] A user receives a push notification on their device and taps the notification to open the application, which displays a summary and an advertisement on the screen.

[0539] Step 7:

[0540] The user views the summary and taps the "Like" button. If necessary, the user can enter their opinion or feedback in the comment field and tap the submit button. This inputs the user's feedback.

[0541] Step 8:

[0542] The device sends the user's rating (likes) and comment data to the server, which is then recorded as user feedback.

[0543] Step 9:

[0544] The server stores the submitted rating and comment data in a database, which is later used for analysis and feedback evaluation.

[0545] Step 10:

[0546] The server moderates newly submitted comments to check for inappropriate content. Inappropriate comments are filtered out, and only appropriate comments are saved.

[0547] Step 11:

[0548] The server manages the advertising data and collects user interaction data, which is used to evaluate the effectiveness of the advertisements and generate reports as feedback to the advertisers.

[0549] Through these steps, this system provides a mechanism for quickly and efficiently communicating the contents of local assembly discussions to citizens and reflecting citizen feedback in the administration.

[0550] Example 1

[0551] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0552] While the content of local assembly discussions is important information for citizens, it is difficult to quickly and concisely grasp this information. Furthermore, there are limited ways for citizens to provide feedback on the content of discussions, resulting in a lack of two-way communication with local governments. Furthermore, local assembly information is not properly integrated with external advertising, limiting opportunities for local advertisers. To solve these issues, an appropriate system is needed.

[0553] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0554] In this invention, the server includes means for acquiring the contents of local assembly discussions, means for converting the acquired discussion contents into text using speech recognition, means for summarizing the converted text using natural language processing, means for adding advertisements to the summaries, means for distributing the summaries to users, means for users to post ratings or comments on the summaries, means for saving the ratings and comments in a database, means for moderating the content of the ratings and comments, and means for appropriately placing advertisements together with the summaries. This allows for quick and concise provision of local assembly information to citizens, smooth collection of feedback from citizens, and two-way communication. It also allows for effective provision of information to local advertisers.

[0555] A "local assembly" refers to the assembly of a local government, an institution that discusses and makes decisions on local public policies, budgets, etc.

[0556] "Discussion content" refers to all statements and debates made by members of parliament and representatives in local assemblies, as well as all related information.

[0557] A "means" refers to a device, technique, method, or process used to accomplish a particular purpose.

[0558] "Speech recognition" refers to the technology of converting speech into text data.

[0559] "Text conversion" refers to the conversion of audio information extracted from audio data or video data into readable text information.

[0560] "Natural language processing" refers to the technology that enables computers to understand, interpret, and generate human language.

[0561] A "summary" is a short, concise summary of the main content.

[0562] "Advertising" refers to the means of providing information to publicize a product, service, or business.

[0563] "Distribution" refers to the act of transmitting information so that specific users receive the information.

[0564] "Users" refer to people who access the system, consume the information provided, and provide feedback such as ratings and comments.

[0565] "Feedback" refers to reactions to the system and its offerings, such as ratings and comments provided by users.

[0566] "Evaluation" refers to the act of a user expressing their opinion or impression about the provided summary or content.

[0567] "Comment" refers to the act of a user inputting detailed opinions or impressions about the provided summary or content in words.

[0568] A "database" refers to an electronic system for systematically storing and managing information.

[0569] "Moderation" refers to the process of reviewing comments and feedback provided by users and determining whether they are appropriate.

[0570] "Advertiser" refers to an individual or organization that provides advertising and seeks to profit from such advertising.

[0571] "Interaction data" refers to information about user actions and reactions, which is a record of the interaction between a system and a user.

[0572] The present invention is a system that quickly and easily provides citizens with the content of discussions in local assemblies and collects their feedback. This system functions in cooperation with three parties: a server, a terminal, and a user. Specific embodiments are described below.

[0573] Server Functions and Processing

[0574] 1. Acquiring discussion content

[0575] The server retrieves audio and video data via an API each time a local council session ends. This data is collected in real time or periodically. The API can be publicly available from the local council or from a dedicated data provider.

[0576] 2. Text conversion using voice recognition

[0577] The server passes the acquired voice data to a voice recognition engine (e.g., a voice recognition cloud service) and converts it into text data, which is used for subsequent processing.

[0578] 3. Natural Language Processing Summarization

[0579] The server passes the converted discussion content to a natural language processing engine (e.g., a natural language processing library), extracts key points, and generates a summary. The summary is adjusted to an optimal length to convey the discussion content concisely.

[0580] 4. Adding Ads

[0581] The server adds advertisements from local vendors to the generated summary, and the advertisements are appropriately placed to attract the user's interest.

[0582] 5. Summary distribution

[0583] The server delivers the generated summary and advertisement to registered users via push notification (e.g., push notification cloud service).

[0584] 6. Accepting ratings and comments

[0585] The server accepts user ratings and comments on the summaries and stores them in a database, which is later used for analysis and engagement evaluation.

[0586] 7. Moderation and Ad Management

[0587] The server moderates content as new comments are posted, filtering out inappropriate content, and also manages advertising data and generates reports for advertisers.

[0588] Terminal functions and processing

[0589] 1. View the summary and enter your rating and comments

[0590] The device receives the push notification, and when the user taps the notification, the application opens and displays the summary. The user can then rate or comment on the summary using the "Like" button or comment field. The entered rating or comment is then sent to the server.

[0591] User operations

[0592] 1. Review and feedback on the summary

[0593] Users receive a push notification from their device and can review the summary. They can rate the summary and add comments, providing feedback on the local council discussion.

[0594] Specific examples

[0595] For example, if a new park development plan is discussed at a local assembly on July 5th, the server will obtain the audio data via an API after the assembly ends. Next, a speech recognition engine will convert the audio into text, and a natural language processing engine will extract key points and generate a summary. After that, User A's comment, "I'm happy that a new park will be built!", will be moderated and saved in the database. This summary, along with an advertisement for a local cafe, will be sent to User B via a push notification.

[0596] Prompt Sentence Examples

[0597] "A new park development plan was discussed at the local council meeting on July 5th. Please summarize the key points of the discussion."

[0598] This system will enable important information from local councils to be communicated to citizens quickly and efficiently, creating an environment in which citizen feedback can be reflected in local administration.

[0599] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0600] Step 1:

[0601] The server acquires the contents of discussions in local assemblies.

[0602] Specific operation: The server uses the API published by the local council to obtain audio and video data in real time or periodically.

[0603] Input: Local council audio and video data.

[0604] Output: Audio data file.

[0605] Step 2:

[0606] The server passes the acquired voice data to a voice recognition engine and converts it into text data.

[0607] Specific operation: The server sends the voice data to the voice recognition cloud service and obtains the corresponding text data.

[0608] Input: Audio data file.

[0609] Output: Text data.

[0610] Step 3:

[0611] The server passes the converted discussion content to a natural language processing engine, extracts important points, and generates a summary.

[0612] How it works: The server inputs text data into a natural language processing library and generates a summary of appropriate length. The summary is constructed by scoring important keywords and sentences and extracting the top content.

[0613] Input: Text data.

[0614] Output: Summary statement.

[0615] Step 4:

[0616] The server adds advertisements provided by local businesses to the summary.

[0617] What it does: The server extracts relevant ads from an ad database and inserts them into specific parts of the summary. Relevance is determined by taking into account keywords and themes that interest the user.

[0618] Input: Abstract, ad data.

[0619] Output: Summary text with ad added.

[0620] Step 5:

[0621] The server delivers the generated summary and advertisement to the user via push notification.

[0622] Specific operation: The server retrieves the device tokens of all users from the user database, and uses the push notification cloud service to generate and deliver notifications containing summary text and advertisements.

[0623] Input: Ad summary, user device token.

[0624] Output: Push notification.

[0625] Step 6:

[0626] The user receives a push notification on their device, opens the app, and views the summary.

[0627] What happens: The user taps the notification and sees the summary in the app, which includes an ad.

[0628] Enter: push notification.

[0629] Output: Summary and advertisement.

[0630] Step 7:

[0631] The user inputs an evaluation and comments on the summary.

[0632] What it does: Users use the "Like" button and comment field below the summary to enter their own rating or comment.

[0633] Input: User rating and comments.

[0634] Output: The input rating and comment data.

[0635] Step 8:

[0636] The terminal transmits the ratings and comments entered by the user to the server.

[0637] Specific operation: The entered ratings and comments are sent from the app to the server, where the data received is stored in a database.

[0638] Input: Rating and comment data.

[0639] Output: Send to server.

[0640] Step 9:

[0641] The server stores the submitted ratings and comments in a database.

[0642] Specific operation: The server stores the received rating and comment data in the corresponding location in the database.

[0643] Input: Rating and comment data.

[0644] Output: Database update.

[0645] Step 10:

[0646] The server moderates each new comment as it is posted, filtering out inappropriate content.

[0647] What it does: Moderation algorithms are applied to automatically identify and filter inappropriate keywords and content, with manual review where necessary.

[0648] Input: Rating and comment data.

[0649] Output: Moderated comment data.

[0650] Step 11:

[0651] The server manages the advertising data and generates reports for the advertisers.

[0652] Specific operation: The server collects and analyzes the logs of ad display, and automatically generates and sends reports to advertisers.

[0653] Input: Ad display log.

[0654] Output: Report to advertiser.

[0655] (Application example 1)

[0656] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0657] The content of discussions in local assemblies is difficult for many citizens to understand, and it is difficult to follow in real time. There is also a lack of mechanisms for citizens to quickly have their opinions reflected in local government. Furthermore, there are limited means to increase the effectiveness of advertising for local businesses. It is necessary to resolve these issues, increase citizen engagement, and promote transparency in local government and the sharing of discussion content.

[0658] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0659] In this invention, the server includes means for acquiring the contents of discussions in local assemblies, means for converting the acquired contents into text using speech recognition, means for summarizing the converted text using natural language processing, means for adding advertisements to the summaries, means for distributing the summaries to users, means for users to post ratings or comments on the summaries, means for storing the ratings and comments in a database, means for displaying the contents of discussions in local assemblies in a virtual space, and means for users to provide feedback in the virtual space. This allows citizens to quickly and efficiently understand the contents of discussions in local assemblies and provide feedback of their opinions in real time. It also enhances the advertising effectiveness of local businesses.

[0660] A "local assembly" is an institution in a local government that deliberates and decides on bills.

[0661] "Discussion content" refers to the various issues, proposals, and opinions that are deliberated in local assemblies.

[0662] "Speech recognition" is a technology that converts speech into text.

[0663] "Text conversion" refers to the process of converting audio or video data into text information.

[0664] "Natural language processing" is a technology that allows computers to understand, analyze, and generate human language.

[0665] "Summarization" refers to extracting important parts of text data and summarizing them concisely.

[0666] "Advertising" is promotional information intended to widely publicize a particular product or service.

[0667] "Distribution" refers to the transmission of information or data to multiple users.

[0668] "User" refers to citizens or individual users of the system.

[0669] "Evaluation" refers to a user expressing an opinion about a particular piece of information or service.

[0670] "Comments" refers to opinions or feedback provided by users.

[0671] A "database" is a collection of data that allows information to be managed efficiently and easily searched and updated.

[0672] "Virtual space" refers to a virtual three-dimensional space created using computer technology.

[0673] "Feedback" refers to reactions such as ratings and comments from users.

[0674] The system for implementing this invention consists of three main elements: a server, a terminal, and a user.

[0675] Server Features

[0676] The server efficiently retrieves the contents of local assembly discussions, converts them into text using speech recognition, then generates summaries using a natural language processing engine, adds advertisements to the summaries, and delivers them to users in the virtual space.

[0677] Specifically, the process is as follows: The server passes the voice data obtained from the local assembly to a voice recognition engine (for example, Google's voice recognition API) and converts it into text data. This text data is then used by OpenAI's natural language processing engine (such as GPT-3) to extract key points and generate a summary. Advertising information from local businesses is then appropriately inserted into the generated summary.

[0678] Device Features

[0679] The device functions as a device for users to view the summary and post ratings and comments. It can be a smartphone, tablet, or PC. When a user taps the push notification, an application opens and the summary is displayed. The user can then use the "Like!" button or comment field at the bottom of the summary to enter a rating or comment.

[0680] Virtual space functions

[0681] The virtual space provides a place where users can experience the contents of local assembly discussions in a virtual three-dimensional space. Specifically, there is a virtual assembly hall where the discussion contents are displayed, and users gathered there can view the discussion contents in real time and provide feedback. This can be achieved, for example, through a VR headset or a smartphone app.

[0682] example

[0683] For example, if a local assembly is debating a new city park plan, the server captures the audio data after the assembly. The audio data is converted into text using Google's speech recognition API, and a summary is generated using GPT-3. The summary is then supplemented with an advertisement for a local cafe and delivered to the user in the virtual space.

[0684] Users receive a push notification through their smartphone app and open it to view the summary. The user's comment, "I'm happy that a new park will be built!", is sent from the device to a server and stored in a database. This data will later be analyzed by the local government and used as part of their feedback.

[0685] Prompt Sentence Examples

[0686] Here is an example prompt for generating a summary:

[0687] "Summarize the following text: Today, the local assembly discussed the design of a new park. Council members exchanged opinions on the park's location, design, and budget, and debated points to improve the quality of life for residents."

[0688] This system allows citizens to quickly and efficiently understand the content of discussions in local assemblies and provide feedback on their opinions in real time. It can also increase the effectiveness of advertising for local businesses.

[0689] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0690] Step 1:

[0691] The server acquires audio data of the local assembly discussions. After each assembly, audio and video data are collected in real time or periodically through the API. In this step, the audio data is the input, and the output is the audio file stored on the server.

[0692] Step 2:

[0693] The server passes the acquired voice data to a voice recognition engine and converts it into text data. Specifically, it uses Google's voice recognition API to analyze the voice data and convert the discussion content into text. In this step, voice data is the input and text data is the output.

[0694] Step 3:

[0695] The server passes the converted data to a natural language processing engine, extracts key points, and generates a summary. Specifically, OpenAI's GPT-3 is used to generate a summary of the text data. In this step, the text data is the input and the summary is the output.

[0696] Step 4:

[0697] The server adds advertising information provided by local businesses to the summary. The advertising information is inserted appropriately in a position that is likely to attract the user's interest. In this step, the summary and advertising data are the input, and the summary with the added advertisement is the output.

[0698] Step 5:

[0699] The server delivers the generated summary and advertisement to registered users' devices via push notification. The push notification includes the title of the summary and a brief introduction. In this step, the summary with advertisement is the input, and the push notification to the user's device is the output.

[0700] Step 6:

[0701] The user taps the push notification on their device, opens the application, and reads the summary. The device displays the summary and provides an interface where the user can rate or comment using the "Like" button and comment field. In this step, receiving the push notification is the input, and the display of the summary and the interface for comment input are the outputs.

[0702] Step 7:

[0703] The terminal sends the ratings and comments entered by the user to the server. The ratings and comments entered by the user on the interface are generated as data and then sent. In this step, the user's ratings and comments are the input, and the data sent to the server is the output.

[0704] Step 8:

[0705] The server stores the submitted ratings and comments in a database. The stored data is later used for analysis and engagement evaluation. In this step, the rating and comment data are the input and the storage in the database is the output.

[0706] Step 9:

[0707] The server moderates each new comment as it is posted, filtering out inappropriate content, and also manages advertising data and generates reports for advertisers. In this step, user comments and advertising data are the input, and filtered comments and reports for advertisers are the output.

[0708] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0709] This invention combines an emotion engine with a system that acquires and summarizes the contents of local assembly discussions and distributes them to citizens, and collects and analyzes feedback from citizens. An example is shown below.

[0710] System Configuration

[0711] Server: This server has the function of acquiring parliamentary data, converting it into text using speech recognition, and summarizing it using natural language processing. It also adds advertisements to the summaries and delivers them to users. It also moderates user comments and analyzes their sentiment using an emotion engine.

[0712] Device: A device on which users can view summaries of discussions and enter ratings and comments. This includes smartphones, tablets, and PCs.

[0713] Users: Citizens who use the application to view summaries of local council discussions and provide ratings and comments.

[0714] Specific processing of the program

[0715] 1. Acquiring discussion content

[0716] After the local assembly session ends, the server uses an API to retrieve the audio data, which includes the contents of the assembly discussion and is then converted into text in a later process.

[0717] 2. Text conversion using voice recognition

[0718] The server passes the acquired voice data to a voice recognition engine, which converts it into text data in real time or in batches. The voice recognition engine analyzes the content of the voice and outputs it as text information.

[0719] 3. Natural Language Processing Summarization

[0720] The server passes the text data output by the speech recognition engine to the natural language processing engine, which extracts important points from the text data and generates a summary. This summary is a concise summary of the content.

[0721] 4. Adding Ads

[0722] The server adds advertising information provided by local businesses to the generated abstract, which is then formatted to be displayed with the abstract and delivered to the user.

[0723] 5. Summary distribution

[0724] The server compiles the summary and advertisement and sends a push notification to registered users, which includes the title of the summary and a brief intro designed to grab the user's attention.

[0725] 6. Accepting ratings and comments

[0726] A user receives a push notification on their device and taps the notification to open the application, which displays a summary and an advertisement on the screen.

[0727] The user views the summary and taps the "Like" button. If necessary, the user can enter their opinion or feedback in the comment field and tap the submit button. This inputs the user's feedback.

[0728] 7. Data transmission and storage

[0729] The device sends the user's rating (likes) and comment data to the server, which is then recorded as user feedback.

[0730] The server stores the submitted rating and comment data in a database, which is later used for analysis and feedback evaluation.

[0731] 8. Comment Moderation

[0732] The server moderates newly submitted comments to check for inappropriate content. Inappropriate comments are filtered out, and only appropriate comments are saved.

[0733] 9. Sentiment Analysis

[0734] The server passes the user's comment data to the emotion engine for emotion analysis. The emotion engine analyzes the content of the comment and identifies the emotion contained therein (e.g., joy, anger, sadness, etc.).

[0735] The server evaluates citizens' emotional reactions to the content of the discussion based on the emotional data analyzed by the emotion engine.

[0736] 10. Adjusting Ad Display

[0737] The server then adjusts the content of the advertisements based on the emotion data identified by the emotion engine. For example, if a user has a positive opinion, the server displays an advertisement that corresponds to that opinion, thereby personalizing the advertisements.

[0738] Specific examples

[0739] For example, if a new park development plan is discussed at a local assembly on July 5th, the server will collect the audio data after the assembly ends. The audio will then be converted into text using a speech recognition engine, and a natural language processing engine will extract key points to generate a summary. An advertisement for a local cafe will then be added to the summary, and the summary and advertisement will be sent to User A via push notification.

[0740] User A taps the notification, reads the summary in the application, and taps the "Like" button. He also posts a comment saying, "I'm so happy that a new park is opening!" The device sends the comment data to the server. The server stores the comment data in a database and simultaneously passes the comment to the emotion engine for analysis. The emotion engine identifies the emotion "joy," and the server uses this emotion data to adjust the display of more relevant ads to the same User A the next time.

[0741] Through this series of steps, the system not only quickly and efficiently conveys the contents of discussions in local assemblies to citizens and reflects citizen feedback in the administration, but also grasps their emotional reactions and further improves the accuracy of advertising.

[0742] The processing flow will be explained below.

[0743] Step 1:

[0744] The server uses an API to retrieve audio data after the local assembly session has ended. This audio data contains the contents of the assembly discussions and is stored for later processing to convert it into text.

[0745] Step 2:

[0746] The server passes the acquired voice data to a voice recognition engine, which converts it into text data in real time or in batches. The voice recognition engine analyzes the content of the voice and outputs the corresponding text information.

[0747] Step 3:

[0748] The server passes the text data output by the speech recognition engine to a natural language processing engine, which extracts important points from the text data and generates a concise summary of the discussion.

[0749] Step 4:

[0750] The server adds advertising information provided by local businesses to the generated abstract, which is then formatted to be displayed in the appropriate location along with the abstract.

[0751] Step 5:

[0752] The server compiles the summary and advertisement and sends a push notification to registered users, which includes the title of the summary and a brief intro designed to grab the user's attention.

[0753] Step 6:

[0754] A user receives a push notification on their device and taps the notification to open the application, which displays a summary and an advertisement on their device screen.

[0755] Step 7:

[0756] The user views the summary and taps the "Like" button. If necessary, the user can enter their opinion or feedback in the comment field and tap the submit button. This inputs the user's feedback.

[0757] Step 8:

[0758] The device sends the user's ratings (likes) and comment data to the server, which is then recorded as user feedback.

[0759] Step 9:

[0760] The server stores the submitted rating and comment data in a database, which is later used for analysis and engagement measurement.

[0761] Step 10:

[0762] The server moderates newly submitted comments to check for inappropriate content, filtering out inappropriate comments and keeping only appropriate comments.

[0763] Step 11:

[0764] The server passes the user's comment data to the emotion engine, which analyzes the content of the comment and identifies the emotions contained therein (e.g., joy, anger, sadness, etc.).

[0765] Step 12:

[0766] The server uses the emotion data identified by the emotion engine to assess citizens' emotional reactions to the content of the discussion, thereby understanding how citizens feel.

[0767] Step 13:

[0768] The server adjusts the content of the advertisements based on the emotional data. For example, if the user has a positive opinion, the server will display an advertisement that corresponds to that opinion, thereby personalizing the advertisements.

[0769] To give a concrete example, if a new park development plan is discussed at a local assembly on July 5th, the server will acquire the voice data after the assembly ends and convert it into text using a speech recognition engine. The server will then generate a summary using a natural language processing engine, add an advertisement for a local cafe to the summary, and send a push notification to User A. User A taps the notification to view the summary, then enters a comment such as "I'm so happy that a new park is being built!" and sends it. The server will pass this comment to the emotion engine, identify the emotion of joy, and adjust and deliver relevant advertisements to User A next time. By obtaining feedback using the emotion engine in this way, it is possible to understand citizens' reactions and deliver appropriate advertisements, thereby improving the effectiveness of the entire system.

[0770] Example 2

[0771] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0772] The content of discussions in local assemblies is important public information that should be shared with citizens. However, there is a need for a method to efficiently collect, summarize, and quickly provide this content to citizens. There is also a need for a method to improve government services by obtaining feedback from citizens and analyzing their emotional reactions. Current systems have difficulty meeting these requirements, and there are challenges, particularly with the additional time and effort required for sentiment analysis and advertising personalization.

[0773] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0774] In this invention, the server includes: means for acquiring the contents of discussions in local assemblies; means for converting the acquired contents of discussions into text using speech recognition; means for summarizing the converted text using natural language processing; means for adding advertisements to the summaries; means for distributing the summaries to users; means for users to post ratings or comments on the summaries; means for saving the ratings and comments in a database; means including a sentiment analysis engine for analyzing the saved rating and comment data and identifying sentiment; and means for adjusting the content of advertisements based on the results of the sentiment analysis. This makes it possible to quickly and efficiently provide the contents of discussions in local assemblies to citizens, and to deeply understand citizen feedback and reflect it in administrative services.

[0775] A local assembly is an administrative body in a specific region where representatives of local residents gather to deliberate and decide on local laws, budgets, policies, etc.

[0776] "Discussion content" refers to the totality of information and opinions that are taken up as topics in local assemblies and debated among participants.

[0777] A "means" is a method, process, or device used to achieve a particular purpose.

[0778] "Speech recognition" is a technology that analyzes voice data and converts it into text data.

[0779] "Text conversion" is the process of analyzing non-textual information such as audio data and image data and converting it into textual information.

[0780] "Natural language processing" is a field of technology in which computers understand, analyze, and generate human language.

[0781] "Summarization" refers to the process of extracting important information from long text data and summarizing it in a short, concise form, or the result of that process.

[0782] An "advertisement" is a message intended to inform the general public about a particular product, service, or information.

[0783] "Users" refers to local residents and the general public who use this system.

[0784] "Distribution" is the process of getting information or data to a specific recipient.

[0785] "Evaluation" is the act of a user giving positive or negative feedback on the content of a discussion or a summary.

[0786] A "comment" is an act in which a user writes down their own opinion or impression on the content of a discussion or a summary.

[0787] A "database" is a digital system for efficiently storing, retrieving, updating, and managing large amounts of data.

[0788] An "emotion analysis engine" is a technology that analyzes text data and identifies the emotions contained within it (e.g., joy, anger, sadness, etc.).

[0789] "Moderation" is the process of reviewing posted comments to check for inappropriate content.

[0790] This invention is a system that aims to improve administrative services by providing the contents of local assembly discussions to citizens quickly and efficiently and collecting and analyzing feedback from citizens. The components of the system and their specific operation are described below.

[0791] System Configuration

[0792] 1. Server

[0793] The server will play a central role in collecting and processing data from local council discussions. The server's main functions are as follows:

[0794] Acquiring discussion content

[0795] The server uses an API to retrieve audio data after a local assembly session ends. This audio data contains the contents of the assembly discussion and is later converted to text. Specifically, the server hits the API endpoint, sends parameters such as the date and session ID, and downloads the audio data. For example, the audio data is saved in MP3 format.

[0796] Voice Recognition

[0797] The server passes the acquired voice data to a speech recognition engine (e.g., Google Cloud Speech-to-Text) and converts the voice into text data. The server then puts the voice data into a queue for batch processing, and the speech recognition engine converts it into text. This text data is saved in the form of meeting minutes.

[0798] Natural Language Processing

[0799] The server passes the text data output from the speech recognition engine to a natural language processing engine (e.g., OpenAI GPT-4), extracts important points, and generates a summary. The server then prompts the engine with a prompt such as "Summarize the following text," and stores the generated summary within the system.

[0800] Adding Ads

[0801] The server adds advertising information provided by local businesses to the generated summary, searches for relevant advertisements (e.g., coupon information for local stores) in the advertisement database, and adds the advertisements to the end of the summary. The formatted summary and advertisements are saved in HTML or JSON format.

[0802] Summary distribution

[0803] The server uses a push notification service (e.g., Firebase Cloud Messaging) to deliver the summary and advertisement to the user. It obtains the registered user's device ID and sends a notification message containing the summary, advertisement title, and intro. The notification is displayed on the user's device.

[0804] Accepting ratings and comments

[0805] The server stores the ratings (likes) and comments that users make on summaries in a database, allowing us to use user feedback for future analysis.

[0806] Comment moderation

[0807] The server moderates newly submitted comments, using a text filtering library (e.g., CleanSpeak) to check for inappropriate content, rejecting comments that are deemed inappropriate, and storing only appropriate comments.

[0808] Sentiment analysis

[0809] The server passes the user's comment data to a sentiment analysis engine (e.g., IBM Watson Natural Language Understanding) to identify emotions from the content of the comment. The sentiment analysis engine analyzes emotions such as joy, anger, and sadness, and stores the analysis results in a database.

[0810] Ad display adjustment

[0811] The server adjusts the content of advertisements based on the analysis results of the emotion engine. For example, if the user expresses the emotion of "joy," it will display advertisements related to that emotion (such as ticket information for a new park opening event) as the next push notification or in-app advertisement.

[0812] 2. Terminal

[0813] The terminal is a device that allows users to view summaries of discussions and enter ratings and comments. This includes smartphones, tablets, and PCs.

[0814] 3. Users

[0815] Users are citizens who use the application to view summaries of local assembly discussions and provide ratings and comments.

[0816] Specific examples

[0817] For example, if a new park development plan is discussed at a local council meeting on July 5th, the specific process would be as follows:

[0818] 1. The server uses the API to obtain the parliamentary audio data.

[0819] 2. The server passes the acquired voice data to Google Cloud Speech-to-Text and converts it into text data.

[0820] 3. The server passes the text data to OpenAI GPT-4, which generates a summary with the prompt "Summarize the following text."

[0821] 4. The server adds advertisements for local stores to the generated summary.

[0822] 5. The server uses Firebase Cloud Messaging to send a push notification containing the summary and advertisement to User A.

[0823] 6. User A taps the notification, reads the summary in the app, taps "Like," and posts a comment saying, "I'm so happy that the new park is opening!"

[0824] 7. The device sends the comment data to the server, which stores it in a database.

[0825] 8. The server moderates comments and only appropriate comments are left.

[0826] 9. The server passes the comment data to the emotion engine to identify the emotion "joy."

[0827] 10. The server configures the next push notification to display an ad related to User A.

[0828] Prompt Sentence Examples

[0829] Here are some examples of prompts for generative AI models:

[0830] "A new park development plan was discussed at the local council meeting on July 5th. Please summarize the content of that discussion."

[0831] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0832] Step 1:

[0833] Acquiring discussion content

[0834] After the local council session ends, the server hits the API endpoint and sends parameters such as the date and session ID to retrieve the audio data. The input is the API endpoint and required parameters. The output is audio data in MP3 format. Specifically, an HTTP request is sent and the audio data is received as a response.

[0835] Step 2:

[0836] Speech recognition to text

[0837] The server places the acquired audio data into a batch processing queue and passes it to a speech recognition engine (e.g., Google Cloud Speech-to-Text). The input is audio data in MP3 format. The output is text data in minutes format. Specifically, the URL of the audio file is passed to the speech recognition engine, and the text data is received as the processing result. An API is called on the server and the results are saved.

[0838] Step 3:

[0839] Natural language processing summarization

[0840] The server passes the text data output from the speech recognition engine to a natural language processing engine (e.g., OpenAI GPT-4), and issues instructions using a prompt such as "Summarize the following text." The input is the text data of the minutes and the prompt. The output is a summary. Specifically, the text is sent to the natural language processing engine along with the prompt, and the generated summary is received.

[0841] Step 4:

[0842] Adding Ads

[0843] The server adds advertising information provided by local businesses to the generated summary. The input is the summary and an advertising database. The output is the summary with the advertising added. Specifically, it searches for relevant advertisements in the advertising database and adds the advertisements to the end of the summary. The formatted summary and advertisements are saved in HTML or JSON format.

[0844] Step 5:

[0845] Summary distribution

[0846] The server uses a push notification service (e.g., Firebase Cloud Messaging) to deliver the summary and advertisement to the user. The input is the summary, advertisement, and the registered user's device ID. The output is the result of sending the push notification. Specifically, a notification message containing the summary, advertisement title, and intro is created and sent via Firebase Cloud Messaging. The notification is then displayed on the user's device.

[0847] Step 6:

[0848] Accepting ratings and comments

[0849] The user taps the push notification, opens the application, and views the summary and advertisement. The input is the push notification and user actions (tap, comment). The output is rating data (likes) and comment data. Specifically, the user taps the "Like" button, or enters their opinion or feedback in the comment section and taps the send button. The feedback data is sent from the device to the server.

[0850] Step 7:

[0851] Data transmission and storage

[0852] The device sends user ratings (likes) and comment data to the server. The input is rating data (likes) and comment data. The output is the database update results. Specifically, the device receives ratings and comments, stores them in the appropriate fields, and saves them in the database. The saved data is used for later analysis.

[0853] Step 8:

[0854] Comment moderation

[0855] The server moderates newly submitted comments. The input is the comment data. The output is the result of only saving appropriate comments. Specifically, a text filtering library (e.g. CleanSpeak) is used to check for inappropriate content. Comments that are deemed inappropriate are rejected, and only appropriate comments are saved in the database.

[0856] Step 9:

[0857] Sentiment analysis

[0858] The server passes user comment data to a sentiment analysis engine (e.g., IBM Watson Natural Language Understanding) to identify emotions from the comments. The input is comment data. The output is the sentiment analysis results. Specifically, the comment data is sent to the sentiment analysis engine, which identifies emotions such as joy, anger, and sadness, receives the results, and stores them in a database.

[0859] Step 10:

[0860] Ad display adjustment

[0861] The server adjusts the ad content based on the emotion engine's analysis results. The input is the emotion analysis results. The output is the adjusted ad content. Specifically, based on the analysis results, if the user expresses the emotion of "joy," for example, the server sets the next push notification or in-app ad to display an ad related to that emotion (e.g., ticket information for a new park opening event).

[0862] (Application example 2)

[0863] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0864] While systems already exist for effectively communicating the content of local assembly discussions to citizens and efficiently collecting and analyzing their feedback, there is no established system for collecting customer feedback in brick-and-mortar stores and personalizing advertisements based on that feedback. As a result, brick-and-mortar stores are unable to quickly and appropriately reflect the diverse emotions and opinions of customers, and are unable to maximize the effectiveness of advertising. There is also a lack of automated systems for moderating inappropriate comments and personalizing advertisements based on sentiment analysis.

[0865] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0866] In this invention, the server includes means for acquiring the content of discussions in local assemblies, means for converting the acquired content of discussions into text using speech recognition, means for summarizing the converted text of discussions using natural language processing, means for adding advertisements to the summaries, means for distributing the summaries to users, means for users to post ratings or comments on the summaries, means for saving the ratings and comments in a database, means for collecting customer feedback from physical stores, means for moderating the collected feedback, means for sentiment analysis of the feedback data, and means for personalizing advertisements based on the sentiment analysis. This makes it possible to quickly and appropriately reflect customer feedback even in physical stores and personalize advertisements based on the results.

[0867] "Local assembly discussions" are records of the content of discussions held by assembly members and other stakeholders in local government assemblies.

[0868] "Speech recognition" is a technology that analyzes voice data and converts it into corresponding text data.

[0869] "Natural language processing" is a technology that analyzes text data and automatically understands its meaning and structure.

[0870] A "summary" is a short summary of the main points extracted from a long text.

[0871] "Adding advertisements" refers to the technique or act of inserting advertisements into abstracts or other content.

[0872] "Delivering to the user" refers to the technique or act of sending the created summary and related information to the user's terminal.

[0873] Posting a "rating or comment" means that a user inputs and sends their opinion or feedback on the content.

[0874] "Storing in a database" means recording the collected evaluations and comments in digital form so that they can be referenced and analyzed as needed.

[0875] "Brick-and-mortar customer feedback" refers to the opinions and evaluations that customers give about their experiences and products in brick-and-mortar stores.

[0876] "Moderation" refers to the process of examining the content of comments posted by users and removing or modifying inappropriate comments.

[0877] "Sentiment analysis" is a technology that identifies and analyzes emotional elements (e.g., joy, anger, sadness, etc.) from text data.

[0878] "Personalizing ads" means customizing advertising content based on the characteristics and behavior of individual users.

[0879] This invention is a system that acquires the contents of discussions in local assemblies, efficiently conveys them to citizens, and collects and analyzes feedback from citizens. We will specifically explain how this system can be applied to brick-and-mortar stores to collect customer feedback and personalize advertisements through sentiment analysis.

[0880] server

[0881] The server has the following main functions:

[0882] The contents of local assembly discussions are obtained via API.

[0883] The acquired voice data is passed to a voice recognition engine (e.g., Google Cloud Speech-to-Text) and converted into text data.

[0884] Summarize text data using a natural language processing engine (e.g., IBM Watson NLU).

[0885] Add an ad to the summary.

[0886] The summary and advertisement are delivered to the user's device via push notification.

[0887] Moderating customer feedback collected in physical stores.

[0888] Analyze the moderated feedback data with a sentiment engine (e.g., Microsoft Azure Text Analytics).

[0889] Personalize advertisements displayed on in-store digital signage based on sentiment analysis.

[0890] Terminal

[0891] The terminal has the following main features:

[0892] A device that allows users to view the summary and provide feedback. This includes tablets (e.g., iPad) and smartphones (e.g., iPhone).

[0893] An interface that displays a summary and allows users to post ratings or comments.

[0894] A feedback device installed in a physical store for customers to enter their feedback.

[0895] User

[0896] Specifically, the user performs the following operations:

[0897] Receive summary text via push notification on your smartphone or tablet.

[0898] Open the push notification, read the summary, click the "Like" button, or post a comment.

[0899] Enter feedback about your shopping experience and products using feedback devices in physical stores.

[0900] Specific examples of hardware and software used

[0901] Hardware

[0902] Tablet device (e.g. iPad)

[0903] Smartphone (e.g. iPhone)

[0904] Server (Cloud server, e.g. AWS EC2)

[0905] Digital Signage

[0906] software

[0907] Speech recognition engine (e.g. Google Cloud Speech-to-Text)

[0908] Natural language processing engine (e.g. IBM Watson NLU)

[0909] Sentiment engine (e.g. Microsoft Azure Text Analytics)

[0910] Database (e.g. MySQL)

[0911] Application (dedicated feedback collection app)

[0912] Specific examples of data processing procedures

[0913] For example, Customer A sends a comment to a physical store's feedback app saying, "The new cake was delicious!" The server receives the comment and converts it into text using a speech recognition engine. Next, a summary is generated using a natural language processing engine and an advertisement is added to the summary. This summary and advertisement are then sent to User A via a push notification. User A opens the notification, reads the summary, and clicks the "Like!" button to post feedback. The server receives this feedback and analyzes it using an emotion engine. The analysis results are transmitted to digital signage, and an advertisement corresponding to the emotion is displayed.

[0914] Prompt Sentence Examples

[0915] "Comment: "The new cake was absolutely delicious!"

[0916] Emotion: "Joy"

[0917] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0918] Step 1:

[0919] The server acquires the contents of the local assembly discussions. This acquisition is done by downloading audio data via an API. The acquired audio data is the input, and the audio data stored on the server is the output.

[0920] Step 2:

[0921] The server passes the acquired voice data to a voice recognition engine (e.g., Google Cloud Speech-to-Text) and converts it into text data. The voice recognition engine analyzes the voice data and generates corresponding text data. The input is voice data, and the output is text data.

[0922] Step 3:

[0923] The server passes the text data output from the speech recognition engine to a natural language processing engine (e.g., IBM Watson NLU), which summarizes the text data. The natural language processing engine extracts important points from the text data and generates a summary. The input is text data, and the output is a summary.

[0924] Step 4:

[0925] The server adds advertisements to the generated summary. The advertisements are stored in a database in advance, and the appropriate one is selected based on the content of the summary. The input is the summary and the advertisement database, and the output is the summary with the advertisement added.

[0926] Step 5:

[0927] The server delivers the summary with the ad added to the user's device. The delivery is done via push notification and sent to the application installed by the user. The input is the summary with the ad added, and the output is the push notification sent to the user's device.

[0928] Step 6:

[0929] A user receives a push notification on their device, taps the notification to open the application, and then views the summary and advertisement. Here, the input is the push notification, and the output is the user's viewing action.

[0930] Step 7:

[0931] Users rate or comment on summaries. Users click the "Like" button or enter their opinions in the comment field and submit them. The input is the user's rating or comment, and the output is the user's feedback data.

[0932] Step 8:

[0933] The terminal sends the user's rating and comment data to the server, which receives this data and stores it in a database. The input is the user's feedback data, and the output is the data stored in the database.

[0934] Step 9:

[0935] The server moderates newly submitted comment data, filtering comment content and eliminating inappropriate content. The input is user comment data, and the output is moderated comment data.

[0936] Step 10:

[0937] The server passes the moderated comment data to a sentiment engine (e.g., Microsoft Azure Text Analytics) for sentiment analysis. The sentiment engine analyzes the comment content and identifies the sentiment. The input is the moderated comment data, and the output is sentiment data.

[0938] Step 11:

[0939] The server personalizes the advertisements displayed on the digital signage in the store based on the emotional data identified by the emotion engine. This allows the optimal advertisement to be displayed according to the user's emotions. The input is emotional data, and the output is a personalized advertisement display.

[0940] Example prompt sentence:

[0941] text

[0942] "Comment: The new cake was delicious!

[0943] Emotion: Joy

[0944] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0945] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0946] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0947] [Third embodiment]

[0948] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0949] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0950] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0951] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0952] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0953] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0954] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0955] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0956] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0957] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0958] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0959] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0960] The present invention is a system that acquires the contents of discussions in local assemblies, summarizes them so that citizens can quickly understand them, and provides feedback on their opinions and evaluations. An example is shown below.

[0961] System Configuration

[0962] Server: Acquires parliamentary data, converts it into text using speech recognition, and summarizes it using natural language processing. It also adds advertisements to the summarized discussion content and delivers it to users.

[0963] Device: A device on which users can view summaries of discussions and enter ratings and comments. This includes smartphones, tablets, and PCs.

[0964] Users: Citizens who use the application to view summaries of local council discussions and rate and comment on them.

[0965] Processing content

[0966] 1. Acquiring discussion content

[0967] The server retrieves audio and video data through the API after each local council session, either in real time or periodically.

[0968] 2. Text conversion using voice recognition

[0969] The server passes the acquired voice data to a voice recognition engine and converts it into text data, which is used in the next step.

[0970] 3. Natural Language Processing Summarization

[0971] The server passes the converted data to a natural language processing engine, extracts key points, and generates a summary based on the extracted points. This summary is of an appropriate length to convey the content concisely.

[0972] 4. Adding Ads

[0973] The server adds advertising information provided by local businesses to the abstracts, which are appropriately inserted into each abstract and positioned to attract the user's attention.

[0974] 5. Summary distribution

[0975] The server delivers the generated summary and advertisement to registered users via push notification, which includes the title of the summary and a brief introduction.

[0976] 6. Accepting ratings and comments

[0977] Users tap the push notification on their device, open the app, and view the summary. They can then rate or comment using the "Like" button or comment field below the summary.

[0978] The terminal transmits the ratings and comments entered by the user to the server.

[0979] 7. Data Retention

[0980] The server stores the submitted ratings and comments in a database, which is later used for analysis and engagement evaluation.

[0981] 8. Moderation and Ad Management

[0982] The server moderates each new comment as it is posted, filtering out inappropriate content, and also manages advertising data and generates reports for advertisers.

[0983] Specific examples

[0984] For example, if a new park development plan is discussed at a local assembly on July 5th, the server will acquire the audio data after the assembly ends. Next, a speech recognition engine will convert the audio into text, and a natural language processing engine will extract key points and generate a summary. After that, the opinions and ratings of User A, who posted a comment saying, "I'm happy that a new park will be built!", will be moderated and saved in the database. An advertisement for a local cafe will be added to this summary and sent to User B via push notification.

[0985] This system will enable discussions in local assemblies to be communicated to citizens quickly and efficiently, creating an environment in which citizen feedback can be reflected in local administration.

[0986] The processing flow will be explained below.

[0987] Step 1:

[0988] After the local assembly session ends, the server uses the API to retrieve the audio data, which includes the contents of the assembly discussion and is later converted into text.

[0989] Step 2:

[0990] The server passes the acquired voice data to a voice recognition engine, which converts it into text data in real time or in batches. The voice recognition engine analyzes the content of the voice and outputs it as text information.

[0991] Step 3:

[0992] The server passes the text data output by the speech recognition engine to the natural language processing engine, which extracts important points from the text data and generates a summary. This summary is a concise summary of the content.

[0993] Step 4:

[0994] The server adds advertising information provided by local businesses to the generated abstract, which is then formatted to be displayed with the abstract and delivered to the user.

[0995] Step 5:

[0996] The server compiles the summary and advertisement and sends a push notification to registered users, which includes the title of the summary and a brief intro designed to grab the user's attention.

[0997] Step 6:

[0998] A user receives a push notification on their device and taps the notification to open the application, which displays a summary and an advertisement on the screen.

[0999] Step 7:

[1000] The user views the summary and taps the "Like" button. If necessary, the user can enter their opinion or feedback in the comment field and tap the submit button. This inputs the user's feedback.

[1001] Step 8:

[1002] The device sends the user's rating (likes) and comment data to the server, which is then recorded as user feedback.

[1003] Step 9:

[1004] The server stores the submitted rating and comment data in a database, which is later used for analysis and feedback evaluation.

[1005] Step 10:

[1006] The server moderates newly submitted comments to check for inappropriate content. Inappropriate comments are filtered out, and only appropriate comments are saved.

[1007] Step 11:

[1008] The server manages the advertising data and collects user interaction data, which is used to evaluate the effectiveness of the advertisements and generate reports as feedback to the advertisers.

[1009] Through these steps, this system provides a mechanism for quickly and efficiently communicating the contents of local assembly discussions to citizens and reflecting citizen feedback in the administration.

[1010] Example 1

[1011] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1012] While the content of local assembly discussions is important information for citizens, it is difficult to quickly and concisely grasp this information. Furthermore, there are limited ways for citizens to provide feedback on the content of discussions, resulting in a lack of two-way communication with local governments. Furthermore, local assembly information is not properly integrated with external advertising, limiting opportunities for local advertisers. To solve these issues, an appropriate system is needed.

[1013] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1014] In this invention, the server includes means for acquiring the contents of local assembly discussions, means for converting the acquired discussion contents into text using speech recognition, means for summarizing the converted text using natural language processing, means for adding advertisements to the summaries, means for distributing the summaries to users, means for users to post ratings or comments on the summaries, means for saving the ratings and comments in a database, means for moderating the content of the ratings and comments, and means for appropriately placing advertisements together with the summaries. This allows for quick and concise provision of local assembly information to citizens, smooth collection of feedback from citizens, and two-way communication. It also allows for effective provision of information to local advertisers.

[1015] A "local assembly" refers to the assembly of a local government, an institution that discusses and makes decisions on local public policies, budgets, etc.

[1016] "Discussion content" refers to all statements and debates made by members of parliament and representatives in local assemblies, as well as all related information.

[1017] A "means" refers to a device, technique, method, or process used to accomplish a particular purpose.

[1018] "Speech recognition" refers to the technology of converting speech into text data.

[1019] "Text conversion" refers to the conversion of audio information extracted from audio data or video data into readable text information.

[1020] "Natural language processing" refers to the technology that enables computers to understand, interpret, and generate human language.

[1021] A "summary" is a short, concise summary of the main content.

[1022] "Advertising" refers to the means of providing information to publicize a product, service, or business.

[1023] "Distribution" refers to the act of transmitting information so that specific users receive the information.

[1024] "Users" refer to people who access the system, consume the information provided, and provide feedback such as ratings and comments.

[1025] "Feedback" refers to reactions to the system and its offerings, such as ratings and comments provided by users.

[1026] "Evaluation" refers to the act of a user expressing their opinion or impression about the provided summary or content.

[1027] "Comment" refers to the act of a user inputting detailed opinions or impressions about the provided summary or content in words.

[1028] A "database" refers to an electronic system for systematically storing and managing information.

[1029] "Moderation" refers to the process of reviewing comments and feedback provided by users and determining whether they are appropriate.

[1030] "Advertiser" refers to an individual or organization that provides advertising and seeks to profit from such advertising.

[1031] "Interaction data" refers to information about user actions and reactions, which is a record of the interaction between a system and a user.

[1032] The present invention is a system that quickly and easily provides citizens with the content of discussions in local assemblies and collects their feedback. This system functions in cooperation with three parties: a server, a terminal, and a user. Specific embodiments are described below.

[1033] Server Functions and Processing

[1034] 1. Acquiring discussion content

[1035] The server retrieves audio and video data via an API each time a local council session ends. This data is collected in real time or periodically. The API can be publicly available from the local council or from a dedicated data provider.

[1036] 2. Text conversion using voice recognition

[1037] The server passes the acquired voice data to a voice recognition engine (e.g., a voice recognition cloud service) and converts it into text data, which is used for subsequent processing.

[1038] 3. Natural Language Processing Summarization

[1039] The server passes the converted discussion content to a natural language processing engine (e.g., a natural language processing library), extracts key points, and generates a summary. The summary is adjusted to an optimal length to convey the discussion content concisely.

[1040] 4. Adding Ads

[1041] The server adds advertisements from local vendors to the generated summary, and the advertisements are appropriately placed to attract the user's interest.

[1042] 5. Summary distribution

[1043] The server delivers the generated summary and advertisement to registered users via push notification (e.g., push notification cloud service).

[1044] 6. Accepting ratings and comments

[1045] The server accepts user ratings and comments on the summaries and stores them in a database, which is later used for analysis and engagement evaluation.

[1046] 7. Moderation and Ad Management

[1047] The server moderates content as new comments are posted, filtering out inappropriate content, and also manages advertising data and generates reports for advertisers.

[1048] Terminal functions and processing

[1049] 1. View the summary and enter your rating and comments

[1050] The device receives the push notification, and when the user taps the notification, the application opens and displays the summary. The user can then rate or comment on the summary using the "Like" button or comment field. The entered rating or comment is then sent to the server.

[1051] User operations

[1052] 1. Review and feedback on the summary

[1053] Users receive a push notification from their device and can review the summary. They can rate the summary and add comments, providing feedback on the local council discussion.

[1054] Specific examples

[1055] For example, if a new park development plan is discussed at a local assembly on July 5th, the server will obtain the audio data via an API after the assembly ends. Next, a speech recognition engine will convert the audio into text, and a natural language processing engine will extract key points and generate a summary. After that, User A's comment, "I'm happy that a new park will be built!", will be moderated and saved in the database. This summary, along with an advertisement for a local cafe, will be sent to User B via a push notification.

[1056] Prompt Sentence Examples

[1057] "A new park development plan was discussed at the local council meeting on July 5th. Please summarize the key points of the discussion."

[1058] This system will enable important information from local councils to be communicated to citizens quickly and efficiently, creating an environment in which citizen feedback can be reflected in local administration.

[1059] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1060] Step 1:

[1061] The server acquires the contents of discussions in local assemblies.

[1062] Specific operation: The server uses the API published by the local council to obtain audio and video data in real time or periodically.

[1063] Input: Local council audio and video data.

[1064] Output: Audio data file.

[1065] Step 2:

[1066] The server passes the acquired voice data to a voice recognition engine and converts it into text data.

[1067] Specific operation: The server sends the voice data to the voice recognition cloud service and obtains the corresponding text data.

[1068] Input: Audio data file.

[1069] Output: Text data.

[1070] Step 3:

[1071] The server passes the converted discussion content to a natural language processing engine, extracts important points, and generates a summary.

[1072] How it works: The server inputs text data into a natural language processing library and generates a summary of appropriate length. The summary is constructed by scoring important keywords and sentences and extracting the top content.

[1073] Input: Text data.

[1074] Output: Summary statement.

[1075] Step 4:

[1076] The server adds advertisements provided by local businesses to the summary.

[1077] What it does: The server extracts relevant ads from an ad database and inserts them into specific parts of the summary. Relevance is determined by taking into account keywords and themes that interest the user.

[1078] Input: Abstract, ad data.

[1079] Output: Summary text with ad added.

[1080] Step 5:

[1081] The server delivers the generated summary and advertisement to the user via push notification.

[1082] Specific operation: The server retrieves the device tokens of all users from the user database, and uses the push notification cloud service to generate and deliver notifications containing summary text and advertisements.

[1083] Input: Ad summary, user device token.

[1084] Output: Push notification.

[1085] Step 6:

[1086] The user receives a push notification on their device, opens the app, and views the summary.

[1087] What happens: The user taps the notification and sees the summary in the app, which includes an ad.

[1088] Enter: push notification.

[1089] Output: Summary and advertisement.

[1090] Step 7:

[1091] The user inputs an evaluation and comments on the summary.

[1092] What it does: Users use the "Like" button and comment field below the summary to enter their own rating or comment.

[1093] Input: User rating and comments.

[1094] Output: The input rating and comment data.

[1095] Step 8:

[1096] The terminal transmits the ratings and comments entered by the user to the server.

[1097] Specific operation: The entered ratings and comments are sent from the app to the server, where the data received is stored in a database.

[1098] Input: Rating and comment data.

[1099] Output: Send to server.

[1100] Step 9:

[1101] The server stores the submitted ratings and comments in a database.

[1102] Specific operation: The server stores the received rating and comment data in the corresponding location in the database.

[1103] Input: Rating and comment data.

[1104] Output: Database update.

[1105] Step 10:

[1106] The server moderates each new comment as it is posted, filtering out inappropriate content.

[1107] What it does: Moderation algorithms are applied to automatically identify and filter inappropriate keywords and content, with manual review where necessary.

[1108] Input: Rating and comment data.

[1109] Output: Moderated comment data.

[1110] Step 11:

[1111] The server manages the advertising data and generates reports for the advertisers.

[1112] Specific operation: The server collects and analyzes the logs of ad display, and automatically generates and sends reports to advertisers.

[1113] Input: Ad display log.

[1114] Output: Report to advertiser.

[1115] (Application example 1)

[1116] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1117] The content of discussions in local assemblies is difficult for many citizens to understand, and it is difficult to follow in real time. There is also a lack of mechanisms for citizens to quickly have their opinions reflected in local government. Furthermore, there are limited means to increase the effectiveness of advertising for local businesses. It is necessary to resolve these issues, increase citizen engagement, and promote transparency in local government and the sharing of discussion content.

[1118] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1119] In this invention, the server includes means for acquiring the contents of discussions in local assemblies, means for converting the acquired contents into text using speech recognition, means for summarizing the converted text using natural language processing, means for adding advertisements to the summaries, means for distributing the summaries to users, means for users to post ratings or comments on the summaries, means for storing the ratings and comments in a database, means for displaying the contents of discussions in local assemblies in a virtual space, and means for users to provide feedback in the virtual space. This allows citizens to quickly and efficiently understand the contents of discussions in local assemblies and provide feedback of their opinions in real time. It also enhances the advertising effectiveness of local businesses.

[1120] A "local assembly" is an institution in a local government that deliberates and decides on bills.

[1121] "Discussion content" refers to the various issues, proposals, and opinions that are deliberated in local assemblies.

[1122] "Speech recognition" is a technology that converts speech into text.

[1123] "Text conversion" refers to the process of converting audio or video data into text information.

[1124] "Natural language processing" is a technology that allows computers to understand, analyze, and generate human language.

[1125] "Summarization" refers to extracting important parts of text data and summarizing them concisely.

[1126] "Advertising" is promotional information intended to widely publicize a particular product or service.

[1127] "Distribution" refers to the transmission of information or data to multiple users.

[1128] "User" refers to citizens or individual users of the system.

[1129] "Evaluation" refers to a user expressing an opinion about a particular piece of information or service.

[1130] "Comments" refers to opinions or feedback provided by users.

[1131] A "database" is a collection of data that allows information to be managed efficiently and easily searched and updated.

[1132] "Virtual space" refers to a virtual three-dimensional space created using computer technology.

[1133] "Feedback" refers to reactions such as ratings and comments from users.

[1134] The system for implementing this invention consists of three main elements: a server, a terminal, and a user.

[1135] Server Features

[1136] The server efficiently retrieves the contents of local assembly discussions, converts them into text using speech recognition, then generates summaries using a natural language processing engine, adds advertisements to the summaries, and delivers them to users in the virtual space.

[1137] Specifically, the process is as follows: The server passes the voice data obtained from the local assembly to a voice recognition engine (for example, Google's voice recognition API) and converts it into text data. This text data is then used by OpenAI's natural language processing engine (such as GPT-3) to extract key points and generate a summary. Advertising information from local businesses is then appropriately inserted into the generated summary.

[1138] Device Features

[1139] The device functions as a device for users to view the summary and post ratings and comments. It can be a smartphone, tablet, or PC. When a user taps the push notification, an application opens and the summary is displayed. The user can then use the "Like!" button or comment field at the bottom of the summary to enter a rating or comment.

[1140] Virtual space functions

[1141] The virtual space provides a place where users can experience the contents of local assembly discussions in a virtual three-dimensional space. Specifically, there is a virtual assembly hall where the discussion contents are displayed, and users gathered there can view the discussion contents in real time and provide feedback. This can be achieved, for example, through a VR headset or a smartphone app.

[1142] example

[1143] For example, if a local assembly is debating a new city park plan, the server captures the audio data after the assembly. The audio data is converted into text using Google's speech recognition API, and a summary is generated using GPT-3. The summary is then supplemented with an advertisement for a local cafe and delivered to the user in the virtual space.

[1144] Users receive a push notification through their smartphone app and open it to view the summary. The user's comment, "I'm happy that a new park will be built!", is sent from the device to a server and stored in a database. This data will later be analyzed by the local government and used as part of their feedback.

[1145] Prompt Sentence Examples

[1146] Here is an example prompt for generating a summary:

[1147] "Summarize the following text: Today, the local assembly discussed the design of a new park. Council members exchanged opinions on the park's location, design, and budget, and debated points to improve the quality of life for residents."

[1148] This system allows citizens to quickly and efficiently understand the content of discussions in local assemblies and provide feedback on their opinions in real time. It can also increase the effectiveness of advertising for local businesses.

[1149] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1150] Step 1:

[1151] The server acquires audio data of the local assembly discussions. After each assembly, audio and video data are collected in real time or periodically through the API. In this step, the audio data is the input, and the output is the audio file stored on the server.

[1152] Step 2:

[1153] The server passes the acquired voice data to a voice recognition engine and converts it into text data. Specifically, it uses Google's voice recognition API to analyze the voice data and convert the discussion content into text. In this step, voice data is the input and text data is the output.

[1154] Step 3:

[1155] The server passes the converted data to a natural language processing engine, extracts key points, and generates a summary. Specifically, OpenAI's GPT-3 is used to generate a summary of the text data. In this step, the text data is the input and the summary is the output.

[1156] Step 4:

[1157] The server adds advertising information provided by local businesses to the summary. The advertising information is inserted appropriately in a position that is likely to attract the user's interest. In this step, the summary and advertising data are the input, and the summary with the added advertisement is the output.

[1158] Step 5:

[1159] The server delivers the generated summary and advertisement to registered users' devices via push notification. The push notification includes the title of the summary and a brief introduction. In this step, the summary with advertisement is the input, and the push notification to the user's device is the output.

[1160] Step 6:

[1161] The user taps the push notification on their device, opens the application, and reads the summary. The device displays the summary and provides an interface where the user can rate or comment using the "Like" button and comment field. In this step, receiving the push notification is the input, and the display of the summary and the interface for comment input are the outputs.

[1162] Step 7:

[1163] The terminal sends the ratings and comments entered by the user to the server. The ratings and comments entered by the user on the interface are generated as data and then sent. In this step, the user's ratings and comments are the input, and the data sent to the server is the output.

[1164] Step 8:

[1165] The server stores the submitted ratings and comments in a database. The stored data is later used for analysis and engagement evaluation. In this step, the rating and comment data are the input and the storage in the database is the output.

[1166] Step 9:

[1167] The server moderates each new comment as it is posted, filtering out inappropriate content, and also manages advertising data and generates reports for advertisers. In this step, user comments and advertising data are the input, and filtered comments and reports for advertisers are the output.

[1168] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1169] This invention combines an emotion engine with a system that acquires and summarizes the contents of local assembly discussions and distributes them to citizens, and collects and analyzes feedback from citizens. An example is shown below.

[1170] System Configuration

[1171] Server: This server has the function of acquiring parliamentary data, converting it into text using speech recognition, and summarizing it using natural language processing. It also adds advertisements to the summaries and delivers them to users. It also moderates user comments and analyzes their sentiment using an emotion engine.

[1172] Device: A device on which users can view summaries of discussions and enter ratings and comments. This includes smartphones, tablets, and PCs.

[1173] Users: Citizens who use the application to view summaries of local council discussions and provide ratings and comments.

[1174] Specific processing of the program

[1175] 1. Acquiring discussion content

[1176] After the local assembly session ends, the server uses an API to retrieve the audio data, which includes the contents of the assembly discussion and is then converted into text in a later process.

[1177] 2. Text conversion using voice recognition

[1178] The server passes the acquired voice data to a voice recognition engine, which converts it into text data in real time or in batches. The voice recognition engine analyzes the content of the voice and outputs it as text information.

[1179] 3. Natural Language Processing Summarization

[1180] The server passes the text data output by the speech recognition engine to the natural language processing engine, which extracts important points from the text data and generates a summary. This summary is a concise summary of the content.

[1181] 4. Adding Ads

[1182] The server adds advertising information provided by local businesses to the generated abstract, which is then formatted to be displayed with the abstract and delivered to the user.

[1183] 5. Summary distribution

[1184] The server compiles the summary and advertisement and sends a push notification to registered users, which includes the title of the summary and a brief intro designed to grab the user's attention.

[1185] 6. Accepting ratings and comments

[1186] A user receives a push notification on their device and taps the notification to open the application, which displays a summary and an advertisement on the screen.

[1187] The user views the summary and taps the "Like" button. If necessary, the user can enter their opinion or feedback in the comment field and tap the submit button. This inputs the user's feedback.

[1188] 7. Data transmission and storage

[1189] The device sends the user's rating (likes) and comment data to the server, which is then recorded as user feedback.

[1190] The server stores the submitted rating and comment data in a database, which is later used for analysis and feedback evaluation.

[1191] 8. Comment Moderation

[1192] The server moderates newly submitted comments to check for inappropriate content. Inappropriate comments are filtered out, and only appropriate comments are saved.

[1193] 9. Sentiment Analysis

[1194] The server passes the user's comment data to the emotion engine for emotion analysis. The emotion engine analyzes the content of the comment and identifies the emotion contained therein (e.g., joy, anger, sadness, etc.).

[1195] The server evaluates citizens' emotional reactions to the content of the discussion based on the emotional data analyzed by the emotion engine.

[1196] 10. Adjusting Ad Display

[1197] The server then adjusts the content of the advertisements based on the emotion data identified by the emotion engine. For example, if a user has a positive opinion, the server displays an advertisement that corresponds to that opinion, thereby personalizing the advertisements.

[1198] Specific examples

[1199] For example, if a new park development plan is discussed at a local assembly on July 5th, the server will collect the audio data after the assembly ends. The audio will then be converted into text using a speech recognition engine, and a natural language processing engine will extract key points to generate a summary. An advertisement for a local cafe will then be added to the summary, and the summary and advertisement will be sent to User A via push notification.

[1200] User A taps the notification, reads the summary in the application, and taps the "Like" button. He also posts a comment saying, "I'm so happy that a new park is opening!" The device sends the comment data to the server. The server stores the comment data in a database and simultaneously passes the comment to the emotion engine for analysis. The emotion engine identifies the emotion "joy," and the server uses this emotion data to adjust the display of more relevant ads to the same User A the next time.

[1201] Through this series of steps, the system not only quickly and efficiently conveys the contents of discussions in local assemblies to citizens and reflects citizen feedback in the administration, but also grasps their emotional reactions and further improves the accuracy of advertising.

[1202] The processing flow will be explained below.

[1203] Step 1:

[1204] The server uses an API to retrieve audio data after the local assembly session has ended. This audio data contains the contents of the assembly discussions and is stored for later processing to convert it into text.

[1205] Step 2:

[1206] The server passes the acquired voice data to a voice recognition engine, which converts it into text data in real time or in batches. The voice recognition engine analyzes the content of the voice and outputs the corresponding text information.

[1207] Step 3:

[1208] The server passes the text data output by the speech recognition engine to a natural language processing engine, which extracts important points from the text data and generates a concise summary of the discussion.

[1209] Step 4:

[1210] The server adds advertising information provided by local businesses to the generated abstract, which is then formatted to be displayed in the appropriate location along with the abstract.

[1211] Step 5:

[1212] The server compiles the summary and advertisement and sends a push notification to registered users, which includes the title of the summary and a brief intro designed to grab the user's attention.

[1213] Step 6:

[1214] A user receives a push notification on their device and taps the notification to open the application, which displays a summary and an advertisement on their device screen.

[1215] Step 7:

[1216] The user views the summary and taps the "Like" button. If necessary, the user can enter their opinion or feedback in the comment field and tap the submit button. This inputs the user's feedback.

[1217] Step 8:

[1218] The device sends the user's ratings (likes) and comment data to the server, which is then recorded as user feedback.

[1219] Step 9:

[1220] The server stores the submitted rating and comment data in a database, which is later used for analysis and engagement measurement.

[1221] Step 10:

[1222] The server moderates newly submitted comments to check for inappropriate content, filtering out inappropriate comments and keeping only appropriate comments.

[1223] Step 11:

[1224] The server passes the user's comment data to the emotion engine, which analyzes the content of the comment and identifies the emotions contained therein (e.g., joy, anger, sadness, etc.).

[1225] Step 12:

[1226] The server uses the emotion data identified by the emotion engine to assess citizens' emotional reactions to the content of the discussion, thereby understanding how citizens feel.

[1227] Step 13:

[1228] The server adjusts the content of the advertisements based on the emotional data. For example, if the user has a positive opinion, the server will display an advertisement that corresponds to that opinion, thereby personalizing the advertisements.

[1229] To give a concrete example, if a new park development plan is discussed at a local assembly on July 5th, the server will acquire the voice data after the assembly ends and convert it into text using a speech recognition engine. The server will then generate a summary using a natural language processing engine, add an advertisement for a local cafe to the summary, and send a push notification to User A. User A taps the notification to view the summary, then enters a comment such as "I'm so happy that a new park is being built!" and sends it. The server will pass this comment to the emotion engine, identify the emotion of joy, and adjust and deliver relevant advertisements to User A next time. By obtaining feedback using the emotion engine in this way, it is possible to understand citizens' reactions and deliver appropriate advertisements, thereby improving the effectiveness of the entire system.

[1230] Example 2

[1231] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1232] The content of discussions in local assemblies is important public information that should be shared with citizens. However, there is a need for a method to efficiently collect, summarize, and quickly provide this content to citizens. There is also a need for a method to improve government services by obtaining feedback from citizens and analyzing their emotional reactions. Current systems have difficulty meeting these requirements, and there are challenges, particularly with the additional time and effort required for sentiment analysis and advertising personalization.

[1233] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1234] In this invention, the server includes: means for acquiring the contents of discussions in local assemblies; means for converting the acquired contents of discussions into text using speech recognition; means for summarizing the converted text using natural language processing; means for adding advertisements to the summaries; means for distributing the summaries to users; means for users to post ratings or comments on the summaries; means for saving the ratings and comments in a database; means including a sentiment analysis engine for analyzing the saved rating and comment data and identifying sentiment; and means for adjusting the content of advertisements based on the results of the sentiment analysis. This makes it possible to quickly and efficiently provide the contents of discussions in local assemblies to citizens, and to deeply understand citizen feedback and reflect it in administrative services.

[1235] A local assembly is an administrative body in a specific region where representatives of local residents gather to deliberate and decide on local laws, budgets, policies, etc.

[1236] "Discussion content" refers to the totality of information and opinions that are taken up as topics in local assemblies and debated among participants.

[1237] A "means" is a method, process, or device used to achieve a particular purpose.

[1238] "Speech recognition" is a technology that analyzes voice data and converts it into text data.

[1239] "Text conversion" is the process of analyzing non-textual information such as audio data and image data and converting it into textual information.

[1240] "Natural language processing" is a field of technology in which computers understand, analyze, and generate human language.

[1241] "Summarization" refers to the process of extracting important information from long text data and summarizing it in a short, concise form, or the result of that process.

[1242] An "advertisement" is a message intended to inform the general public about a particular product, service, or information.

[1243] "Users" refers to local residents and the general public who use this system.

[1244] "Distribution" is the process of getting information or data to a specific recipient.

[1245] "Evaluation" is the act of a user giving positive or negative feedback on the content of a discussion or a summary.

[1246] A "comment" is an act in which a user writes down their own opinion or impression on the content of a discussion or a summary.

[1247] A "database" is a digital system for efficiently storing, retrieving, updating, and managing large amounts of data.

[1248] An "emotion analysis engine" is a technology that analyzes text data and identifies the emotions contained within it (e.g., joy, anger, sadness, etc.).

[1249] "Moderation" is the process of reviewing posted comments to check for inappropriate content.

[1250] This invention is a system that aims to improve administrative services by providing the contents of local assembly discussions to citizens quickly and efficiently and collecting and analyzing feedback from citizens. The components of the system and their specific operation are described below.

[1251] System Configuration

[1252] 1. Server

[1253] The server will play a central role in collecting and processing data from local council discussions. The server's main functions are as follows:

[1254] Acquiring discussion content

[1255] The server uses an API to retrieve audio data after a local assembly session ends. This audio data contains the contents of the assembly discussion and is later converted to text. Specifically, the server hits the API endpoint, sends parameters such as the date and session ID, and downloads the audio data. For example, the audio data is saved in MP3 format.

[1256] Voice Recognition

[1257] The server passes the acquired voice data to a speech recognition engine (e.g., Google Cloud Speech-to-Text) and converts the voice into text data. The server then puts the voice data into a queue for batch processing, and the speech recognition engine converts it into text. This text data is saved in the form of meeting minutes.

[1258] Natural Language Processing

[1259] The server passes the text data output from the speech recognition engine to a natural language processing engine (e.g., OpenAI GPT-4), extracts important points, and generates a summary. The server then prompts the engine with a prompt such as "Summarize the following text," and stores the generated summary within the system.

[1260] Adding Ads

[1261] The server adds advertising information provided by local businesses to the generated summary, searches for relevant advertisements (e.g., coupon information for local stores) in the advertisement database, and adds the advertisements to the end of the summary. The formatted summary and advertisements are saved in HTML or JSON format.

[1262] Summary distribution

[1263] The server uses a push notification service (e.g., Firebase Cloud Messaging) to deliver the summary and advertisement to the user. It obtains the registered user's device ID and sends a notification message containing the summary, advertisement title, and intro. The notification is displayed on the user's device.

[1264] Accepting ratings and comments

[1265] The server stores the ratings (likes) and comments that users make on summaries in a database, allowing us to use user feedback for future analysis.

[1266] Comment moderation

[1267] The server moderates newly submitted comments, using a text filtering library (e.g., CleanSpeak) to check for inappropriate content, rejecting comments that are deemed inappropriate, and storing only appropriate comments.

[1268] Sentiment analysis

[1269] The server passes the user's comment data to a sentiment analysis engine (e.g., IBM Watson Natural Language Understanding) to identify emotions from the content of the comment. The sentiment analysis engine analyzes emotions such as joy, anger, and sadness, and stores the analysis results in a database.

[1270] Ad display adjustment

[1271] The server adjusts the content of advertisements based on the analysis results of the emotion engine. For example, if the user expresses the emotion of "joy," it will display advertisements related to that emotion (such as ticket information for a new park opening event) as the next push notification or in-app advertisement.

[1272] 2. Terminal

[1273] The terminal is a device that allows users to view summaries of discussions and enter ratings and comments. This includes smartphones, tablets, and PCs.

[1274] 3. Users

[1275] Users are citizens who use the application to view summaries of local assembly discussions and provide ratings and comments.

[1276] Specific examples

[1277] For example, if a new park development plan is discussed at a local council meeting on July 5th, the specific process would be as follows:

[1278] 1. The server uses the API to obtain the parliamentary audio data.

[1279] 2. The server passes the acquired voice data to Google Cloud Speech-to-Text and converts it into text data.

[1280] 3. The server passes the text data to OpenAI GPT-4, which generates a summary with the prompt "Summarize the following text."

[1281] 4. The server adds advertisements for local stores to the generated summary.

[1282] 5. The server uses Firebase Cloud Messaging to send a push notification containing the summary and advertisement to User A.

[1283] 6. User A taps the notification, reads the summary in the app, taps "Like," and posts a comment saying, "I'm so happy that the new park is opening!"

[1284] 7. The device sends the comment data to the server, which stores it in a database.

[1285] 8. The server moderates comments and only appropriate comments are left.

[1286] 9. The server passes the comment data to the emotion engine to identify the emotion "joy."

[1287] 10. The server configures the next push notification to display an ad related to User A.

[1288] Prompt Sentence Examples

[1289] Here are some examples of prompts for generative AI models:

[1290] "A new park development plan was discussed at the local council meeting on July 5th. Please summarize the content of that discussion."

[1291] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1292] Step 1:

[1293] Acquiring discussion content

[1294] After the local council session ends, the server hits the API endpoint and sends parameters such as the date and session ID to retrieve the audio data. The input is the API endpoint and required parameters. The output is audio data in MP3 format. Specifically, an HTTP request is sent and the audio data is received as a response.

[1295] Step 2:

[1296] Speech recognition to text

[1297] The server places the acquired audio data into a batch processing queue and passes it to a speech recognition engine (e.g., Google Cloud Speech-to-Text). The input is audio data in MP3 format. The output is text data in minutes format. Specifically, the URL of the audio file is passed to the speech recognition engine, and the text data is received as the processing result. An API is called on the server and the results are saved.

[1298] Step 3:

[1299] Natural language processing summarization

[1300] The server passes the text data output from the speech recognition engine to a natural language processing engine (e.g., OpenAI GPT-4), and issues instructions using a prompt such as "Summarize the following text." The input is the text data of the minutes and the prompt. The output is a summary. Specifically, the text is sent to the natural language processing engine along with the prompt, and the generated summary is received.

[1301] Step 4:

[1302] Adding Ads

[1303] The server adds advertising information provided by local businesses to the generated summary. The input is the summary and an advertising database. The output is the summary with the advertising added. Specifically, it searches for relevant advertisements in the advertising database and adds the advertisements to the end of the summary. The formatted summary and advertisements are saved in HTML or JSON format.

[1304] Step 5:

[1305] Summary distribution

[1306] The server uses a push notification service (e.g., Firebase Cloud Messaging) to deliver the summary and advertisement to the user. The input is the summary, advertisement, and the registered user's device ID. The output is the result of sending the push notification. Specifically, a notification message containing the summary, advertisement title, and intro is created and sent via Firebase Cloud Messaging. The notification is then displayed on the user's device.

[1307] Step 6:

[1308] Accepting ratings and comments

[1309] The user taps the push notification, opens the application, and views the summary and advertisement. The input is the push notification and user actions (tap, comment). The output is rating data (likes) and comment data. Specifically, the user taps the "Like" button, or enters their opinion or feedback in the comment section and taps the send button. The feedback data is sent from the device to the server.

[1310] Step 7:

[1311] Data transmission and storage

[1312] The device sends user ratings (likes) and comment data to the server. The input is rating data (likes) and comment data. The output is the database update results. Specifically, the device receives ratings and comments, stores them in the appropriate fields, and saves them in the database. The saved data is used for later analysis.

[1313] Step 8:

[1314] Comment moderation

[1315] The server moderates newly submitted comments. The input is the comment data. The output is the result of only saving appropriate comments. Specifically, a text filtering library (e.g. CleanSpeak) is used to check for inappropriate content. Comments that are deemed inappropriate are rejected, and only appropriate comments are saved in the database.

[1316] Step 9:

[1317] Sentiment analysis

[1318] The server passes user comment data to a sentiment analysis engine (e.g., IBM Watson Natural Language Understanding) to identify emotions from the comments. The input is comment data. The output is the sentiment analysis results. Specifically, the comment data is sent to the sentiment analysis engine, which identifies emotions such as joy, anger, and sadness, receives the results, and stores them in a database.

[1319] Step 10:

[1320] Ad display adjustment

[1321] The server adjusts the ad content based on the emotion engine's analysis results. The input is the emotion analysis results. The output is the adjusted ad content. Specifically, based on the analysis results, if the user expresses the emotion of "joy," for example, the server sets the next push notification or in-app ad to display an ad related to that emotion (e.g., ticket information for a new park opening event).

[1322] (Application example 2)

[1323] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1324] While systems already exist for effectively communicating the content of local assembly discussions to citizens and efficiently collecting and analyzing their feedback, there is no established system for collecting customer feedback in brick-and-mortar stores and personalizing advertisements based on that feedback. As a result, brick-and-mortar stores are unable to quickly and appropriately reflect the diverse emotions and opinions of customers, and are unable to maximize the effectiveness of advertising. There is also a lack of automated systems for moderating inappropriate comments and personalizing advertisements based on sentiment analysis.

[1325] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1326] In this invention, the server includes means for acquiring the content of discussions in local assemblies, means for converting the acquired content of discussions into text using speech recognition, means for summarizing the converted text of discussions using natural language processing, means for adding advertisements to the summaries, means for distributing the summaries to users, means for users to post ratings or comments on the summaries, means for saving the ratings and comments in a database, means for collecting customer feedback from physical stores, means for moderating the collected feedback, means for sentiment analysis of the feedback data, and means for personalizing advertisements based on the sentiment analysis. This makes it possible to quickly and appropriately reflect customer feedback even in physical stores and personalize advertisements based on the results.

[1327] "Local assembly discussions" are records of the content of discussions held by assembly members and other stakeholders in local government assemblies.

[1328] "Speech recognition" is a technology that analyzes voice data and converts it into corresponding text data.

[1329] "Natural language processing" is a technology that analyzes text data and automatically understands its meaning and structure.

[1330] A "summary" is a short summary of the main points extracted from a long text.

[1331] "Adding advertisements" refers to the technique or act of inserting advertisements into abstracts or other content.

[1332] "Delivering to the user" refers to the technique or act of sending the created summary and related information to the user's terminal.

[1333] Posting a "rating or comment" means that a user inputs and sends their opinion or feedback on the content.

[1334] "Storing in a database" means recording the collected evaluations and comments in digital form so that they can be referenced and analyzed as needed.

[1335] "Brick-and-mortar customer feedback" refers to the opinions and evaluations that customers give about their experiences and products in brick-and-mortar stores.

[1336] "Moderation" refers to the process of examining the content of comments posted by users and removing or modifying inappropriate comments.

[1337] "Sentiment analysis" is a technology that identifies and analyzes emotional elements (e.g., joy, anger, sadness, etc.) from text data.

[1338] "Personalizing ads" means customizing advertising content based on the characteristics and behavior of individual users.

[1339] This invention is a system that acquires the contents of discussions in local assemblies, efficiently conveys them to citizens, and collects and analyzes feedback from citizens. We will specifically explain how this system can be applied to brick-and-mortar stores to collect customer feedback and personalize advertisements through sentiment analysis.

[1340] server

[1341] The server has the following main functions:

[1342] The contents of local assembly discussions are obtained via API.

[1343] The acquired voice data is passed to a voice recognition engine (e.g., Google Cloud Speech-to-Text) and converted into text data.

[1344] Summarize text data using a natural language processing engine (e.g., IBM Watson NLU).

[1345] Add an ad to the summary.

[1346] The summary and advertisement are delivered to the user's device via push notification.

[1347] Moderating customer feedback collected in physical stores.

[1348] Analyze the moderated feedback data with a sentiment engine (e.g., Microsoft Azure Text Analytics).

[1349] Personalize advertisements displayed on in-store digital signage based on sentiment analysis.

[1350] Terminal

[1351] The terminal has the following main features:

[1352] A device that allows users to view the summary and provide feedback. This includes tablets (e.g., iPad) and smartphones (e.g., iPhone).

[1353] An interface that displays a summary and allows users to post ratings or comments.

[1354] A feedback device installed in a physical store for customers to enter their feedback.

[1355] User

[1356] Specifically, the user performs the following operations:

[1357] Receive summary text via push notification on your smartphone or tablet.

[1358] Open the push notification, read the summary, click the "Like" button, or post a comment.

[1359] Enter feedback about your shopping experience and products using feedback devices in physical stores.

[1360] Specific examples of hardware and software used

[1361] Hardware

[1362] Tablet device (e.g. iPad)

[1363] Smartphone (e.g. iPhone)

[1364] Server (Cloud server, e.g. AWS EC2)

[1365] Digital Signage

[1366] software

[1367] Speech recognition engine (e.g. Google Cloud Speech-to-Text)

[1368] Natural language processing engine (e.g. IBM Watson NLU)

[1369] Sentiment engine (e.g. Microsoft Azure Text Analytics)

[1370] Database (e.g. MySQL)

[1371] Application (dedicated feedback collection app)

[1372] Specific examples of data processing procedures

[1373] For example, Customer A sends a comment to a physical store's feedback app saying, "The new cake was delicious!" The server receives the comment and converts it into text using a speech recognition engine. Next, a summary is generated using a natural language processing engine and an advertisement is added to the summary. This summary and advertisement are then sent to User A via a push notification. User A opens the notification, reads the summary, and clicks the "Like!" button to post feedback. The server receives this feedback and analyzes it using an emotion engine. The analysis results are transmitted to digital signage, and an advertisement corresponding to the emotion is displayed.

[1374] Prompt Sentence Examples

[1375] "Comment: "The new cake was absolutely delicious!"

[1376] Emotion: "Joy"

[1377] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1378] Step 1:

[1379] The server acquires the contents of the local assembly discussions. This acquisition is done by downloading audio data via an API. The acquired audio data is the input, and the audio data stored on the server is the output.

[1380] Step 2:

[1381] The server passes the acquired voice data to a voice recognition engine (e.g., Google Cloud Speech-to-Text) and converts it into text data. The voice recognition engine analyzes the voice data and generates corresponding text data. The input is voice data, and the output is text data.

[1382] Step 3:

[1383] The server passes the text data output from the speech recognition engine to a natural language processing engine (e.g., IBM Watson NLU), which summarizes the text data. The natural language processing engine extracts important points from the text data and generates a summary. The input is text data, and the output is a summary.

[1384] Step 4:

[1385] The server adds advertisements to the generated summary. The advertisements are stored in a database in advance, and the appropriate one is selected based on the content of the summary. The input is the summary and the advertisement database, and the output is the summary with the advertisement added.

[1386] Step 5:

[1387] The server delivers the summary with the ad added to the user's device. The delivery is done via push notification and sent to the application installed by the user. The input is the summary with the ad added, and the output is the push notification sent to the user's device.

[1388] Step 6:

[1389] A user receives a push notification on their device, taps the notification to open the application, and then views the summary and advertisement. Here, the input is the push notification, and the output is the user's viewing action.

[1390] Step 7:

[1391] Users rate or comment on summaries. Users click the "Like" button or enter their opinions in the comment field and submit them. The input is the user's rating or comment, and the output is the user's feedback data.

[1392] Step 8:

[1393] The terminal sends the user's rating and comment data to the server, which receives this data and stores it in a database. The input is the user's feedback data, and the output is the data stored in the database.

[1394] Step 9:

[1395] The server moderates newly submitted comment data, filtering comment content and eliminating inappropriate content. The input is user comment data, and the output is moderated comment data.

[1396] Step 10:

[1397] The server passes the moderated comment data to a sentiment engine (e.g., Microsoft Azure Text Analytics) for sentiment analysis. The sentiment engine analyzes the comment content and identifies the sentiment. The input is the moderated comment data, and the output is sentiment data.

[1398] Step 11:

[1399] The server personalizes the advertisements displayed on the digital signage in the store based on the emotional data identified by the emotion engine. This allows the optimal advertisement to be displayed according to the user's emotions. The input is emotional data, and the output is a personalized advertisement display.

[1400] Example prompt sentence:

[1401] text

[1402] "Comment: The new cake was delicious!

[1403] Emotion: Joy

[1404] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1405] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1406] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1407] [Fourth embodiment]

[1408] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1409] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1410] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1411] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1412] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1413] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1414] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1415] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1416] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1417] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1418] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1419] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1420] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1421] The present invention is a system that acquires the contents of discussions in local assemblies, summarizes them so that citizens can quickly understand them, and provides feedback on their opinions and evaluations. An example is shown below.

[1422] System Configuration

[1423] Server: Acquires parliamentary data, converts it into text using speech recognition, and summarizes it using natural language processing. It also adds advertisements to the summarized discussion content and delivers it to users.

[1424] Device: A device on which users can view summaries of discussions and enter ratings and comments. This includes smartphones, tablets, and PCs.

[1425] Users: Citizens who use the application to view summaries of local council discussions and rate and comment on them.

[1426] Processing content

[1427] 1. Acquiring discussion content

[1428] The server retrieves audio and video data through the API after each local council session, either in real time or periodically.

[1429] 2. Text conversion using voice recognition

[1430] The server passes the acquired voice data to a voice recognition engine and converts it into text data, which is used in the next step.

[1431] 3. Natural Language Processing Summarization

[1432] The server passes the converted data to a natural language processing engine, extracts key points, and generates a summary based on the extracted points. This summary is of an appropriate length to convey the content concisely.

[1433] 4. Adding Ads

[1434] The server adds advertising information provided by local businesses to the abstracts, which are appropriately inserted into each abstract and positioned to attract the user's attention.

[1435] 5. Summary distribution

[1436] The server delivers the generated summary and advertisement to registered users via push notification, which includes the title of the summary and a brief introduction.

[1437] 6. Accepting ratings and comments

[1438] Users tap the push notification on their device, open the app, and view the summary. They can then rate or comment using the "Like" button or comment field below the summary.

[1439] The terminal transmits the ratings and comments entered by the user to the server.

[1440] 7. Data Retention

[1441] The server stores the submitted ratings and comments in a database, which is later used for analysis and engagement evaluation.

[1442] 8. Moderation and Ad Management

[1443] The server moderates each new comment as it is posted, filtering out inappropriate content, and also manages advertising data and generates reports for advertisers.

[1444] Specific examples

[1445] For example, if a new park development plan is discussed at a local assembly on July 5th, the server will acquire the audio data after the assembly ends. Next, a speech recognition engine will convert the audio into text, and a natural language processing engine will extract key points and generate a summary. After that, the opinions and ratings of User A, who posted a comment saying, "I'm happy that a new park will be built!", will be moderated and saved in the database. An advertisement for a local cafe will be added to this summary and sent to User B via push notification.

[1446] This system will enable discussions in local assemblies to be communicated to citizens quickly and efficiently, creating an environment in which citizen feedback can be reflected in local administration.

[1447] The processing flow will be explained below.

[1448] Step 1:

[1449] After the local assembly session ends, the server uses the API to retrieve the audio data, which includes the contents of the assembly discussion and is later converted into text.

[1450] Step 2:

[1451] The server passes the acquired voice data to a voice recognition engine, which converts it into text data in real time or in batches. The voice recognition engine analyzes the content of the voice and outputs it as text information.

[1452] Step 3:

[1453] The server passes the text data output by the speech recognition engine to the natural language processing engine, which extracts important points from the text data and generates a summary. This summary is a concise summary of the content.

[1454] Step 4:

[1455] The server adds advertising information provided by local businesses to the generated abstract, which is then formatted to be displayed with the abstract and delivered to the user.

[1456] Step 5:

[1457] The server compiles the summary and advertisement and sends a push notification to registered users, which includes the title of the summary and a brief intro designed to grab the user's attention.

[1458] Step 6:

[1459] A user receives a push notification on their device and taps the notification to open the application, which displays a summary and an advertisement on the screen.

[1460] Step 7:

[1461] The user views the summary and taps the "Like" button. If necessary, the user can enter their opinion or feedback in the comment field and tap the submit button. This inputs the user's feedback.

[1462] Step 8:

[1463] The device sends the user's rating (likes) and comment data to the server, which is then recorded as user feedback.

[1464] Step 9:

[1465] The server stores the submitted rating and comment data in a database, which is later used for analysis and feedback evaluation.

[1466] Step 10:

[1467] The server moderates newly submitted comments to check for inappropriate content. Inappropriate comments are filtered out, and only appropriate comments are saved.

[1468] Step 11:

[1469] The server manages the advertising data and collects user interaction data, which is used to evaluate the effectiveness of the advertisements and generate reports as feedback to the advertisers.

[1470] Through these steps, this system provides a mechanism for quickly and efficiently communicating the contents of local assembly discussions to citizens and reflecting citizen feedback in the administration.

[1471] Example 1

[1472] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1473] While the content of local assembly discussions is important information for citizens, it is difficult to quickly and concisely grasp this information. Furthermore, there are limited ways for citizens to provide feedback on the content of discussions, resulting in a lack of two-way communication with local governments. Furthermore, local assembly information is not properly integrated with external advertising, limiting opportunities for local advertisers. To solve these issues, an appropriate system is needed.

[1474] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1475] In this invention, the server includes means for acquiring the contents of local assembly discussions, means for converting the acquired discussion contents into text using speech recognition, means for summarizing the converted text using natural language processing, means for adding advertisements to the summaries, means for distributing the summaries to users, means for users to post ratings or comments on the summaries, means for saving the ratings and comments in a database, means for moderating the content of the ratings and comments, and means for appropriately placing advertisements together with the summaries. This allows for quick and concise provision of local assembly information to citizens, smooth collection of feedback from citizens, and two-way communication. It also allows for effective provision of information to local advertisers.

[1476] A "local assembly" refers to the assembly of a local government, an institution that discusses and makes decisions on local public policies, budgets, etc.

[1477] "Discussion content" refers to all statements and debates made by members of parliament and representatives in local assemblies, as well as all related information.

[1478] A "means" refers to a device, technique, method, or process used to accomplish a particular purpose.

[1479] "Speech recognition" refers to the technology of converting speech into text data.

[1480] "Text conversion" refers to the conversion of audio information extracted from audio data or video data into readable text information.

[1481] "Natural language processing" refers to the technology that enables computers to understand, interpret, and generate human language.

[1482] A "summary" is a short, concise summary of the main content.

[1483] "Advertising" refers to the means of providing information to publicize a product, service, or business.

[1484] "Distribution" refers to the act of transmitting information so that specific users receive the information.

[1485] "Users" refer to people who access the system, consume the information provided, and provide feedback such as ratings and comments.

[1486] "Feedback" refers to reactions to the system and its offerings, such as ratings and comments provided by users.

[1487] "Evaluation" refers to the act of a user expressing their opinion or impression about the provided summary or content.

[1488] "Comment" refers to the act of a user inputting detailed opinions or impressions about the provided summary or content in words.

[1489] A "database" refers to an electronic system for systematically storing and managing information.

[1490] "Moderation" refers to the process of reviewing comments and feedback provided by users and determining whether they are appropriate.

[1491] "Advertiser" refers to an individual or organization that provides advertising and seeks to profit from such advertising.

[1492] "Interaction data" refers to information about user actions and reactions, which is a record of the interaction between a system and a user.

[1493] The present invention is a system that quickly and easily provides citizens with the content of discussions in local assemblies and collects their feedback. This system functions in cooperation with three parties: a server, a terminal, and a user. Specific embodiments are described below.

[1494] Server Functions and Processing

[1495] 1. Acquiring discussion content

[1496] The server retrieves audio and video data via an API each time a local council session ends. This data is collected in real time or periodically. The API can be publicly available from the local council or from a dedicated data provider.

[1497] 2. Text conversion using voice recognition

[1498] The server passes the acquired voice data to a voice recognition engine (e.g., a voice recognition cloud service) and converts it into text data, which is used for subsequent processing.

[1499] 3. Natural Language Processing Summarization

[1500] The server passes the converted discussion content to a natural language processing engine (e.g., a natural language processing library), extracts key points, and generates a summary. The summary is adjusted to an optimal length to convey the discussion content concisely.

[1501] 4. Adding Ads

[1502] The server adds advertisements from local vendors to the generated summary, and the advertisements are appropriately placed to attract the user's interest.

[1503] 5. Summary distribution

[1504] The server delivers the generated summary and advertisement to registered users via push notification (e.g., push notification cloud service).

[1505] 6. Accepting ratings and comments

[1506] The server accepts user ratings and comments on the summaries and stores them in a database, which is later used for analysis and engagement evaluation.

[1507] 7. Moderation and Ad Management

[1508] The server moderates content as new comments are posted, filtering out inappropriate content, and also manages advertising data and generates reports for advertisers.

[1509] Terminal functions and processing

[1510] 1. View the summary and enter your rating and comments

[1511] The device receives the push notification, and when the user taps the notification, the application opens and displays the summary. The user can then rate or comment on the summary using the "Like" button or comment field. The entered rating or comment is then sent to the server.

[1512] User operations

[1513] 1. Review and feedback on the summary

[1514] Users receive a push notification from their device and can review the summary. They can rate the summary and add comments, providing feedback on the local council discussion.

[1515] Specific examples

[1516] For example, if a new park development plan is discussed at a local assembly on July 5th, the server will obtain the audio data via an API after the assembly ends. Next, a speech recognition engine will convert the audio into text, and a natural language processing engine will extract key points and generate a summary. After that, User A's comment, "I'm happy that a new park will be built!", will be moderated and saved in the database. This summary, along with an advertisement for a local cafe, will be sent to User B via a push notification.

[1517] Prompt Sentence Examples

[1518] "A new park development plan was discussed at the local council meeting on July 5th. Please summarize the key points of the discussion."

[1519] This system will enable important information from local councils to be communicated to citizens quickly and efficiently, creating an environment in which citizen feedback can be reflected in local administration.

[1520] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1521] Step 1:

[1522] The server acquires the contents of discussions in local assemblies.

[1523] Specific operation: The server uses the API published by the local council to obtain audio and video data in real time or periodically.

[1524] Input: Local council audio and video data.

[1525] Output: Audio data file.

[1526] Step 2:

[1527] The server passes the acquired voice data to a voice recognition engine and converts it into text data.

[1528] Specific operation: The server sends the voice data to the voice recognition cloud service and obtains the corresponding text data.

[1529] Input: Audio data file.

[1530] Output: Text data.

[1531] Step 3:

[1532] The server passes the converted discussion content to a natural language processing engine, extracts important points, and generates a summary.

[1533] How it works: The server inputs text data into a natural language processing library and generates a summary of appropriate length. The summary is constructed by scoring important keywords and sentences and extracting the top content.

[1534] Input: Text data.

[1535] Output: Summary statement.

[1536] Step 4:

[1537] The server adds advertisements provided by local businesses to the summary.

[1538] What it does: The server extracts relevant ads from an ad database and inserts them into specific parts of the summary. Relevance is determined by taking into account keywords and themes that interest the user.

[1539] Input: Abstract, ad data.

[1540] Output: Summary text with ad added.

[1541] Step 5:

[1542] The server delivers the generated summary and advertisement to the user via push notification.

[1543] Specific operation: The server retrieves the device tokens of all users from the user database, and uses the push notification cloud service to generate and deliver notifications containing summary text and advertisements.

[1544] Input: Ad summary, user device token.

[1545] Output: Push notification.

[1546] Step 6:

[1547] The user receives a push notification on their device, opens the app, and views the summary.

[1548] What happens: The user taps the notification and sees the summary in the app, which includes an ad.

[1549] Enter: push notification.

[1550] Output: Summary and advertisement.

[1551] Step 7:

[1552] The user inputs an evaluation and comments on the summary.

[1553] What it does: Users use the "Like" button and comment field below the summary to enter their own rating or comment.

[1554] Input: User rating and comments.

[1555] Output: The input rating and comment data.

[1556] Step 8:

[1557] The terminal transmits the ratings and comments entered by the user to the server.

[1558] Specific operation: The entered ratings and comments are sent from the app to the server, where the data received is stored in a database.

[1559] Input: Rating and comment data.

[1560] Output: Send to server.

[1561] Step 9:

[1562] The server stores the submitted ratings and comments in a database.

[1563] Specific operation: The server stores the received rating and comment data in the corresponding location in the database.

[1564] Input: Rating and comment data.

[1565] Output: Database update.

[1566] Step 10:

[1567] The server moderates each new comment as it is posted, filtering out inappropriate content.

[1568] What it does: Moderation algorithms are applied to automatically identify and filter inappropriate keywords and content, with manual review where necessary.

[1569] Input: Rating and comment data.

[1570] Output: Moderated comment data.

[1571] Step 11:

[1572] The server manages the advertising data and generates reports for the advertisers.

[1573] Specific operation: The server collects and analyzes the logs of ad display, and automatically generates and sends reports to advertisers.

[1574] Input: Ad display log.

[1575] Output: Report to advertiser.

[1576] (Application example 1)

[1577] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1578] The content of discussions in local assemblies is difficult for many citizens to understand, and it is difficult to follow in real time. There is also a lack of mechanisms for citizens to quickly have their opinions reflected in local government. Furthermore, there are limited means to increase the effectiveness of advertising for local businesses. It is necessary to resolve these issues, increase citizen engagement, and promote transparency in local government and the sharing of discussion content.

[1579] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1580] In this invention, the server includes means for acquiring the contents of discussions in local assemblies, means for converting the acquired contents into text using speech recognition, means for summarizing the converted text using natural language processing, means for adding advertisements to the summaries, means for distributing the summaries to users, means for users to post ratings or comments on the summaries, means for storing the ratings and comments in a database, means for displaying the contents of discussions in local assemblies in a virtual space, and means for users to provide feedback in the virtual space. This allows citizens to quickly and efficiently understand the contents of discussions in local assemblies and provide feedback of their opinions in real time. It also enhances the advertising effectiveness of local businesses.

[1581] A "local assembly" is an institution in a local government that deliberates and decides on bills.

[1582] "Discussion content" refers to the various issues, proposals, and opinions that are deliberated in local assemblies.

[1583] "Speech recognition" is a technology that converts speech into text.

[1584] "Text conversion" refers to the process of converting audio or video data into text information.

[1585] "Natural language processing" is a technology that allows computers to understand, analyze, and generate human language.

[1586] "Summarization" refers to extracting important parts of text data and summarizing them concisely.

[1587] "Advertising" is promotional information intended to widely publicize a particular product or service.

[1588] "Distribution" refers to the transmission of information or data to multiple users.

[1589] "User" refers to citizens or individual users of the system.

[1590] "Evaluation" refers to a user expressing an opinion about a particular piece of information or service.

[1591] "Comments" refers to opinions or feedback provided by users.

[1592] A "database" is a collection of data that allows information to be managed efficiently and easily searched and updated.

[1593] "Virtual space" refers to a virtual three-dimensional space created using computer technology.

[1594] "Feedback" refers to reactions such as ratings and comments from users.

[1595] The system for implementing this invention consists of three main elements: a server, a terminal, and a user.

[1596] Server Features

[1597] The server efficiently retrieves the contents of local assembly discussions, converts them into text using speech recognition, then generates summaries using a natural language processing engine, adds advertisements to the summaries, and delivers them to users in the virtual space.

[1598] Specifically, the process is as follows: The server passes the voice data obtained from the local assembly to a voice recognition engine (for example, Google's voice recognition API) and converts it into text data. This text data is then used by OpenAI's natural language processing engine (such as GPT-3) to extract key points and generate a summary. Advertising information from local businesses is then appropriately inserted into the generated summary.

[1599] Device Features

[1600] The device functions as a device for users to view the summary and post ratings and comments. It can be a smartphone, tablet, or PC. When a user taps the push notification, an application opens and the summary is displayed. The user can then use the "Like!" button or comment field at the bottom of the summary to enter a rating or comment.

[1601] Virtual space functions

[1602] The virtual space provides a place where users can experience the contents of local assembly discussions in a virtual three-dimensional space. Specifically, there is a virtual assembly hall where the discussion contents are displayed, and users gathered there can view the discussion contents in real time and provide feedback. This can be achieved, for example, through a VR headset or a smartphone app.

[1603] example

[1604] For example, if a local assembly is debating a new city park plan, the server captures the audio data after the assembly. The audio data is converted into text using Google's speech recognition API, and a summary is generated using GPT-3. The summary is then supplemented with an advertisement for a local cafe and delivered to the user in the virtual space.

[1605] Users receive a push notification through their smartphone app and open it to view the summary. The user's comment, "I'm happy that a new park will be built!", is sent from the device to a server and stored in a database. This data will later be analyzed by the local government and used as part of their feedback.

[1606] Prompt Sentence Examples

[1607] Here is an example prompt for generating a summary:

[1608] "Summarize the following text: Today, the local assembly discussed the design of a new park. Council members exchanged opinions on the park's location, design, and budget, and debated points to improve the quality of life for residents."

[1609] This system allows citizens to quickly and efficiently understand the content of discussions in local assemblies and provide feedback on their opinions in real time. It can also increase the effectiveness of advertising for local businesses.

[1610] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1611] Step 1:

[1612] The server acquires audio data of the local assembly discussions. After each assembly, audio and video data are collected in real time or periodically through the API. In this step, the audio data is the input, and the output is the audio file stored on the server.

[1613] Step 2:

[1614] The server passes the acquired voice data to a voice recognition engine and converts it into text data. Specifically, it uses Google's voice recognition API to analyze the voice data and convert the discussion content into text. In this step, voice data is the input and text data is the output.

[1615] Step 3:

[1616] The server passes the converted data to a natural language processing engine, extracts key points, and generates a summary. Specifically, OpenAI's GPT-3 is used to generate a summary of the text data. In this step, the text data is the input and the summary is the output.

[1617] Step 4:

[1618] The server adds advertising information provided by local businesses to the summary. The advertising information is inserted appropriately in a position that is likely to attract the user's interest. In this step, the summary and advertising data are the input, and the summary with the added advertisement is the output.

[1619] Step 5:

[1620] The server delivers the generated summary and advertisement to registered users' devices via push notification. The push notification includes the title of the summary and a brief introduction. In this step, the summary with advertisement is the input, and the push notification to the user's device is the output.

[1621] Step 6:

[1622] The user taps the push notification on their device, opens the application, and reads the summary. The device displays the summary and provides an interface where the user can rate or comment using the "Like" button and comment field. In this step, receiving the push notification is the input, and the display of the summary and the interface for comment input are the outputs.

[1623] Step 7:

[1624] The terminal sends the ratings and comments entered by the user to the server. The ratings and comments entered by the user on the interface are generated as data and then sent. In this step, the user's ratings and comments are the input, and the data sent to the server is the output.

[1625] Step 8:

[1626] The server stores the submitted ratings and comments in a database. The stored data is later used for analysis and engagement evaluation. In this step, the rating and comment data are the input and the storage in the database is the output.

[1627] Step 9:

[1628] The server moderates each new comment as it is posted, filtering out inappropriate content, and also manages advertising data and generates reports for advertisers. In this step, user comments and advertising data are the input, and filtered comments and reports for advertisers are the output.

[1629] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1630] This invention combines an emotion engine with a system that acquires and summarizes the contents of local assembly discussions and distributes them to citizens, and collects and analyzes feedback from citizens. An example is shown below.

[1631] System Configuration

[1632] Server: This server has the function of acquiring parliamentary data, converting it into text using speech recognition, and summarizing it using natural language processing. It also adds advertisements to the summaries and delivers them to users. It also moderates user comments and analyzes their sentiment using an emotion engine.

[1633] Device: A device on which users can view summaries of discussions and enter ratings and comments. This includes smartphones, tablets, and PCs.

[1634] Users: Citizens who use the application to view summaries of local council discussions and provide ratings and comments.

[1635] Specific processing of the program

[1636] 1. Acquiring discussion content

[1637] After the local assembly session ends, the server uses an API to retrieve the audio data, which includes the contents of the assembly discussion and is then converted into text in a later process.

[1638] 2. Text conversion using voice recognition

[1639] The server passes the acquired voice data to a voice recognition engine, which converts it into text data in real time or in batches. The voice recognition engine analyzes the content of the voice and outputs it as text information.

[1640] 3. Natural Language Processing Summarization

[1641] The server passes the text data output by the speech recognition engine to the natural language processing engine, which extracts important points from the text data and generates a summary. This summary is a concise summary of the content.

[1642] 4. Adding Ads

[1643] The server adds advertising information provided by local businesses to the generated abstract, which is then formatted to be displayed with the abstract and delivered to the user.

[1644] 5. Summary distribution

[1645] The server compiles the summary and advertisement and sends a push notification to registered users, which includes the title of the summary and a brief intro designed to grab the user's attention.

[1646] 6. Accepting ratings and comments

[1647] A user receives a push notification on their device and taps the notification to open the application, which displays a summary and an advertisement on the screen.

[1648] The user views the summary and taps the "Like" button. If necessary, the user can enter their opinion or feedback in the comment field and tap the submit button. This inputs the user's feedback.

[1649] 7. Data transmission and storage

[1650] The device sends the user's rating (likes) and comment data to the server, which is then recorded as user feedback.

[1651] The server stores the submitted rating and comment data in a database, which is later used for analysis and feedback evaluation.

[1652] 8. Comment Moderation

[1653] The server moderates newly submitted comments to check for inappropriate content. Inappropriate comments are filtered out, and only appropriate comments are saved.

[1654] 9. Sentiment Analysis

[1655] The server passes the user's comment data to the emotion engine for emotion analysis. The emotion engine analyzes the content of the comment and identifies the emotion contained therein (e.g., joy, anger, sadness, etc.).

[1656] The server evaluates citizens' emotional reactions to the content of the discussion based on the emotional data analyzed by the emotion engine.

[1657] 10. Adjusting Ad Display

[1658] The server then adjusts the content of the advertisements based on the emotion data identified by the emotion engine. For example, if a user has a positive opinion, the server displays an advertisement that corresponds to that opinion, thereby personalizing the advertisements.

[1659] Specific examples

[1660] For example, if a new park development plan is discussed at a local assembly on July 5th, the server will collect the audio data after the assembly ends. The audio will then be converted into text using a speech recognition engine, and a natural language processing engine will extract key points to generate a summary. An advertisement for a local cafe will then be added to the summary, and the summary and advertisement will be sent to User A via push notification.

[1661] User A taps the notification, reads the summary in the application, and taps the "Like" button. He also posts a comment saying, "I'm so happy that a new park is opening!" The device sends the comment data to the server. The server stores the comment data in a database and simultaneously passes the comment to the emotion engine for analysis. The emotion engine identifies the emotion "joy," and the server uses this emotion data to adjust the display of more relevant ads to the same User A the next time.

[1662] Through this series of steps, the system not only quickly and efficiently conveys the contents of discussions in local assemblies to citizens and reflects citizen feedback in the administration, but also grasps their emotional reactions and further improves the accuracy of advertising.

[1663] The processing flow will be explained below.

[1664] Step 1:

[1665] The server uses an API to retrieve audio data after the local assembly session has ended. This audio data contains the contents of the assembly discussions and is stored for later processing to convert it into text.

[1666] Step 2:

[1667] The server passes the acquired voice data to a voice recognition engine, which converts it into text data in real time or in batches. The voice recognition engine analyzes the content of the voice and outputs the corresponding text information.

[1668] Step 3:

[1669] The server passes the text data output by the speech recognition engine to a natural language processing engine, which extracts important points from the text data and generates a concise summary of the discussion.

[1670] Step 4:

[1671] The server adds advertising information provided by local businesses to the generated abstract, which is then formatted to be displayed in the appropriate location along with the abstract.

[1672] Step 5:

[1673] The server compiles the summary and advertisement and sends a push notification to registered users, which includes the title of the summary and a brief intro designed to grab the user's attention.

[1674] Step 6:

[1675] A user receives a push notification on their device and taps the notification to open the application, which displays a summary and an advertisement on their device screen.

[1676] Step 7:

[1677] The user views the summary and taps the "Like" button. If necessary, the user can enter their opinion or feedback in the comment field and tap the submit button. This inputs the user's feedback.

[1678] Step 8:

[1679] The device sends the user's ratings (likes) and comment data to the server, which is then recorded as user feedback.

[1680] Step 9:

[1681] The server stores the submitted rating and comment data in a database, which is later used for analysis and engagement measurement.

[1682] Step 10:

[1683] The server moderates newly submitted comments to check for inappropriate content, filtering out inappropriate comments and keeping only appropriate comments.

[1684] Step 11:

[1685] The server passes the user's comment data to the emotion engine, which analyzes the content of the comment and identifies the emotions contained therein (e.g., joy, anger, sadness, etc.).

[1686] Step 12:

[1687] The server uses the emotion data identified by the emotion engine to assess citizens' emotional reactions to the content of the discussion, thereby understanding how citizens feel.

[1688] Step 13:

[1689] The server adjusts the content of the advertisements based on the emotional data. For example, if the user has a positive opinion, the server will display an advertisement that corresponds to that opinion, thereby personalizing the advertisements.

[1690] To give a concrete example, if a new park development plan is discussed at a local assembly on July 5th, the server will acquire the voice data after the assembly ends and convert it into text using a speech recognition engine. The server will then generate a summary using a natural language processing engine, add an advertisement for a local cafe to the summary, and send a push notification to User A. User A taps the notification to view the summary, then enters a comment such as "I'm so happy that a new park is being built!" and sends it. The server will pass this comment to the emotion engine, identify the emotion of joy, and adjust and deliver relevant advertisements to User A next time. By obtaining feedback using the emotion engine in this way, it is possible to understand citizens' reactions and deliver appropriate advertisements, thereby improving the effectiveness of the entire system.

[1691] Example 2

[1692] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1693] The content of discussions in local assemblies is important public information that should be shared with citizens. However, there is a need for a method to efficiently collect, summarize, and quickly provide this content to citizens. There is also a need for a method to improve government services by obtaining feedback from citizens and analyzing their emotional reactions. Current systems have difficulty meeting these requirements, and there are challenges, particularly with the additional time and effort required for sentiment analysis and advertising personalization.

[1694] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1695] In this invention, the server includes: means for acquiring the contents of discussions in local assemblies; means for converting the acquired contents of discussions into text using speech recognition; means for summarizing the converted text using natural language processing; means for adding advertisements to the summaries; means for distributing the summaries to users; means for users to post ratings or comments on the summaries; means for saving the ratings and comments in a database; means including a sentiment analysis engine for analyzing the saved rating and comment data and identifying sentiment; and means for adjusting the content of advertisements based on the results of the sentiment analysis. This makes it possible to quickly and efficiently provide the contents of discussions in local assemblies to citizens, and to deeply understand citizen feedback and reflect it in administrative services.

[1696] A local assembly is an administrative body in a specific region where representatives of local residents gather to deliberate and decide on local laws, budgets, policies, etc.

[1697] "Discussion content" refers to the totality of information and opinions that are taken up as topics in local assemblies and debated among participants.

[1698] A "means" is a method, process, or device used to achieve a particular purpose.

[1699] "Speech recognition" is a technology that analyzes voice data and converts it into text data.

[1700] "Text conversion" is the process of analyzing non-textual information such as audio data and image data and converting it into textual information.

[1701] "Natural language processing" is a field of technology in which computers understand, analyze, and generate human language.

[1702] "Summarization" refers to the process of extracting important information from long text data and summarizing it in a short, concise form, or the result of that process.

[1703] An "advertisement" is a message intended to inform the general public about a particular product, service, or information.

[1704] "Users" refers to local residents and the general public who use this system.

[1705] "Distribution" is the process of getting information or data to a specific recipient.

[1706] "Evaluation" is the act of a user giving positive or negative feedback on the content of a discussion or a summary.

[1707] A "comment" is an act in which a user writes down their own opinion or impression on the content of a discussion or a summary.

[1708] A "database" is a digital system for efficiently storing, retrieving, updating, and managing large amounts of data.

[1709] An "emotion analysis engine" is a technology that analyzes text data and identifies the emotions contained within it (e.g., joy, anger, sadness, etc.).

[1710] "Moderation" is the process of reviewing posted comments to check for inappropriate content.

[1711] This invention is a system that aims to improve administrative services by providing the contents of local assembly discussions to citizens quickly and efficiently and collecting and analyzing feedback from citizens. The components of the system and their specific operation are described below.

[1712] System Configuration

[1713] 1. Server

[1714] The server will play a central role in collecting and processing data from local council discussions. The server's main functions are as follows:

[1715] Acquiring discussion content

[1716] The server uses an API to retrieve audio data after a local assembly session ends. This audio data contains the contents of the assembly discussion and is later converted to text. Specifically, the server hits the API endpoint, sends parameters such as the date and session ID, and downloads the audio data. For example, the audio data is saved in MP3 format.

[1717] Voice Recognition

[1718] The server passes the acquired voice data to a speech recognition engine (e.g., Google Cloud Speech-to-Text) and converts the voice into text data. The server then puts the voice data into a queue for batch processing, and the speech recognition engine converts it into text. This text data is saved in the form of meeting minutes.

[1719] Natural Language Processing

[1720] The server passes the text data output from the speech recognition engine to a natural language processing engine (e.g., OpenAI GPT-4), extracts important points, and generates a summary. The server then prompts the engine with a prompt such as "Summarize the following text," and stores the generated summary within the system.

[1721] Adding Ads

[1722] The server adds advertising information provided by local businesses to the generated summary, searches for relevant advertisements (e.g., coupon information for local stores) in the advertisement database, and adds the advertisements to the end of the summary. The formatted summary and advertisements are saved in HTML or JSON format.

[1723] Summary distribution

[1724] The server uses a push notification service (e.g., Firebase Cloud Messaging) to deliver the summary and advertisement to the user. It obtains the registered user's device ID and sends a notification message containing the summary, advertisement title, and intro. The notification is displayed on the user's device.

[1725] Accepting ratings and comments

[1726] The server stores the ratings (likes) and comments that users make on summaries in a database, allowing us to use user feedback for future analysis.

[1727] Comment moderation

[1728] The server moderates newly submitted comments, using a text filtering library (e.g., CleanSpeak) to check for inappropriate content, rejecting comments that are deemed inappropriate, and storing only appropriate comments.

[1729] Sentiment analysis

[1730] The server passes the user's comment data to a sentiment analysis engine (e.g., IBM Watson Natural Language Understanding) to identify emotions from the content of the comment. The sentiment analysis engine analyzes emotions such as joy, anger, and sadness, and stores the analysis results in a database.

[1731] Ad display adjustment

[1732] The server adjusts the content of advertisements based on the analysis results of the emotion engine. For example, if the user expresses the emotion of "joy," it will display advertisements related to that emotion (such as ticket information for a new park opening event) as the next push notification or in-app advertisement.

[1733] 2. Terminal

[1734] The terminal is a device that allows users to view summaries of discussions and enter ratings and comments. This includes smartphones, tablets, and PCs.

[1735] 3. Users

[1736] Users are citizens who use the application to view summaries of local assembly discussions and provide ratings and comments.

[1737] Specific examples

[1738] For example, if a new park development plan is discussed at a local council meeting on July 5th, the specific process would be as follows:

[1739] 1. The server uses the API to obtain the parliamentary audio data.

[1740] 2. The server passes the acquired voice data to Google Cloud Speech-to-Text and converts it into text data.

[1741] 3. The server passes the text data to OpenAI GPT-4, which generates a summary with the prompt "Summarize the following text."

[1742] 4. The server adds advertisements for local stores to the generated summary.

[1743] 5. The server uses Firebase Cloud Messaging to send a push notification containing the summary and advertisement to User A.

[1744] 6. User A taps the notification, reads the summary in the app, taps "Like," and posts a comment saying, "I'm so happy that the new park is opening!"

[1745] 7. The device sends the comment data to the server, which stores it in a database.

[1746] 8. The server moderates comments and only appropriate comments are left.

[1747] 9. The server passes the comment data to the emotion engine to identify the emotion "joy."

[1748] 10. The server configures the next push notification to display an ad related to User A.

[1749] Prompt Sentence Examples

[1750] Here are some examples of prompts for generative AI models:

[1751] "A new park development plan was discussed at the local council meeting on July 5th. Please summarize the content of that discussion."

[1752] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1753] Step 1:

[1754] Acquiring discussion content

[1755] After the local council session ends, the server hits the API endpoint and sends parameters such as the date and session ID to retrieve the audio data. The input is the API endpoint and required parameters. The output is audio data in MP3 format. Specifically, an HTTP request is sent and the audio data is received as a response.

[1756] Step 2:

[1757] Speech recognition to text

[1758] The server places the acquired audio data into a batch processing queue and passes it to a speech recognition engine (e.g., Google Cloud Speech-to-Text). The input is audio data in MP3 format. The output is text data in minutes format. Specifically, the URL of the audio file is passed to the speech recognition engine, and the text data is received as the processing result. An API is called on the server and the results are saved.

[1759] Step 3:

[1760] Natural language processing summarization

[1761] The server passes the text data output from the speech recognition engine to a natural language processing engine (e.g., OpenAI GPT-4), and issues instructions using a prompt such as "Summarize the following text." The input is the text data of the minutes and the prompt. The output is a summary. Specifically, the text is sent to the natural language processing engine along with the prompt, and the generated summary is received.

[1762] Step 4:

[1763] Adding Ads

[1764] The server adds advertising information provided by local businesses to the generated summary. The input is the summary and an advertising database. The output is the summary with the advertising added. Specifically, it searches for relevant advertisements in the advertising database and adds the advertisements to the end of the summary. The formatted summary and advertisements are saved in HTML or JSON format.

[1765] Step 5:

[1766] Summary distribution

[1767] The server uses a push notification service (e.g., Firebase Cloud Messaging) to deliver the summary and advertisement to the user. The input is the summary, advertisement, and the registered user's device ID. The output is the result of sending the push notification. Specifically, a notification message containing the summary, advertisement title, and intro is created and sent via Firebase Cloud Messaging. The notification is then displayed on the user's device.

[1768] Step 6:

[1769] Accepting ratings and comments

[1770] The user taps the push notification, opens the application, and views the summary and advertisement. The input is the push notification and user actions (tap, comment). The output is rating data (likes) and comment data. Specifically, the user taps the "Like" button, or enters their opinion or feedback in the comment section and taps the send button. The feedback data is sent from the device to the server.

[1771] Step 7:

[1772] Data transmission and storage

[1773] The device sends user ratings (likes) and comment data to the server. The input is rating data (likes) and comment data. The output is the database update results. Specifically, the device receives ratings and comments, stores them in the appropriate fields, and saves them in the database. The saved data is used for later analysis.

[1774] Step 8:

[1775] Comment moderation

[1776] The server moderates newly submitted comments. The input is the comment data. The output is the result of only saving appropriate comments. Specifically, a text filtering library (e.g. CleanSpeak) is used to check for inappropriate content. Comments that are deemed inappropriate are rejected, and only appropriate comments are saved in the database.

[1777] Step 9:

[1778] Sentiment analysis

[1779] The server passes user comment data to a sentiment analysis engine (e.g., IBM Watson Natural Language Understanding) to identify emotions from the comments. The input is comment data. The output is the sentiment analysis results. Specifically, the comment data is sent to the sentiment analysis engine, which identifies emotions such as joy, anger, and sadness, receives the results, and stores them in a database.

[1780] Step 10:

[1781] Ad display adjustment

[1782] The server adjusts the ad content based on the emotion engine's analysis results. The input is the emotion analysis results. The output is the adjusted ad content. Specifically, based on the analysis results, if the user expresses the emotion of "joy," for example, the server sets the next push notification or in-app ad to display an ad related to that emotion (e.g., ticket information for a new park opening event).

[1783] (Application example 2)

[1784] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1785] While systems already exist for effectively communicating the content of local assembly discussions to citizens and efficiently collecting and analyzing their feedback, there is no established system for collecting customer feedback in brick-and-mortar stores and personalizing advertisements based on that feedback. As a result, brick-and-mortar stores are unable to quickly and appropriately reflect the diverse emotions and opinions of customers, and are unable to maximize the effectiveness of advertising. There is also a lack of automated systems for moderating inappropriate comments and personalizing advertisements based on sentiment analysis.

[1786] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1787] In this invention, the server includes means for acquiring the content of discussions in local assemblies, means for converting the acquired content of discussions into text using speech recognition, means for summarizing the converted text of discussions using natural language processing, means for adding advertisements to the summaries, means for distributing the summaries to users, means for users to post ratings or comments on the summaries, means for saving the ratings and comments in a database, means for collecting customer feedback from physical stores, means for moderating the collected feedback, means for sentiment analysis of the feedback data, and means for personalizing advertisements based on the sentiment analysis. This makes it possible to quickly and appropriately reflect customer feedback even in physical stores and personalize advertisements based on the results.

[1788] "Local assembly discussions" are records of the content of discussions held by assembly members and other stakeholders in local government assemblies.

[1789] "Speech recognition" is a technology that analyzes voice data and converts it into corresponding text data.

[1790] "Natural language processing" is a technology that analyzes text data and automatically understands its meaning and structure.

[1791] A "summary" is a short summary of the main points extracted from a long text.

[1792] "Adding advertisements" refers to the technique or act of inserting advertisements into abstracts or other content.

[1793] "Delivering to the user" refers to the technique or act of sending the created summary and related information to the user's terminal.

[1794] Posting a "rating or comment" means that a user inputs and sends their opinion or feedback on the content.

[1795] "Storing in a database" means recording the collected evaluations and comments in digital form so that they can be referenced and analyzed as needed.

[1796] "Brick-and-mortar customer feedback" refers to the opinions and evaluations that customers give about their experiences and products in brick-and-mortar stores.

[1797] "Moderation" refers to the process of examining the content of comments posted by users and removing or modifying inappropriate comments.

[1798] "Sentiment analysis" is a technology that identifies and analyzes emotional elements (e.g., joy, anger, sadness, etc.) from text data.

[1799] "Personalizing ads" means customizing advertising content based on the characteristics and behavior of individual users.

[1800] This invention is a system that acquires the contents of discussions in local assemblies, efficiently conveys them to citizens, and collects and analyzes feedback from citizens. We will specifically explain how this system can be applied to brick-and-mortar stores to collect customer feedback and personalize advertisements through sentiment analysis.

[1801] server

[1802] The server has the following main functions:

[1803] The contents of local assembly discussions are obtained via API.

[1804] The acquired voice data is passed to a voice recognition engine (e.g., Google Cloud Speech-to-Text) and converted into text data.

[1805] Summarize text data using a natural language processing engine (e.g., IBM Watson NLU).

[1806] Add an ad to the summary.

[1807] The summary and advertisement are delivered to the user's device via push notification.

[1808] Moderating customer feedback collected in physical stores.

[1809] Analyze the moderated feedback data with a sentiment engine (e.g., Microsoft Azure Text Analytics).

[1810] Personalize advertisements displayed on in-store digital signage based on sentiment analysis.

[1811] Terminal

[1812] The terminal has the following main features:

[1813] A device that allows users to view the summary and provide feedback. This includes tablets (e.g., iPad) and smartphones (e.g., iPhone).

[1814] An interface that displays a summary and allows users to post ratings or comments.

[1815] A feedback device installed in a physical store for customers to enter their feedback.

[1816] User

[1817] Specifically, the user performs the following operations:

[1818] Receive summary text via push notification on your smartphone or tablet.

[1819] Open the push notification, read the summary, click the "Like" button, or post a comment.

[1820] Enter feedback about your shopping experience and products using feedback devices in physical stores.

[1821] Specific examples of hardware and software used

[1822] Hardware

[1823] Tablet device (e.g. iPad)

[1824] Smartphone (e.g. iPhone)

[1825] Server (Cloud server, e.g. AWS EC2)

[1826] Digital Signage

[1827] software

[1828] Speech recognition engine (e.g. Google Cloud Speech-to-Text)

[1829] Natural language processing engine (e.g. IBM Watson NLU)

[1830] Sentiment engine (e.g. Microsoft Azure Text Analytics)

[1831] Database (e.g. MySQL)

[1832] Application (dedicated feedback collection app)

[1833] Specific examples of data processing procedures

[1834] For example, Customer A sends a comment to a physical store's feedback app saying, "The new cake was delicious!" The server receives the comment and converts it into text using a speech recognition engine. Next, a summary is generated using a natural language processing engine and an advertisement is added to the summary. This summary and advertisement are then sent to User A via a push notification. User A opens the notification, reads the summary, and clicks the "Like!" button to post feedback. The server receives this feedback and analyzes it using an emotion engine. The analysis results are transmitted to digital signage, and an advertisement corresponding to the emotion is displayed.

[1835] Prompt Sentence Examples

[1836] "Comment: "The new cake was absolutely delicious!"

[1837] Emotion: "Joy"

[1838] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1839] Step 1:

[1840] The server acquires the contents of the local assembly discussions. This acquisition is done by downloading audio data via an API. The acquired audio data is the input, and the audio data stored on the server is the output.

[1841] Step 2:

[1842] The server passes the acquired voice data to a voice recognition engine (e.g., Google Cloud Speech-to-Text) and converts it into text data. The voice recognition engine analyzes the voice data and generates corresponding text data. The input is voice data, and the output is text data.

[1843] Step 3:

[1844] The server passes the text data output from the speech recognition engine to a natural language processing engine (e.g., IBM Watson NLU), which summarizes the text data. The natural language processing engine extracts important points from the text data and generates a summary. The input is text data, and the output is a summary.

[1845] Step 4:

[1846] The server adds advertisements to the generated summary. The advertisements are stored in a database in advance, and the appropriate one is selected based on the content of the summary. The input is the summary and the advertisement database, and the output is the summary with the advertisement added.

[1847] Step 5:

[1848] The server delivers the summary with the ad added to the user's device. The delivery is done via push notification and sent to the application installed by the user. The input is the summary with the ad added, and the output is the push notification sent to the user's device.

[1849] Step 6:

[1850] A user receives a push notification on their device, taps the notification to open the application, and then views the summary and advertisement. Here, the input is the push notification, and the output is the user's viewing action.

[1851] Step 7:

[1852] Users rate or comment on summaries. Users click the "Like" button or enter their opinions in the comment field and submit them. The input is the user's rating or comment, and the output is the user's feedback data.

[1853] Step 8:

[1854] The terminal sends the user's rating and comment data to the server, which receives this data and stores it in a database. The input is the user's feedback data, and the output is the data stored in the database.

[1855] Step 9:

[1856] The server moderates newly submitted comment data, filtering comment content and eliminating inappropriate content. The input is user comment data, and the output is moderated comment data.

[1857] Step 10:

[1858] The server passes the moderated comment data to a sentiment engine (e.g., Microsoft Azure Text Analytics) for sentiment analysis. The sentiment engine analyzes the comment content and identifies the sentiment. The input is the moderated comment data, and the output is sentiment data.

[1859] Step 11:

[1860] The server personalizes the advertisements displayed on the digital signage in the store based on the emotional data identified by the emotion engine. This allows the optimal advertisement to be displayed according to the user's emotions. The input is emotional data, and the output is a personalized advertisement display.

[1861] Example prompt sentence:

[1862] text

[1863] "Comment: The new cake was delicious!

[1864] Emotion: Joy

[1865] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1866] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1867] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1868] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1869] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1870] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1871] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1872] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1873] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1874] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1875] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1876] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1877] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1878] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1879] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1880] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1881] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1882] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1883] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1884] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1885] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1886] The following is further disclosed regarding the above embodiment.

[1887] (Claim 1)

[1888] A means of obtaining the contents of local assembly discussions;

[1889] A means for converting the acquired discussion content into text using voice recognition;

[1890] A means for summarizing the content of a discussion that has been converted into text using natural language processing;

[1891] a means for adding advertisements to the summary;

[1892] means for delivering the summary to a user;

[1893] a means for users to post ratings or comments on the abstracts;

[1894] The system includes a means for storing the ratings and comments in a database.

[1895] (Claim 2)

[1896] 10. The system of claim 1, further comprising means for moderating comments posted by users.

[1897] (Claim 3)

[1898] 10. The system of claim 1, further comprising means for managing advertisements provided by local advertisers and collecting and analyzing user interaction data.

[1899] "Example 1"

[1900] (Claim 1)

[1901] A means of obtaining the contents of local assembly discussions;

[1902] A means for converting the acquired discussion content into text using voice recognition;

[1903] A means for summarizing the content of a discussion that has been converted into text using natural language processing;

[1904] a means for adding advertisements to the summary;

[1905] means for delivering the summary to a user;

[1906] a means for users to post ratings or comments on the abstracts;

[1907] a means for storing the ratings and comments in a database;

[1908] a means of moderating the content of ratings or comments;

[1909] means for appropriately placing the advertisement together with the abstract;

[1910] A system including:

[1911] (Claim 2)

[1912] 10. The system of claim 1, further comprising means for moderating user-submitted ratings and comments in real time.

[1913] (Claim 3)

[1914] 10. The system of claim 1, further comprising means for managing advertising information and aggregating and analyzing user interaction data.

[1915] "Application Example 1"

[1916] (Claim 1)

[1917] A means of obtaining the contents of local assembly discussions;

[1918] A means for converting the acquired discussion content into text using voice recognition;

[1919] A means for summarizing the content of a discussion that has been converted into text using natural language processing;

[1920] a means for adding advertisements to the summary;

[1921] means for delivering the summary to a user;

[1922] a means for users to post ratings or comments on the abstracts;

[1923] a means for storing the ratings and comments in a database;

[1924] A means of displaying the contents of local assembly discussions in a virtual space;

[1925] A system that includes a means for users to provide feedback within a virtual space.

[1926] (Claim 2)

[1927] 10. The system of claim 1, further comprising means for moderating comments posted by users.

[1928] (Claim 3)

[1929] 10. The system of claim 1, further comprising means for managing advertisements provided by local advertisers and collecting and analyzing user interaction data.

[1930] "Example 2: Combining Emotion Engines"

[1931] (Claim 1)

[1932] A means of obtaining the contents of local assembly discussions;

[1933] A means for converting the acquired discussion content into text using voice recognition;

[1934] A means for summarizing the content of a discussion that has been converted into text using natural language processing;

[1935] a means for adding advertisements to the summary;

[1936] means for delivering the summary to a user;

[1937] a means for users to post ratings or comments on the abstracts;

[1938] a means for storing the ratings and comments in a database;

[1939] means including a sentiment analysis engine for analyzing the stored ratings and comments data and identifying se...

Claims

1. A means of obtaining the contents of local assembly discussions; A means for converting the acquired discussion content into text using voice recognition; A means for summarizing the content of a discussion that has been converted into text using natural language processing; a means for adding advertisements to the summary; means for delivering the summary to a user; a means for users to post ratings or comments on the abstracts; The system includes a means for storing the ratings and comments in a database.

2. The system of claim 1 , further comprising means for moderating comments posted by users.

3. 10. The system of claim 1, further comprising means for managing advertisements provided by local advertisers and for collecting and analyzing user interaction data.

Images