System

Abstract

A system is provided.SOLUTION: A system comprising: means for acquiring a news article; means for analyzing the acquired news article and extracting a main keyword and a category; means for selecting an appropriate illustration template based on the extracted category; means for illustrating content of the news article based on the selected template; means for generating one or more illustration variations; and means for storing and distributing the generated illustration variations to a user terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Today, news articles are circulating in huge volumes, and it is often difficult to understand their contents, especially from text-based articles. Readers living busy lives need a way to quickly and easily grasp the contents of the news. It is also important to provide information in a format that meets the visual preferences and accessibility needs of each reader. This makes it necessary to provide news in a format that is easier to understand and tailored to each individual user. [Means for solving the problem]

[0005] The present invention provides a system that automatically acquires and analyzes news articles to extract key keywords and categories. Based on the extracted categories, it selects an appropriate illustration template to illustrate the content of the news article. It also generates one or more illustration variations (e.g., different languages, dark mode, different font sizes), stores these illustration variations, and delivers them to user devices.

[0006] The user terminal displays the received news article and associated illustrations and allows the user to select display options. Based on the user's selection, the system displays the most appropriate illustration variation, facilitating understanding of the news and meeting the visual needs of individual users. In this way, the system can promote visual understanding of news articles and meet the needs of a diverse range of users.

[0007] A "news article" refers to any medium, such as text, image, or video, that reports current events or information.

[0008] "Extraction" refers to the process of automatically extracting key keywords and category information from news articles.

[0009] A "category" refers to a group or genre (e.g., incident, economy, science, sports, etc.) used to classify news articles.

[0010] "Illustrated template" refers to a basic blueprint that provides a visual form or format appropriate for a particular category.

[0011] "Illustrations" refer to graphics or illustrations that visually represent the content of a news article.

[0012] "Illustration Variations" refers to different formats of an illustration (e.g., different languages, theme colors, font sizes).

[0013] "Storage" refers to the process of recording the generated illustrations and their variations in a database or storage.

[0014] "Delivery" refers to the process of transmitting and providing stored illustrations and news articles to a user's device.

[0015] "User terminal" refers to a device (e.g., smartphone, tablet, or PC) used to display news articles and illustrations.

[0016] "Display options" refers to settings that allow users to select and change the display format of illustrations.

[0017] "Analysis" refers to the process of analyzing the content of a news article and extracting important information.

[0018] "Template selection" refers to the act of selecting an appropriate diagram template based on the extracted category information. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] This invention is a system that automatically visualizes news articles. It analyzes the content of news articles and generates and distributes appropriate illustrations, allowing readers to easily and quickly understand the information. To implement this system, it is necessary to execute the following steps: news article acquisition, analysis, illustration generation, variation creation, storage, and distribution. The specific flow of each process is explained below.

[0041] Server (backend) processing

[0042] 1. Retrieving news articles

[0043] The server periodically retrieves the latest news articles via RSS feeds or news APIs, capturing information such as the article title, body of text, publication date, and category.

[0044] 2. Preprocessing of news articles

[0045] Natural language processing (NLP) techniques are used to extract key keywords and phrases from retrieved news articles, and based on the results of this analysis, articles are classified into specific categories (e.g., crime, economy, science, sports, etc.).

[0046] 3. Select a template by category

[0047] Depending on the category, the server will select an appropriate illustration template, for example, a map or timeline template for an incident article.

[0048] 4. Generating illustrations

[0049] The server then creates an illustration of the news article based on the selected template, using map information and time series data to generate a visually easy-to-understand illustration.

[0050] 5. Accessibility-aware variation generation

[0051] Create one or more illustrated variations, such as dark mode, different languages, different text sizes, etc.

[0052] 6. Saving and distributing illustrations

[0053] The generated illustrations and their variations are stored in a database and prepared for distribution to user terminals.

[0054] Terminal (client-side) processing

[0055] 1. Reading the news

[0056] When a user opens the news app, the latest news articles are displayed along with corresponding illustrations. The device caches the data retrieved from the server to provide a smooth display.

[0057] 2. Customizing the illustrations

[0058] From the app's settings screen, users can select the display format of the illustrations (e.g., dark mode, language, font size). When the user changes their settings, the device selects and displays the best version of the illustration from the saved variations.

[0059] 3. Display of illustrations

[0060] The most appropriate illustrations will be displayed based on the user's selection, for example, if dark mode is selected, the illustrations will also be visually displayed in dark mode.

[0061] User interaction and experience

[0062] 1. Viewing news articles and illustrations

[0063] Users can browse the news list in the news app and select an article they are interested in. When they open an article, a corresponding illustration is displayed at the same time, allowing them to visually understand the content.

[0064] 2. Change settings

[0065] Users can change the display options in the settings screen to suit their preferences, such as increasing the font size or changing the display language.

[0066] 3. Multilingual support available

[0067] Users can change the language of articles and illustrations by changing the settings, allowing them to be displayed in different languages, which allows for a wide range of readership.

[0068] As a concrete example, consider the following scenario: When a user browses an "article about an incident" in a news app, the article contains content about an "incident that occurred in Shibuya Ward, Tokyo." The analysis engine extracts the keywords "Shibuya Ward, Tokyo" and "incident" and classifies the category as "incident." The server selects a template for the "incident category" and generates an illustration that combines a map and time-series data. Furthermore, it creates multiple variations (e.g., for dark mode, English version, uppercase version) and stores them in a database.

[0069] When a user opens this news article on their device, they will see the map and timeline illustration. If they select dark mode from the settings screen, the settings will be reflected immediately and the illustration will change to the dark mode version.

[0070] In this way, the present invention displays the contents of news articles in a visually easy-to-understand manner, meeting the needs of a wide variety of users.

[0071] The processing flow will be explained below.

[0072] Server (backend) processing

[0073] Step 1:

[0074] The server periodically queries the RSS feed or news API to retrieve new news articles, along with basic information such as the article title, body of the article, publication date, and category.

[0075] Step 2:

[0076] The server analyzes the retrieved news articles using natural language processing (NLP) technology to extract key keywords and phrases, such as "Shibuya Ward, Tokyo" and "incident."

[0077] Step 3:

[0078] The server automatically classifies news articles into specific categories (incidents, economics, science, sports, etc.) based on the extracted keywords and phrases.

[0079] Step 4:

[0080] The server selects an appropriate illustration template for each category, for example, a map and timeline template for articles in the incident category.

[0081] Step 5:

[0082] The server then creates an illustration of the news article based on the selected template, using map information and time-series data to generate a visually easy-to-understand illustration.

[0083] Step 6:

[0084] For accessibility reasons, the server creates one or more variations of the illustration, including dark mode, different languages ​​(e.g., English, Japanese), different font sizes, etc.

[0085] Step 7:

[0086] The server stores the generated illustrations and their variations in a database and prepares them for distribution to user terminals.

[0087] Terminal (client-side) processing

[0088] Step 1:

[0089] When a user opens the news app, the device calls the server's API to retrieve the latest news articles and corresponding illustrations.

[0090] Step 2:

[0091] The device temporarily caches received news articles and illustrations to provide smooth display.

[0092] Step 3:

[0093] A user scrolls through the news list and taps on an article they are interested in. The article details screen opens, along with a corresponding illustration.

[0094] Step 4:

[0095] The device's settings screen allows the user to select display options (language, theme, text size, etc.).

[0096] Step 5:

[0097] Depending on the user's settings, the device retrieves the optimal illustration variation from the server and changes the display format. For example, if dark mode is selected, the illustration will also be displayed in dark mode.

[0098] User interaction and experience

[0099] Step 1:

[0100] Users scroll through the news list in a news app, find an article that interests them, and tap on it.

[0101] Step 2:

[0102] When you open an article, you'll see details about the news story along with visually compelling illustrations, such as a map showing the location of the incident and a timeline of chronological events.

[0103] Step 3:

[0104] The user accesses the settings screen and changes display options, for example, changing the language from English to Japanese or switching to dark mode.

[0105] Step 4:

[0106] Your preferences are respected and articles and illustrations are displayed in the format of your choice, providing the best possible display based on your visual needs and preferences.

[0107] Example 1

[0108] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0109] Traditional news articles are often presented in text only, which means it takes time for readers to quickly and easily understand the information. Furthermore, the lack of visual information makes it difficult to understand particularly complex news content. Furthermore, the inability to customize the content to meet users' visual preferences and needs poses challenges in terms of accessibility and usability.

[0110] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0111] In this invention, the server includes means for acquiring news articles, means for analyzing the acquired news articles and extracting major keywords and categories, means for selecting an appropriate illustration template based on the extracted category, means for illustrating the content of the news article based on the selected template, means for generating multiple variations with different themes, languages, font sizes, etc., and means for saving the generated illustration variations and distributing them to user terminals. This makes it possible to display the content of news articles in a visually easy-to-understand manner and meet the needs of a variety of users.

[0112] A "news article" is a piece of text that describes the latest information or events distributed by a news organization or information provider.

[0113] A "server" is a central computer system that stores, processes, and distributes data.

[0114] A "terminal" is a client device that is directly operated by a user, and includes smartphones, tablets, PCs, etc.

[0115] An "acquisition means" is a process or device for automatically collecting news articles from external sources.

[0116] "Means for analysis" refers to the processing or device that analyzes the content of the acquired news articles using techniques such as natural language processing and extracts key keywords and categories.

[0117] An "illustrated template" is a pre-designed type or format for visually displaying the content of a news article.

[0118] A "means for visualizing" is a process or device for visualizing the content of a news article based on a selected graphical template.

[0119] "Means for generating variations" refers to a process or device for creating multiple formats from a basic illustration, with different themes, languages, font sizes, etc., according to the user's needs.

[0120] The "storing means" refers to a process or device for permanently storing the generated illustration variations in a database or the like.

[0121] The "distribution means" refers to a process or device for transmitting the saved illustration variations to a user terminal.

[0122] The "displaying means" refers to a process or device for displaying the news article and related illustrations received by the user terminal on the screen.

[0123] "Display options" are settings that the user can select to change the display format of the illustration.

[0124] "Different languages" refers to a format in which illustrated variations are generated in multiple languages, such as Japanese and English.

[0125] A "theme" is a style setting that changes the appearance and color of an illustration.

[0126] "Text size" is a setting item that changes the size of the text in the illustration.

[0127] The present invention is a system that automatically visualizes news articles, in which the server, terminal, and user elements work together to analyze the content of the news article, appropriately illustrate the information, generate variations, and deliver them to the user.

[0128] Server Processing

[0129] The server retrieves the latest news articles using an RSS feed or news API. Specifically, the server receives JSON-formatted data from a news provider using an HTTP request, and extracts information such as the article title, body text, publication date and category from that data. For example, the News API or Google News API may be used.

[0130] The server then applies natural language processing (NLP) techniques to the retrieved news articles. It uses Python libraries like Spacy and NLTK to extract key keywords and phrases from the articles and classify them into categories (e.g., news, economics, science, sports, etc.). This analysis involves techniques such as tokenization, named entity recognition (NER), and Term Frequency-Inverse Document Frequency (TF-IDF).

[0131] Based on the analysis results, the server classifies the news articles into specific categories and selects a corresponding illustration template for that category. This template refers to a type of visual display of the news content, such as a map, timeline, or graph. For example, D3.js or a geographic information system is used to create the map, and a library such as Chart.js is used to display the timeline.

[0132] Based on the selected template, the server creates a visual representation of the news article. For example, to display the geographic location of an incident, map data is retrieved from the OpenStreetMap API and plotted. Timeline data is parsed using the Datetime library as appropriate.

[0133] Additionally, the server generates multiple variations of the generated illustrations to improve their accessibility, such as dark mode, different languages ​​(using the Google Translate API), and different font sizes. Variation generation is implemented using CSS media queries and internationalization (i18n) libraries.

[0134] The generated diagrams and their variations are stored in a database such as MongoDB or MySQL, and then a service such as Firebase Cloud Messaging (FCM) is used to send notifications to user devices.

[0135] Terminal handling

[0136] When a user opens the news app, the device retrieves the latest news articles and illustrations from the server, and the retrieved data is cached in local storage using the Room library and SQLite to provide a smooth display.

[0137] Users can choose how the icons are displayed in the app's settings. Options include dark mode, different languages, and font sizes. User settings are saved in SharedPreferences and UserDefaults. When a setting is changed, the icons are instantly updated to reflect that setting.

[0138] The device will display the best illustration variation based on the user's settings. For example, if the user selects dark mode, the device will apply stylesheets and CSS classes to display dark mode illustrations.

[0139] User interaction and experience

[0140] Users can browse through a list of articles within the news app and select the news article they are interested in. When they tap on an article, a related illustration is displayed at the same time, helping them to understand the content intuitively. For example, if they select a news article about an incident that occurred in Shibuya Ward, Tokyo, an illustration will appear that provides geographical and chronological information about the incident.

[0141] By changing the display options in the settings screen, users can set the display format to suit their preferences. They can also easily increase the font size and change the display language, making it possible to meet a variety of needs.

[0142] Example prompt sentence:

[0143] "Visualize a news story about an incident that occurred in Shibuya Ward, Tokyo, and generate an illustration using maps and time series data. Create multiple variations of the illustration, including a dark mode version, English version, and uppercase version."

[0144] In this way, the present invention can display the contents of news articles in a visually easy-to-understand manner and meet the needs of a wide variety of users.

[0145] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0146] Step 1: Get the news article

[0147] The server retrieves news articles from external sources. Specifically, it periodically retrieves the latest news articles using RSS feeds or news APIs (e.g., NewsAPI, Google News API). This process involves sending an HTTP request and receiving a response in JSON format. The input is the endpoint URL of the news API, and the output is JSON data that includes the article title, body text, publication date and time, category, etc.

[0148] Step 2: Preprocessing the news article

[0149] The server analyzes the retrieved news articles. It uses Python natural language processing libraries (e.g., Spacy, NLTK) to extract key keywords and phrases. The input is the text of the news article obtained in step 1, and the output is the extracted keywords and categories. Specifically, it tokenizes the article text and performs named entity recognition (NER) and TF-IDF (Term Frequency-Inverse Document Frequency).

[0150] Step 3: Select a template by category

[0151] The server classifies the news article into a specific category based on the analysis results and selects the corresponding illustration template. The input is the category extracted in step 2, and the output is the illustration template corresponding to the category. For example, for the incident category, a map or timeline template is selected.

[0152] Step 4: Generate the diagram

[0153] The server visualizes the content of the news article based on the selected template. For example, if geographic information is displayed, the OpenStreetMap API is used to plot the locations of incidents. If time series data is included, it is parsed using the Datetime library to generate a timeline. The input is the news article text and the selected template, and the output is the generated illustration. Specific operations include retrieving map data and generating maps in HTML or SVG format.

[0154] Step 5: Generate accessible variations

[0155] The server generates variations of different themes, languages, and font sizes to improve the accessibility of the generated illustrations. The input is the illustration generated in step 4, and the output is multiple variations of the illustration. Specific behaviors include applying dark mode using CSS media queries and changing the illustration language using the Google Translate API.

[0156] Step 6: Save and distribute your diagrams

[0157] The server stores the generated diagrams and their variations in a database, using, for example, MongoDB or MySQL. It also uses a service such as Firebase Cloud Messaging (FCM) to send notifications to user devices. The input is the diagram variations, and the output is the diagram stored in the database and the notification sent. Specifically, the data is saved using an INSERT query, and notifications are sent using the FCM API.

[0158] Step 7: Read the news

[0159] The user opens the news app to view the latest news articles and illustrations. The device caches and displays the data obtained from the server. The input is the news article and illustration data sent from the server, and the output is the news article and illustration displayed on the device screen. Specifically, the data is obtained via an HTTP request, and the cached data is managed using the Room library and SQLite.

[0160] Step 8: Customize your diagram

[0161] The user can select the display format of the illustrations (e.g., dark mode, language, font size) in the app's settings screen. The device saves the user's selection and reflects it in the display. The input is the user's settings, and the output is the updated display format. Specific behavior is to save the settings in SharedPreferences or UserDefaults and update the UI in real time.

[0162] Step 9: Displaying the diagram

[0163] The optimal illustration is displayed on the device based on the user's selection. For example, if dark mode is selected, the CSS class and style sheet are changed to display an illustration for dark mode. The input is the user's settings and illustration data, and the output is the illustration in the applied display format. Specifically, the HTML and CSS are dynamically updated.

[0164] In this way, the present invention can display the contents of news articles in a visually easy-to-understand manner and meet the needs of a wide variety of users.

[0165] (Application example 1)

[0166] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0167] Conventional news article distribution systems lack the ability to generate illustrations to aid visual understanding, and therefore do not meet the needs of brick-and-mortar stores, especially in environments where real-time information provision is required. Furthermore, the lack of multilingual support and interactive display options makes it difficult to effectively provide information to customers with diverse backgrounds. Given these circumstances, there is a need for a system that can visualize and instantly distribute news articles in brick-and-mortar stores, as well as provide multilingual support and customizable display options.

[0168] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0169] In this invention, the server includes means for acquiring news articles, means for analyzing the acquired news articles and extracting major keywords and categories, means for selecting an appropriate illustration template based on the extracted category, means for illustrating the content of the news article based on the selected template, means for generating one or more illustration variations, means for saving the generated illustration variations and distributing them to a user terminal, means for translating the news article into multiple languages, and means for displaying the generated illustration variations on a display in a physical store. This enables visual display of news articles and multilingual support in physical stores, improving the efficiency of providing information to customers.

[0170] "News Article" means a news article obtained from an internet source.

[0171] "Means of acquisition" refers to methods of automatically downloading news articles from the Internet or news feeds.

[0172] The "analysis method" refers to a technique that uses natural language processing technology to analyze the content of acquired news articles and extract key keywords and phrases.

[0173] "Major keywords and categories" refer to words that are important for understanding the content of a news article and classifications based on them.

[0174] The "selection means" is a method for automatically selecting the most suitable diagram template based on the analysis results.

[0175] An "illustrated template" is a predefined format for visually representing the content of a news article.

[0176] "Visualization tools" are techniques that use selected templates to generate graphs and charts to visually display the content of a news article.

[0177] "Illustration Variations" are illustrations generated in multiple formats, such as different display formats, languages, themes, font sizes, etc.

[0178] The "means for storing and distributing to the user terminal" is a technology for storing the generated illustration variations in a database and transmitting them to the user's terminal via a network.

[0179] A "means for translating news articles into multiple languages" is a translation technique for converting the content of a news article into a different language.

[0180] "Means for displaying on a display in a physical store" refers to a method for displaying the generated illustration on a display installed in a physical store.

[0181] "Means for enabling users to select display options" refers to technology that provides an interface that allows users to select their preferred display format, language, and theme.

[0182] "Visual display" refers to presenting information in a format that is easy to understand visually.

[0183] "Multilingual support" refers to the ability to provide information in multiple languages.

[0184] The present invention is a system for automatically visualizing news articles, providing multilingual support and customizable display options. Specific embodiments are described below.

[0185] Server Processing

[0186] The server retrieves, analyzes, illustrates, and distributes news articles using the following hardware and software:

[0187] Hardware: Server

[0188] Software: News API, Google Translate API, Natural Language Processing (NLP) technology, Matplotlib

[0189] 1. Retrieving news articles

[0190] The server periodically retrieves the latest news articles from the Internet using a news API.

[0191] 2. News article analysis

[0192] NLP technology is used to extract key keywords and categories from retrieved news articles. For example, keywords such as "Shibuya Ward, Tokyo" and "incident" are extracted from an article titled "An incident occurred in Shibuya Ward, Tokyo."

[0193] 3. Select an illustrated template

[0194] Based on the extracted category, an appropriate diagram template is automatically selected. For example, for the category "incidents," a map or timeline template is selected.

[0195] 4. Generating illustrations

[0196] Based on the selected template, we generate illustrations that visually represent the content of the news article, using Matplotlib to create maps and timeline graphs.

[0197] 5. Generating illustrated variations

[0198] Generate one or more illustrative variations, such as dark mode, different languages, and different font sizes.

[0199] 6. News article translation

[0200] Use the Google Translate API to translate news articles into other languages, for example, from English to Japanese, or from Japanese to English.

[0201] 7. Saving and distributing illustrated variations

[0202] The generated illustration variations are stored in a database and prepared for distribution to the user's terminal.

[0203] Terminal handling

[0204] The terminal (a display in a physical store or the user's mobile device) displays the received news article and illustrations, which can be customized according to the user's specifications.

[0205] Hardware: Digital signage displays in physical stores, users' smartphones

[0206] Software: News viewing application

[0207] 1. Reading the news

[0208] When a user opens the news app, the latest news articles are displayed along with corresponding illustrations. The device caches the data retrieved from the server to provide a smooth display.

[0209] 2. Customizing the illustrations

[0210] From the app's settings screen, users can select the display format of the illustrations (e.g., dark mode, language, font size). When the user changes their settings, the device selects and displays the best version of the illustration from the saved variations.

[0211] 3. Display of illustrations

[0212] The most appropriate illustrations will be displayed based on the user's selection, for example, if dark mode is selected, the illustrations will also be visually displayed in dark mode.

[0213] Specific examples

[0214] Showing news articles:

[0215] Title: "The incident that occurred in Shibuya Ward, Tokyo"

[0216] Content: "Police are continuing their investigation into the incident that occurred last night in Shibuya Ward, Tokyo..."

[0217] Translated text: "An incident occurred last night in Shibuya, Tokyo. The police are continuing their investigation..."

[0218] Generated visualization image:

[0219] It will be saved with the file name visual_20230405_093000.png and displayed on the digital signage.

[0220] Prompt Sentence Examples

[0221] Use the News API to retrieve the latest news articles and then use the Google Translate API to translate the content into English. Then use Matplotlib to generate an image that visually displays the translated content. The generated image will then be displayed on a digital signage display in the store. Write a Python program that implements this process.

[0222] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0223] Step 1:

[0224] Get news articles

[0225] The server periodically retrieves the latest news articles from the Internet using a news API. The input is a news API request, and the output is the retrieved news article data. Specifically, the server sends an HTTP request to the news API and receives news article data in JSON format as a response.

[0226] Step 2:

[0227] News article analysis

[0228] The server uses natural language processing (NLP) techniques to extract key keywords and phrases from retrieved news articles. The input is the retrieved news article data, and the output is the extracted keywords and categories. Specifically, the server uses an NLP library (e.g., spaCy) to analyze the content of the article and extract important keywords and categories.

[0229] Step 3:

[0230] Select an illustrated template

[0231] The server selects an appropriate illustration template based on the extracted category. The input is the extracted category, and the output is the selected illustration template. Specifically, the server searches for templates corresponding to the category from the template library and selects the most appropriate template.

[0232] Step 4:

[0233] Generating illustrations

[0234] The server visualizes the contents of the news article based on the selected template. The input is the content of the news article and the selected template, and the output is the generated illustration. Specifically, the server uses a graph drawing library such as Matplotlib to create the illustration according to the template.

[0235] Step 5:

[0236] Generate illustrated variations

[0237] The server generates variations of the generated illustration, such as different themes (e.g., dark mode), different languages, and different font sizes. The input is a basic illustration, and the output is multiple variations of the illustration. Specifically, the server creates multiple variations with different themes, languages, and font sizes according to the set parameters.

[0238] Step 6:

[0239] News article translation

[0240] The server uses the Google Translate API to translate the content of news articles into multiple languages. The input is the content of the news article written in the original language, and the output is the translated content of the news article. Specifically, the server sends a translation request to the Google Translate API and receives the translation result as a response.

[0241] Step 7:

[0242] Saving and distributing illustrated variations

[0243] The server saves the generated illustration variations in a database and prepares them for distribution to user devices. The input is the illustration variation, and the output is the reference information for the saved illustration and a status ready for distribution. Specifically, the server saves the generated illustration variations in a database and generates and stores reference information such as a URL.

[0244] Step 8:

[0245] Reading the news

[0246] The user terminal displays the latest news articles and corresponding illustrations. The input is the news article and illustration data delivered from the server, and the output is the news and illustrations displayed on the terminal's display. In concrete terms, the terminal displays the news article and illustrations on the screen based on the received data.

[0247] Step 9:

[0248] Customizing the illustrations

[0249] The user terminal allows the user to set the display format of the illustration. The input is the user's selection, and the output is a customized illustration display based on the selection. In concrete terms, the terminal receives the display setting change from the user and selects and displays the optimal illustration variation from the saved variations.

[0250] Step 10:

[0251] Viewing the illustration

[0252] The user terminal displays the illustration based on the selected display format. The input is customized illustration data, and the output is the illustration displayed on the terminal display. In specific operation, the terminal selects and displays an appropriate illustration variation according to the user's settings.

[0253] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0254] This invention combines an emotion engine with a system that acquires, analyzes, and visualizes news articles, enabling the system to recognize user emotions and dynamically customize the display of illustrations based on those emotions. To implement this system, it is necessary to execute the following steps: news article acquisition, analysis, illustration generation, variation creation, emotion recognition, customization, storage, and distribution. The specific flow of each process is explained below.

[0255] Server (backend) processing

[0256] 1. Retrieving news articles

[0257] The server periodically retrieves the latest news articles via RSS feeds or news APIs, along with basic information such as article title, body of text, publication date, and category.

[0258] 2. Preprocessing of news articles

[0259] The server analyzes the retrieved news articles using natural language processing (NLP) technology to extract key keywords and phrases, such as "Shibuya Ward, Tokyo" and "incident."

[0260] 3. Select a template by category

[0261] Based on the extracted keywords and phrases, the server automatically classifies the news articles into specific categories (e.g., incidents, economics, science, sports, etc.) and selects an appropriate illustration template accordingly.

[0262] 4. Generating illustrations

[0263] The server then creates an illustration of the news article based on the selected template, using map information and time-series data to generate a visually easy-to-understand illustration.

[0264] 5. Accessibility-aware variation generation

[0265] The server creates one or more variations of the illustration, such as dark mode, different languages ​​(e.g., English, Japanese), different text sizes, etc.

[0266] 6. Emotion Recognition by Emotion Engine

[0267] The server uses interaction data (camera, microphone, touch operations, etc.) from the user's device to allow the emotion engine to estimate the user's current emotional state (e.g., joy, sadness, surprise, anger, etc.).

[0268] 7. Emotion-based illustrated customization

[0269] The server selects the optimal illustration variation based on the user's emotional data obtained from the emotion engine and dynamically changes the illustration display format. For example, if the user feels sad, it will provide an illustration with a softer color scheme.

[0270] 8. Saving and distributing illustrations

[0271] The server stores the illustration variations selected based on emotion recognition in a database and delivers them to the user's terminal.

[0272] Terminal (client-side) processing

[0273] 1. Reading the news

[0274] When a user opens the news app, the device calls the server's API to retrieve the latest news articles and corresponding illustrations.

[0275] 2. Initial display of the diagram

[0276] The device temporarily caches the received news article and illustrations, and when the news article is opened, the corresponding illustration is initially displayed.

[0277] 3. Acquiring Emotion Data

[0278] The device collects user interaction data (camera footage, voice input, touch operations, etc.) and sends it to the emotion engine, which analyzes it to recognize the user's emotions.

[0279] 4. Customize display options

[0280] Based on the emotion recognition results, the device dynamically adjusts the display options of the illustrations. Users can also manually select display options (e.g., language, theme, font size) from the settings screen.

[0281] User interaction and experience

[0282] 1. Viewing news articles and illustrations

[0283] Users scroll through the list in the news app and select the article they are interested in. When they open the article, the corresponding illustration is displayed along with the details of the news article.

[0284] 2. Emotion-based change

[0285] While the user is browsing the article, the device recognizes the user's emotions in real time and dynamically changes the illustration display style based on the user's emotions. For example, if the user is feeling stressed, the illustration's color scheme will become softer.

[0286] 3. Change settings

[0287] Users can change the display options in the settings screen to suit their preferences, for example, switching from Japanese to English or increasing the font size.

[0288] 4. Use of Emotion History

[0289] The system records the user's emotional history and can provide more customized displays in the future based on the user's emotional patterns.

[0290] As a concrete example, consider the following scenario: When a user browses an "article about an incident" in a news app, the article contains content about an "incident that occurred in Shibuya Ward, Tokyo." The analysis engine extracts the keywords "Shibuya Ward, Tokyo" and "incident" and classifies the category as "incident." The server selects a template for the "incident category" and generates an illustration that combines map and timeline data. It then creates and saves multiple variations (e.g., for dark mode, English version, uppercase version).

[0291] When a user opens this news article on their device, a map and timeline illustration are displayed. At the same time, an emotion engine analyzes the user's facial expression and, if the user expresses sadness, the illustration's color scheme is softened.

[0292] In this way, the present invention not only displays the contents of news articles in a visually easy-to-understand manner, but also dynamically customizes the content according to the user's emotions, thereby enabling more personalized information to be provided.

[0293] The processing flow will be explained below.

[0294] Server (backend) processing

[0295] Step 1:

[0296] The server periodically queries an RSS feed or news API to retrieve new news articles, including basic information such as the article title, body of the article, publication date, and category.

[0297] Step 2:

[0298] The server analyzes the retrieved news articles using natural language processing (NLP) technology to extract key keywords and phrases, such as "Shibuya Ward, Tokyo" and "incident."

[0299] Step 3:

[0300] Based on the extracted keywords and phrases, the server automatically classifies news articles into specific categories (incidents, economics, science, sports, etc.).

[0301] Step 4:

[0302] The server selects an appropriate illustration template for each category, for example, a map or timeline template for an article in the incident category.

[0303] Step 5:

[0304] The server then creates an illustration of the news article based on the selected template, using map information and time-series data to generate a visually easy-to-understand illustration.

[0305] Step 6:

[0306] The server creates one or more illustrated variations for accessibility, including dark mode, different languages ​​(e.g., English, Japanese), different text sizes, etc.

[0307] Step 7:

[0308] The server uses interaction data (camera, microphone, touch operations, etc.) from the user's device to allow the emotion engine to estimate the user's current emotional state (joy, sadness, surprise, anger, etc.).

[0309] Step 8:

[0310] Based on the user's emotional data obtained from the emotion engine, the server selects the optimal illustration variation and dynamically changes the illustration display format. For example, if the user is feeling sad, it will provide an illustration with a softer color scheme.

[0311] Step 9:

[0312] The server stores the generated illustrations and their variations in a database and prepares them for distribution to user terminals.

[0313] Terminal (client-side) processing

[0314] Step 1:

[0315] When a user opens the news app, the device calls the server's API to retrieve the latest news articles and corresponding illustrations.

[0316] Step 2:

[0317] The device temporarily caches received news articles and illustrations to provide smooth display.

[0318] Step 3:

[0319] A user scrolls through the news list and taps on an article they are interested in. The article details screen opens, along with a corresponding illustration.

[0320] Step 4:

[0321] The device collects user interaction data (camera footage, voice input, touch operations, etc.) and sends it to the emotion engine, which analyzes it to recognize the user's emotions.

[0322] Step 5:

[0323] Based on the emotion recognition results, the device dynamically adjusts the display options of the illustrations. Users can also manually select display options (e.g., language, theme, font size) from the settings screen.

[0324] User interaction and experience

[0325] Step 1:

[0326] Users scroll through the news list in a news app, find an article that interests them, and tap on it.

[0327] Step 2:

[0328] When you open an article, you'll see details about the news story along with visually compelling illustrations, such as a map showing the location of the incident and a timeline of chronological events.

[0329] Step 3:

[0330] While the user is browsing the article, the device recognizes the user's emotions in real time and dynamically changes the illustration display style based on the user's emotions. For example, if the user is feeling stressed, the illustration's color scheme will become softer.

[0331] Step 4:

[0332] Users can change the display options in the settings screen to suit their preferences, for example, switching from Japanese to English or increasing the font size.

[0333] Step 5:

[0334] The system records the user's emotional history and can provide more customized displays in the future based on the user's emotional patterns.

[0335] As a concrete example, consider the following scenario: When a user browses an "article about an incident" in a news app, the article contains content about an "incident that occurred in Shibuya Ward, Tokyo." The analysis engine extracts the keywords "Shibuya Ward, Tokyo" and "incident" and classifies the category as "incident." The server selects a template for the "incident category" and generates an illustration that combines map and timeline data. It then creates and saves multiple variations (e.g., for dark mode, English version, uppercase version).

[0336] When a user opens this news article on their device, a map and timeline illustration are displayed. At the same time, an emotion engine analyzes the user's facial expression and, if the user expresses sadness, the illustration's color scheme is softened.

[0337] In this way, the present invention not only displays the contents of news articles in a visually easy-to-understand manner, but also dynamically customizes the content according to the user's emotions, thereby enabling more personalized information to be provided.

[0338] Example 2

[0339] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0340] Conventional news article delivery systems have had difficulty providing information tailored to individual users' emotional states. Furthermore, they were unable to dynamically customize the content of news articles to match the user's emotional state. As a result, the user experience was static, potentially leaving some users dissatisfied.

[0341] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring news articles, means for analyzing the acquired news articles using natural language processing technology and extracting major keywords and categories, means for selecting an appropriate illustration template based on the extracted keywords and categories, means for generating an illustration that visualizes the content of the news article based on the selected template, means for creating one or more illustration variations taking accessibility into consideration, means for collecting user interaction data and recognizing emotions, means for dynamically customizing the illustration variations based on the recognized emotions, and means for saving the generated illustration variations and delivering them to the user terminal. This dynamically customizes the visual display of the news article, enabling more personalized information to be provided based on the user's emotions.

[0342] "Means for obtaining news articles" refers to the function of periodically collecting news articles using a news API or RSS feed.

[0343] "Means of analyzing acquired news articles using natural language processing technology and extracting key keywords and categories" refers to a function that analyzes acquired news articles using natural language processing technology such as morphological analysis and automatically extracts key keywords and categories.

[0344] "Means for selecting an appropriate illustrated template based on extracted keywords and categories" is a function that automatically selects the most appropriate template from multiple pre-prepared illustrated templates based on analyzed keyword and category information.

[0345] "Means for generating illustrations that visualize the contents of a news article based on a selected template" refers to a function that uses a selected illustration template to generate illustrations that represent the contents of a news article in a visually easy-to-understand format, such as a map or timeline.

[0346] "Means for creating one or more variations of illustrations with accessibility in mind" refers to the ability to create variations of the generated illustrations in different formats (e.g., dark mode, different languages, different font sizes) that take into account user visibility and ease of use.

[0347] "Means for collecting user interaction data and recognizing emotions" refers to a function that collects data such as the user's camera footage, voice input, and touch operations, and analyzes it to estimate the user's current emotional state.

[0348] "Means for dynamically customizing illustration variations based on recognized emotions" is a function that selects the optimal illustration variation based on the user's emotional data and changes the color scheme and design of the illustration in real time.

[0349] The "means for saving the generated illustration variations and distributing them to the user's terminal" is a function for saving the generated multiple illustration variations in a database and distributing them to the user's terminal via the Internet.

[0350] MODE FOR CARRYING OUT THE INVENTION

[0351] This invention aims to provide a system for acquiring, analyzing, and illustrating news articles, and to provide a dynamically customized visual display according to the user's emotions. This system uses the following hardware and software:

[0352] Hardware

[0353] Server: Responsible for news article acquisition, analysis, illustration generation, emotion recognition, customization, storage and distribution.

[0354] User devices: Responsible for displaying news articles, collecting and transmitting emotion data, and dynamically changing the display. These devices include smartphones, tablets, and PCs.

[0355] software

[0356] Natural Language Processing (NLP) software: Use libraries such as SpaCy and NLTK to analyze news articles and extract key keywords and categories.

[0357] News API: Uses an API such as Google News API to make an HTTP request to retrieve the latest news articles.

[0358] Illustration generation library: Uses D3.js, Google Maps API, etc. to create visually easy-to-understand illustrations of the contents of news articles.

[0359] Emotion recognition engine: Analyzes user emotions using Microsoft Azure Emotion API, etc.

[0360] Example

[0361] 1. Acquiring and analyzing news articles

[0362] The server periodically retrieves news articles using the Google News API or RSS feeds. The retrieved articles are analyzed using natural language processing technology to extract key keywords and categories. For example, a request like https: / / newsapi.org / v2 / top-headlines?country=jp&apiKey=YOUR_API_KEY is sent.

[0363] 2. Template selection and diagram generation

[0364] Based on the extracted keywords and categories, the server selects an appropriate illustration template. Based on the selected template, it generates illustrations such as maps and timeline data using the Google Maps API and D3.js. For example, if the article is about an incident that occurred in Shibuya Ward, Tokyo, it selects a template in the incident category and illustrates the map and timeline data.

[0365] 3. Accessible Variation Generation

[0366] The server creates multiple variations of the generated illustrations, such as dark mode, different languages ​​(English, Japanese), different font sizes, etc. For example, it applies dark mode styles with CSS and adds different language labels to the illustration data.

[0367] 4. Emotional Data Collection and Recognition

[0368] While the user is browsing a news article, the device collects interaction data from the camera and microphone and sends it to the server. The server's emotion recognition engine analyzes the user's emotional state. For example, the user's emotional state can be recognized as "happiness," "sadness," "surprise," "anger," etc. based on facial expressions and voice analysis.

[0369] 5. Dynamic customization based on emotions

[0370] The server selects the most appropriate illustration variation based on the user's recognized emotion and changes the illustration's color scheme and design in real time. For example, if the user feels sad, it selects an illustration with a soft color scheme.

[0371] 6. Saving and distributing illustrations

[0372] The selected illustration variations are stored in a database and delivered to the user's device via the Internet, using a real-time database such as Firebase to transmit data in real time.

[0373] Specific examples and prompts for the generative AI model

[0374] As a specific example, consider a scenario in which a user views an article about an incident that occurred in Shibuya Ward, Tokyo.

[0375] When a user opens the news app, an article about an incident that occurred in Shibuya Ward, Tokyo, is retrieved from the server. The analysis engine extracts the keywords "Shibuya Ward, Tokyo" and "incident" from the article and classifies it as an "incident" category. The server selects a template that combines a map and a timeline to visualize the news content. The generated variations (e.g., dark mode, English version, uppercase version) are stored in a database.

[0376] When a user opens an article on their device, a map and timeline illustration are displayed. At the same time, the emotion engine analyzes the user's facial expressions and, if the user is expressing sadness, the illustration's color scheme is changed to a softer tone.

[0377] Example prompts to input to a generative AI model:

[0378] A user is viewing an article about an incident that occurred in Shibuya Ward, Tokyo. The article's category is "Incident," and an illustration using map and timeline data should be displayed. If the user's emotion is "Sadness," change the illustration's color scheme to a softer tone.

[0379] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0380] Step 1: Get the news article

[0381] Input: Periodically send requests for news articles using a news API or RSS feed.

[0382] Data processing: The server sends an HTTP request to the API endpoint and receives the returned data in JSON format.

[0383] Output: Data including news article title, body of text, publication date, category, etc.

[0384] Specific operation: Execute a request such as https: / / newsapi.org / v2 / top-headlines?country=jp&apiKey=YOUR_API_KEY.

[0385] Step 2: Preprocessing the news article

[0386] Input: News article data obtained in step 1.

[0387] Data processing: The server analyzes the news articles using natural language processing techniques (e.g., SpaCy, NLTK) to extract key keywords and phrases.

[0388] Output: Extracted keywords and phrases (e.g., "Shibuya Ward, Tokyo" and "incident").

[0389] Specific operation: Performs morphological analysis and extracts important words and phrases in the sentence.

[0390] Step 3: Select a template by category

[0391] Input: Keywords or phrases extracted in step 2.

[0392] Data processing: The server classifies news articles into specific categories based on extracted keywords and phrases, and selects appropriate illustration templates for each category.

[0393] Output: The selected diagram template (e.g., a template for incident categories).

[0394] Specific operation: Based on the analysis results, categorize them into categories such as "incidents," "economy," and "science," and select the corresponding template for each.

[0395] Step 4: Generate the diagram

[0396] Input: The illustrated template selected in step 3 and the content of the news article.

[0397] Data processing: The server generates illustrations that visualize the contents of news articles based on the selected template. Maps and timeline data are created using Google Maps API, D3.js, etc.

[0398] Output: Visualized and illustrated data.

[0399] Specific functions: Display the locations of incidents on a map and create a chronological timeline.

[0400] Step 5: Generate variations with accessibility in mind

[0401] Input: Illustrated data generated in step 4.

[0402] Data processing: The server creates variations of the generated illustrations in different formats (e.g., dark mode, different languages, changed font size).

[0403] Output: Multiple illustrated variations.

[0404] What it does: Changes CSS to apply dark mode styles and adds labels in different languages ​​to illustrated data.

[0405] Step 6: Emotion data collection and emotion recognition

[0406] Input: Interaction data (camera feeds, voice inputs, touch actions, etc.) while the user is viewing an article.

[0407] Data processing: The interaction data collected by the device is sent to the server, where the server's emotion recognition engine analyzes the emotional state.

[0408] Output: User emotion data (e.g., happy, sad, surprised, angry).

[0409] Specific operation: The user's facial expressions and voice are collected through a camera and microphone and analyzed using an emotion recognition model.

[0410] Step 7: Emotion-based illustration customization

[0411] Input: User emotion data recognized in step 6.

[0412] Data processing: The server selects the optimal illustration variation based on the emotion data and dynamically changes the color scheme and design of the illustration.

[0413] Output: Customized illustrated data.

[0414] Specific behavior: If the user expresses sadness, change the color scheme of the illustration to a softer tone.

[0415] Step 8: Save and distribute your diagrams

[0416] Input: Customized diagram data from step 7.

[0417] Data processing: The server stores the customized illustrations in a database and delivers them to the user's device.

[0418] Output: Illustrated data stored on the user's device.

[0419] What it does: Uses a real-time database such as Firebase to send customized diagrams over the internet.

[0420] Step 9: Initial display of news article and illustration

[0421] Input: News article and illustration data distributed in step 8.

[0422] Data processing: The device caches the news article and illustrations it receives, and initially displays the illustrations when the news article is opened.

[0423] Output: News article and illustration displayed on the user's terminal.

[0424] What it does: When you open the news app, it displays the latest news articles and their illustrations.

[0425] Step 10: Customize display options

[0426] Input: User settings changes (language, theme, text size, etc.).

[0427] Data processing: The device dynamically adjusts the display options of the illustration based on user settings changes.

[0428] Output: Customized graphical display.

[0429] Specific operation: The user changes the display options in the settings screen to display the diagram according to their preference.

[0430] (Application example 2)

[0431] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0432] Conventional advertising delivery systems have difficulty customizing ads based on user emotions, limiting their visibility and effectiveness. Furthermore, there is no technology for summarizing news-like information and incorporating it into ads, making it difficult for ads to be interesting to users. Therefore, there is a need for a system that can provide more personalized information by recognizing user emotions in real time and dynamically changing advertising design and content based on those emotions.

[0433] The specification processing by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring a news article, means for analyzing the news article and extracting main keywords and categories, means for selecting an appropriate illustration template based on the extracted category, means for illustrating the content of the news article based on the selected template, means for generating one or more illustration variations, means for saving the generated illustration variations and delivering them to the user terminal, means for recognizing the user's emotions and dynamically changing the display format of the illustration based on the emotions, means for summarizing the news article and incorporating it into the generated advertising content and displaying it, and means for dynamically changing the advertising design based on the user's emotions. This enables dynamic advertising delivery linked to the user's emotions, making it possible to dramatically improve the visibility and effectiveness of advertising.

[0434] A "news article" is a set of documents or information that is reported as news.

[0435] An "advertising distribution system" is a system for distributing advertisements for products and services to users.

[0436] "Emotion recognition" is a technology that analyzes and recognizes a user's emotions from interaction data such as camera and voice.

[0437] An "illustration" is a graphical representation used to make textual information easier to understand visually.

[0438] A "template" is a standard format that serves as the basis for generating diagrams and displaying them.

[0439] "Illustration Variants" are different versions of an illustration that are generated in multiple formats, such as different languages, themes, font sizes, etc.

[0440] "Dynamic change" means making changes automatically in real time according to the situation or conditions.

[0441] "User terminal" refers to a computer device used by a user, such as a smartphone, tablet, or PC.

[0442] "Customization" means changing settings according to the needs and preferences of a particular user.

[0443] A "generative AI model" is an artificial intelligence model that learns from large amounts of data and generates useful output for new data.

[0444] A "prompt sentence" is an instruction sentence that is input to a generative AI model to obtain a specific output.

[0445] The present invention is a dynamic advertising delivery system based on user emotions, and provides technology for customizing advertising content through the acquisition, analysis, and visualization of news articles.

[0446] Server (backend) processing

[0447] The server periodically retrieves news articles through a news API. The retrieved news articles are analyzed using natural language processing (NLP) techniques to extract key keywords and categories. An appropriate illustration template is then selected based on the extracted categories, and the news article is visualized based on the selected template. One or more illustration variations are generated, which may include different languages, themes, and font sizes.

[0448] Next, the emotion engine recognizes the user's emotions. This recognition is performed in real time from interaction information collected using a camera and microphone. Based on the emotion recognition, the display format of the illustrations is dynamically changed, and a summary of the news article is incorporated into the generated advertisement content. The generated advertisement content is then delivered to the user's device in an optimized design. The following specific technologies are used in this series of processes:

[0449] Natural Language Processing (NLP) technology: Keyword analysis of retrieved news articles

[0450] Emotion Recognition Engine: Analyzes user emotions

[0451] Illustration generation engine: Generate illustrations based on keywords

[0452] Database: storing generated illustrations and ad variations

[0453] User terminal (client side) processing

[0454] The user's device displays the news article and corresponding illustrations received from the server. Web technologies such as HTML and CSS are used for the display, allowing for dynamic customization. In addition, the user's device captures the user's emotional data through a camera and microphone and sends it to the server. Based on feedback from the server, the design and color scheme of the advertisement are changed in real time.

[0455] User interaction and experience

[0456] When a user opens the ad display app, they are shown ads that summarize news-like information. For example, an ad for a new smartphone product summarizes and visualizes the latest trends in the smartphone industry. As the user views the ad, the design and color scheme of the ad dynamically changes depending on the user's emotions, providing a more personalized advertising experience.

[0457] Specific examples of prompts are as follows:

[0458] "Summarize the latest smartphone industry trending news and create a visually compelling illustration. If the user is expressing joy, use a brighter color scheme to create a positive impression."

[0459] Using this prompt, the generative AI model generates advertising content, resulting in advertising designs that adapt to the user's emotions.

[0460] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0461] Step 1:

[0462] The server retrieves the latest news articles using the news API. In this step, an API request is sent and JSON-formatted data is received, including basic information such as the article title, body, publication date and time, and category. The input is the request to the news API, and the output is the JSON data of the retrieved news articles.

[0463] Step 2:

[0464] The server analyzes the retrieved news articles and extracts key keywords and categories. In this step, natural language processing (NLP) techniques are used to analyze the article text and extract important keywords and phrases. The input is the text data of the news article, and the output is the extracted keyword and category data.

[0465] Step 3:

[0466] The server selects an appropriate illustration template based on the extracted category. In this step, it selects the illustration template that best suits the extracted category from a predefined template library. The input is the extracted category data, and the output is the selected illustration template.

[0467] Step 4:

[0468] The server then creates an illustration of the news article content based on the selected template. In this step, the text and keywords are converted into a visually easy-to-understand illustration. Specifically, the keywords are arranged in graphs, charts, maps, etc. The input is the news article text and keyword data, and the output is the generated illustration.

[0469] Step 5:

[0470] The server generates one or more variations of the illustration. In this step, the illustration is generated in multiple formats, such as different languages, themes, font sizes, etc. The input is the generated illustration, and the output is each variation of the illustration.

[0471] Step 6:

[0472] The server saves the generated illustration variations and delivers them to the user terminal. In this step, the illustration variations are saved in a database and a link is generated for users to access them. The input is the generated illustration variations, and the output is the link information for the saved illustrations.

[0473] Step 7:

[0474] The terminal displays the received news article and related illustrations. In this step, the news article and illustrations retrieved from the server are displayed on the screen using HTML and CSS. The input is the news article and illustration received from the server, and the output is a display screen that the user can view.

[0475] Step 8:

[0476] The device acquires the user's emotional data through a camera and microphone and sends it to the server. In this step, the device analyzes the user's facial expressions and tone of voice to recognize their emotions. The input is camera video and audio data, and the output is the recognized emotional data.

[0477] Step 9:

[0478] The server uses an emotion engine to recognize the user's emotion and dynamically change the display format of the illustration based on the emotion. In this step, the color scheme and design of the illustration are changed based on the recognized emotion data. The input is the user's emotion data, and the output is a dynamically changed illustration.

[0479] Step 10:

[0480] The server then summarises the news article and incorporates it into the generated advertisement content, and displays it. Specifically, it incorporates the news article summary into the advertisement in a design that adapts to the user's emotions. The input for this step is the news article summary and an emotion-based design template, and the output is optimized advertisement content.

[0481] Step 11:

[0482] The device dynamically changes the ad design based on the user's emotions. In this step, the color scheme and layout of the ad are adjusted in real time to match the user's emotions. The input is the optimized ad data from the server, and the output is the final ad display.

[0483] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0484] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0485] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0486] [Second embodiment]

[0487] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0488] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0489] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0490] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0491] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0492] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0493] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0494] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0495] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0496] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0497] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0498] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0499] This invention is a system that automatically visualizes news articles. It analyzes the content of news articles and generates and distributes appropriate illustrations, allowing readers to easily and quickly understand the information. To implement this system, it is necessary to execute the following steps: news article acquisition, analysis, illustration generation, variation creation, storage, and distribution. The specific flow of each process is explained below.

[0500] Server (backend) processing

[0501] 1. Retrieving news articles

[0502] The server periodically retrieves the latest news articles via RSS feeds or news APIs, capturing information such as the article title, body of text, publication date, and category.

[0503] 2. Preprocessing of news articles

[0504] Natural language processing (NLP) techniques are used to extract key keywords and phrases from retrieved news articles, and based on the results of this analysis, articles are classified into specific categories (e.g., crime, economy, science, sports, etc.).

[0505] 3. Select a template by category

[0506] Depending on the category, the server will select an appropriate illustration template, for example, a map or timeline template for an incident article.

[0507] 4. Generating illustrations

[0508] The server then creates an illustration of the news article based on the selected template, using map information and time series data to generate a visually easy-to-understand illustration.

[0509] 5. Accessibility-aware variation generation

[0510] Create one or more illustrated variations, such as dark mode, different languages, different text sizes, etc.

[0511] 6. Saving and distributing illustrations

[0512] The generated illustrations and their variations are stored in a database and prepared for distribution to user terminals.

[0513] Terminal (client-side) processing

[0514] 1. Reading the news

[0515] When a user opens the news app, the latest news articles are displayed along with corresponding illustrations. The device caches the data retrieved from the server to provide a smooth display.

[0516] 2. Customizing the illustrations

[0517] From the app's settings screen, users can select the display format of the illustrations (e.g., dark mode, language, font size). When the user changes their settings, the device selects and displays the best version of the illustration from the saved variations.

[0518] 3. Display of illustrations

[0519] The most appropriate illustrations will be displayed based on the user's selection, for example, if dark mode is selected, the illustrations will also be visually displayed in dark mode.

[0520] User interaction and experience

[0521] 1. Viewing news articles and illustrations

[0522] Users can browse the news list in the news app and select an article they are interested in. When they open an article, a corresponding illustration is displayed at the same time, allowing them to visually understand the content.

[0523] 2. Change settings

[0524] Users can change the display options in the settings screen to suit their preferences, such as increasing the font size or changing the display language.

[0525] 3. Multilingual support available

[0526] Users can change the language of articles and illustrations by changing the settings, allowing them to be displayed in different languages, which allows for a wide range of readership.

[0527] As a concrete example, consider the following scenario: When a user browses an "article about an incident" in a news app, the article contains content about an "incident that occurred in Shibuya Ward, Tokyo." The analysis engine extracts the keywords "Shibuya Ward, Tokyo" and "incident" and classifies the category as "incident." The server selects a template for the "incident category" and generates an illustration that combines a map and time-series data. Furthermore, it creates multiple variations (e.g., for dark mode, English version, uppercase version) and stores them in a database.

[0528] When a user opens this news article on their device, they will see the map and timeline illustration. If they select dark mode from the settings screen, the settings will be reflected immediately and the illustration will change to the dark mode version.

[0529] In this way, the present invention displays the contents of news articles in a visually easy-to-understand manner, meeting the needs of a wide variety of users.

[0530] The processing flow will be explained below.

[0531] Server (backend) processing

[0532] Step 1:

[0533] The server periodically queries the RSS feed or news API to retrieve new news articles, along with basic information such as the article title, body of the article, publication date, and category.

[0534] Step 2:

[0535] The server analyzes the retrieved news articles using natural language processing (NLP) technology to extract key keywords and phrases, such as "Shibuya Ward, Tokyo" and "incident."

[0536] Step 3:

[0537] The server automatically classifies news articles into specific categories (incidents, economics, science, sports, etc.) based on the extracted keywords and phrases.

[0538] Step 4:

[0539] The server selects an appropriate illustration template for each category, for example, a map and timeline template for articles in the incident category.

[0540] Step 5:

[0541] The server then creates an illustration of the news article based on the selected template, using map information and time-series data to generate a visually easy-to-understand illustration.

[0542] Step 6:

[0543] For accessibility reasons, the server creates one or more variations of the illustration, including dark mode, different languages ​​(e.g., English, Japanese), different font sizes, etc.

[0544] Step 7:

[0545] The server stores the generated illustrations and their variations in a database and prepares them for distribution to user terminals.

[0546] Terminal (client-side) processing

[0547] Step 1:

[0548] When a user opens the news app, the device calls the server's API to retrieve the latest news articles and corresponding illustrations.

[0549] Step 2:

[0550] The device temporarily caches received news articles and illustrations to provide smooth display.

[0551] Step 3:

[0552] A user scrolls through the news list and taps on an article they are interested in. The article details screen opens, along with a corresponding illustration.

[0553] Step 4:

[0554] The device's settings screen allows the user to select display options (language, theme, text size, etc.).

[0555] Step 5:

[0556] Depending on the user's settings, the device retrieves the optimal illustration variation from the server and changes the display format. For example, if dark mode is selected, the illustration will also be displayed in dark mode.

[0557] User interaction and experience

[0558] Step 1:

[0559] Users scroll through the news list in a news app, find an article that interests them, and tap on it.

[0560] Step 2:

[0561] When you open an article, you'll see details about the news story along with visually compelling illustrations, such as a map showing the location of the incident and a timeline of chronological events.

[0562] Step 3:

[0563] The user accesses the settings screen and changes display options, for example, changing the language from English to Japanese or switching to dark mode.

[0564] Step 4:

[0565] Your preferences are respected and articles and illustrations are displayed in the format of your choice, providing the best possible display based on your visual needs and preferences.

[0566] Example 1

[0567] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0568] Traditional news articles are often presented in text only, which means it takes time for readers to quickly and easily understand the information. Furthermore, the lack of visual information makes it difficult to understand particularly complex news content. Furthermore, the inability to customize the content to meet users' visual preferences and needs poses challenges in terms of accessibility and usability.

[0569] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0570] In this invention, the server includes means for acquiring news articles, means for analyzing the acquired news articles and extracting major keywords and categories, means for selecting an appropriate illustration template based on the extracted category, means for illustrating the content of the news article based on the selected template, means for generating multiple variations with different themes, languages, font sizes, etc., and means for saving the generated illustration variations and distributing them to user terminals. This makes it possible to display the content of news articles in a visually easy-to-understand manner and meet the needs of a variety of users.

[0571] A "news article" is a piece of text that describes the latest information or events distributed by a news organization or information provider.

[0572] A "server" is a central computer system that stores, processes, and distributes data.

[0573] A "terminal" is a client device that is directly operated by a user, and includes smartphones, tablets, PCs, etc.

[0574] An "acquisition means" is a process or device for automatically collecting news articles from external sources.

[0575] "Means for analysis" refers to the processing or device that analyzes the content of the acquired news articles using techniques such as natural language processing and extracts key keywords and categories.

[0576] An "illustrated template" is a pre-designed type or format for visually displaying the content of a news article.

[0577] A "means for visualizing" is a process or device for visualizing the content of a news article based on a selected graphical template.

[0578] "Means for generating variations" refers to a process or device for creating multiple formats from a basic illustration, with different themes, languages, font sizes, etc., according to the user's needs.

[0579] The "storing means" refers to a process or device for permanently storing the generated illustration variations in a database or the like.

[0580] The "distribution means" refers to a process or device for transmitting the saved illustration variations to a user terminal.

[0581] The "displaying means" refers to a process or device for displaying the news article and related illustrations received by the user terminal on the screen.

[0582] "Display options" are settings that the user can select to change the display format of the illustration.

[0583] "Different languages" refers to a format in which illustrated variations are generated in multiple languages, such as Japanese and English.

[0584] A "theme" is a style setting that changes the appearance and color of an illustration.

[0585] "Text size" is a setting item that changes the size of the text in the illustration.

[0586] The present invention is a system that automatically visualizes news articles, in which the server, terminal, and user elements work together to analyze the content of the news article, appropriately illustrate the information, generate variations, and deliver them to the user.

[0587] Server Processing

[0588] The server retrieves the latest news articles using an RSS feed or news API. Specifically, the server receives JSON-formatted data from a news provider using an HTTP request, and extracts information such as the article title, body text, publication date and category from that data. For example, the News API or Google News API may be used.

[0589] The server then applies natural language processing (NLP) techniques to the retrieved news articles. It uses Python libraries like Spacy and NLTK to extract key keywords and phrases from the articles and classify them into categories (e.g., news, economics, science, sports, etc.). This analysis involves techniques such as tokenization, named entity recognition (NER), and Term Frequency-Inverse Document Frequency (TF-IDF).

[0590] Based on the analysis results, the server classifies the news articles into specific categories and selects a corresponding illustration template for that category. This template refers to a type of visual display of the news content, such as a map, timeline, or graph. For example, D3.js or a geographic information system is used to create the map, and a library such as Chart.js is used to display the timeline.

[0591] Based on the selected template, the server creates a visual representation of the news article. For example, to display the geographic location of an incident, map data is retrieved from the OpenStreetMap API and plotted. Timeline data is parsed using the Datetime library as appropriate.

[0592] Additionally, the server generates multiple variations of the generated illustrations to improve their accessibility, such as dark mode, different languages ​​(using the Google Translate API), and different font sizes. Variation generation is implemented using CSS media queries and internationalization (i18n) libraries.

[0593] The generated diagrams and their variations are stored in a database such as MongoDB or MySQL, and then a service such as Firebase Cloud Messaging (FCM) is used to send notifications to user devices.

[0594] Terminal handling

[0595] When a user opens the news app, the device retrieves the latest news articles and illustrations from the server, and the retrieved data is cached in local storage using the Room library and SQLite to provide a smooth display.

[0596] Users can choose how the icons are displayed in the app's settings. Options include dark mode, different languages, and font sizes. User settings are saved in SharedPreferences and UserDefaults. When a setting is changed, the icons are instantly updated to reflect that setting.

[0597] The device will display the best illustration variation based on the user's settings. For example, if the user selects dark mode, the device will apply stylesheets and CSS classes to display dark mode illustrations.

[0598] User interaction and experience

[0599] Users can browse through a list of articles within the news app and select the news article they are interested in. When they tap on an article, a related illustration is displayed at the same time, helping them to understand the content intuitively. For example, if they select a news article about an incident that occurred in Shibuya Ward, Tokyo, an illustration will appear that provides geographical and chronological information about the incident.

[0600] By changing the display options in the settings screen, users can set the display format to suit their preferences. They can also easily increase the font size and change the display language, making it possible to meet a variety of needs.

[0601] Example prompt sentence:

[0602] "Visualize a news story about an incident that occurred in Shibuya Ward, Tokyo, and generate an illustration using maps and time series data. Create multiple variations of the illustration, including a dark mode version, English version, and uppercase version."

[0603] In this way, the present invention can display the contents of news articles in a visually easy-to-understand manner and meet the needs of a wide variety of users.

[0604] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0605] Step 1: Get the news article

[0606] The server retrieves news articles from external sources. Specifically, it periodically retrieves the latest news articles using RSS feeds or news APIs (e.g., NewsAPI, Google News API). This process involves sending an HTTP request and receiving a response in JSON format. The input is the endpoint URL of the news API, and the output is JSON data that includes the article title, body text, publication date and time, category, etc.

[0607] Step 2: Preprocessing the news article

[0608] The server analyzes the retrieved news articles. It uses Python natural language processing libraries (e.g., Spacy, NLTK) to extract key keywords and phrases. The input is the text of the news article obtained in step 1, and the output is the extracted keywords and categories. Specifically, it tokenizes the article text and performs named entity recognition (NER) and TF-IDF (Term Frequency-Inverse Document Frequency).

[0609] Step 3: Select a template by category

[0610] The server classifies the news article into a specific category based on the analysis results and selects the corresponding illustration template. The input is the category extracted in step 2, and the output is the illustration template corresponding to the category. For example, for the incident category, a map or timeline template is selected.

[0611] Step 4: Generate the diagram

[0612] The server visualizes the content of the news article based on the selected template. For example, if geographic information is displayed, the OpenStreetMap API is used to plot the locations of incidents. If time series data is included, it is parsed using the Datetime library to generate a timeline. The input is the news article text and the selected template, and the output is the generated illustration. Specific operations include retrieving map data and generating maps in HTML or SVG format.

[0613] Step 5: Generate accessible variations

[0614] The server generates variations of different themes, languages, and font sizes to improve the accessibility of the generated illustrations. The input is the illustration generated in step 4, and the output is multiple variations of the illustration. Specific behaviors include applying dark mode using CSS media queries and changing the illustration language using the Google Translate API.

[0615] Step 6: Save and distribute your diagrams

[0616] The server stores the generated diagrams and their variations in a database, using, for example, MongoDB or MySQL. It also uses a service such as Firebase Cloud Messaging (FCM) to send notifications to user devices. The input is the diagram variations, and the output is the diagram stored in the database and the notification sent. Specifically, the data is saved using an INSERT query, and notifications are sent using the FCM API.

[0617] Step 7: Read the news

[0618] The user opens the news app to view the latest news articles and illustrations. The device caches and displays the data obtained from the server. The input is the news article and illustration data sent from the server, and the output is the news article and illustration displayed on the device screen. Specifically, the data is obtained via an HTTP request, and the cached data is managed using the Room library and SQLite.

[0619] Step 8: Customize your diagram

[0620] The user can select the display format of the illustrations (e.g., dark mode, language, font size) in the app's settings screen. The device saves the user's selection and reflects it in the display. The input is the user's settings, and the output is the updated display format. Specific behavior is to save the settings in SharedPreferences or UserDefaults and update the UI in real time.

[0621] Step 9: Displaying the diagram

[0622] The optimal illustration is displayed on the device based on the user's selection. For example, if dark mode is selected, the CSS class and style sheet are changed to display an illustration for dark mode. The input is the user's settings and illustration data, and the output is the illustration in the applied display format. Specifically, the HTML and CSS are dynamically updated.

[0623] In this way, the present invention can display the contents of news articles in a visually easy-to-understand manner and meet the needs of a wide variety of users.

[0624] (Application example 1)

[0625] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0626] Conventional news article distribution systems lack the ability to generate illustrations to aid visual understanding, and therefore do not meet the needs of brick-and-mortar stores, especially in environments where real-time information provision is required. Furthermore, the lack of multilingual support and interactive display options makes it difficult to effectively provide information to customers with diverse backgrounds. Given these circumstances, there is a need for a system that can visualize and instantly distribute news articles in brick-and-mortar stores, as well as provide multilingual support and customizable display options.

[0627] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0628] In this invention, the server includes means for acquiring news articles, means for analyzing the acquired news articles and extracting major keywords and categories, means for selecting an appropriate illustration template based on the extracted category, means for illustrating the content of the news article based on the selected template, means for generating one or more illustration variations, means for saving the generated illustration variations and distributing them to a user terminal, means for translating the news article into multiple languages, and means for displaying the generated illustration variations on a display in a physical store. This enables visual display of news articles and multilingual support in physical stores, improving the efficiency of providing information to customers.

[0629] "News Article" means a news article obtained from an internet source.

[0630] "Means of acquisition" refers to methods of automatically downloading news articles from the Internet or news feeds.

[0631] The "analysis method" refers to a technique that uses natural language processing technology to analyze the content of acquired news articles and extract key keywords and phrases.

[0632] "Major keywords and categories" refer to words that are important for understanding the content of a news article and classifications based on them.

[0633] The "selection means" is a method for automatically selecting the most suitable diagram template based on the analysis results.

[0634] An "illustrated template" is a predefined format for visually representing the content of a news article.

[0635] "Visualization tools" are techniques that use selected templates to generate graphs and charts to visually display the content of a news article.

[0636] "Illustration Variations" are illustrations generated in multiple formats, such as different display formats, languages, themes, font sizes, etc.

[0637] The "means for storing and distributing to the user terminal" is a technology for storing the generated illustration variations in a database and transmitting them to the user's terminal via a network.

[0638] A "means for translating news articles into multiple languages" is a translation technique for converting the content of a news article into a different language.

[0639] "Means for displaying on a display in a physical store" refers to a method for displaying the generated illustration on a display installed in a physical store.

[0640] "Means for enabling users to select display options" refers to technology that provides an interface that allows users to select their preferred display format, language, and theme.

[0641] "Visual display" refers to presenting information in a format that is easy to understand visually.

[0642] "Multilingual support" refers to the ability to provide information in multiple languages.

[0643] The present invention is a system for automatically visualizing news articles, providing multilingual support and customizable display options. Specific embodiments are described below.

[0644] Server Processing

[0645] The server retrieves, analyzes, illustrates, and distributes news articles using the following hardware and software:

[0646] Hardware: Server

[0647] Software: News API, Google Translate API, Natural Language Processing (NLP) technology, Matplotlib

[0648] 1. Retrieving news articles

[0649] The server periodically retrieves the latest news articles from the Internet using a news API.

[0650] 2. News article analysis

[0651] NLP technology is used to extract key keywords and categories from retrieved news articles. For example, keywords such as "Shibuya Ward, Tokyo" and "incident" are extracted from an article titled "An incident occurred in Shibuya Ward, Tokyo."

[0652] 3. Select an illustrated template

[0653] Based on the extracted category, an appropriate diagram template is automatically selected. For example, for the category "incidents," a map or timeline template is selected.

[0654] 4. Generating illustrations

[0655] Based on the selected template, we generate illustrations that visually represent the content of the news article, using Matplotlib to create maps and timeline graphs.

[0656] 5. Generating illustrated variations

[0657] Generate one or more illustrative variations, such as dark mode, different languages, and different font sizes.

[0658] 6. News article translation

[0659] Use the Google Translate API to translate news articles into other languages, for example, from English to Japanese, or from Japanese to English.

[0660] 7. Saving and distributing illustrated variations

[0661] The generated illustration variations are stored in a database and prepared for distribution to the user's terminal.

[0662] Terminal handling

[0663] The terminal (a display in a physical store or the user's mobile device) displays the received news article and illustrations, which can be customized according to the user's specifications.

[0664] Hardware: Digital signage displays in physical stores, users' smartphones

[0665] Software: News viewing application

[0666] 1. Reading the news

[0667] When a user opens the news app, the latest news articles are displayed along with corresponding illustrations. The device caches the data retrieved from the server to provide a smooth display.

[0668] 2. Customizing the illustrations

[0669] From the app's settings screen, users can select the display format of the illustrations (e.g., dark mode, language, font size). When the user changes their settings, the device selects and displays the best version of the illustration from the saved variations.

[0670] 3. Display of illustrations

[0671] The most appropriate illustrations will be displayed based on the user's selection, for example, if dark mode is selected, the illustrations will also be visually displayed in dark mode.

[0672] Specific examples

[0673] Showing news articles:

[0674] Title: "The incident that occurred in Shibuya Ward, Tokyo"

[0675] Content: "Police are continuing their investigation into the incident that occurred last night in Shibuya Ward, Tokyo..."

[0676] Translated text: "An incident occurred last night in Shibuya, Tokyo. The police are continuing their investigation..."

[0677] Generated visualization image:

[0678] It will be saved with the file name visual_20230405_093000.png and displayed on the digital signage.

[0679] Prompt Sentence Examples

[0680] Use the News API to retrieve the latest news articles and then use the Google Translate API to translate the content into English. Then use Matplotlib to generate an image that visually displays the translated content. The generated image will then be displayed on a digital signage display in the store. Write a Python program that implements this process.

[0681] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0682] Step 1:

[0683] Get news articles

[0684] The server periodically retrieves the latest news articles from the Internet using a news API. The input is a news API request, and the output is the retrieved news article data. Specifically, the server sends an HTTP request to the news API and receives news article data in JSON format as a response.

[0685] Step 2:

[0686] News article analysis

[0687] The server uses natural language processing (NLP) techniques to extract key keywords and phrases from retrieved news articles. The input is the retrieved news article data, and the output is the extracted keywords and categories. Specifically, the server uses an NLP library (e.g., spaCy) to analyze the content of the article and extract important keywords and categories.

[0688] Step 3:

[0689] Select an illustrated template

[0690] The server selects an appropriate illustration template based on the extracted category. The input is the extracted category, and the output is the selected illustration template. Specifically, the server searches for templates corresponding to the category from the template library and selects the most appropriate template.

[0691] Step 4:

[0692] Generating illustrations

[0693] The server visualizes the contents of the news article based on the selected template. The input is the content of the news article and the selected template, and the output is the generated illustration. Specifically, the server uses a graph drawing library such as Matplotlib to create the illustration according to the template.

[0694] Step 5:

[0695] Generate illustrated variations

[0696] The server generates variations of the generated illustration, such as different themes (e.g., dark mode), different languages, and different font sizes. The input is a basic illustration, and the output is multiple variations of the illustration. Specifically, the server creates multiple variations with different themes, languages, and font sizes according to the set parameters.

[0697] Step 6:

[0698] News article translation

[0699] The server uses the Google Translate API to translate the content of news articles into multiple languages. The input is the content of the news article written in the original language, and the output is the translated content of the news article. Specifically, the server sends a translation request to the Google Translate API and receives the translation result as a response.

[0700] Step 7:

[0701] Saving and distributing illustrated variations

[0702] The server saves the generated illustration variations in a database and prepares them for distribution to user devices. The input is the illustration variation, and the output is the reference information for the saved illustration and a status ready for distribution. Specifically, the server saves the generated illustration variations in a database and generates and stores reference information such as a URL.

[0703] Step 8:

[0704] Reading the news

[0705] The user terminal displays the latest news articles and corresponding illustrations. The input is the news article and illustration data delivered from the server, and the output is the news and illustrations displayed on the terminal's display. In concrete terms, the terminal displays the news article and illustrations on the screen based on the received data.

[0706] Step 9:

[0707] Customizing the illustrations

[0708] The user terminal allows the user to set the display format of the illustration. The input is the user's selection, and the output is a customized illustration display based on the selection. In concrete terms, the terminal receives the display setting change from the user and selects and displays the optimal illustration variation from the saved variations.

[0709] Step 10:

[0710] Viewing the illustration

[0711] The user terminal displays the illustration based on the selected display format. The input is customized illustration data, and the output is the illustration displayed on the terminal display. In specific operation, the terminal selects and displays an appropriate illustration variation according to the user's settings.

[0712] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0713] This invention combines an emotion engine with a system that acquires, analyzes, and visualizes news articles, enabling the system to recognize user emotions and dynamically customize the display of illustrations based on those emotions. To implement this system, it is necessary to execute the following steps: news article acquisition, analysis, illustration generation, variation creation, emotion recognition, customization, storage, and distribution. The specific flow of each process is explained below.

[0714] Server (backend) processing

[0715] 1. Retrieving news articles

[0716] The server periodically retrieves the latest news articles via RSS feeds or news APIs, along with basic information such as article title, body of text, publication date, and category.

[0717] 2. Preprocessing of news articles

[0718] The server analyzes the retrieved news articles using natural language processing (NLP) technology to extract key keywords and phrases, such as "Shibuya Ward, Tokyo" and "incident."

[0719] 3. Select a template by category

[0720] Based on the extracted keywords and phrases, the server automatically classifies the news articles into specific categories (e.g., incidents, economics, science, sports, etc.) and selects an appropriate illustration template accordingly.

[0721] 4. Generating illustrations

[0722] The server then creates an illustration of the news article based on the selected template, using map information and time-series data to generate a visually easy-to-understand illustration.

[0723] 5. Accessibility-aware variation generation

[0724] The server creates one or more variations of the illustration, such as dark mode, different languages ​​(e.g., English, Japanese), different text sizes, etc.

[0725] 6. Emotion Recognition by Emotion Engine

[0726] The server uses interaction data (camera, microphone, touch operations, etc.) from the user's device to allow the emotion engine to estimate the user's current emotional state (e.g., joy, sadness, surprise, anger, etc.).

[0727] 7. Emotion-based illustrated customization

[0728] The server selects the optimal illustration variation based on the user's emotional data obtained from the emotion engine and dynamically changes the illustration display format. For example, if the user feels sad, it will provide an illustration with a softer color scheme.

[0729] 8. Saving and distributing illustrations

[0730] The server stores the illustration variations selected based on emotion recognition in a database and delivers them to the user's terminal.

[0731] Terminal (client-side) processing

[0732] 1. Reading the news

[0733] When a user opens the news app, the device calls the server's API to retrieve the latest news articles and corresponding illustrations.

[0734] 2. Initial display of the diagram

[0735] The device temporarily caches the received news article and illustrations, and when the news article is opened, the corresponding illustration is initially displayed.

[0736] 3. Acquiring Emotion Data

[0737] The device collects user interaction data (camera footage, voice input, touch operations, etc.) and sends it to the emotion engine, which analyzes it to recognize the user's emotions.

[0738] 4. Customize display options

[0739] Based on the emotion recognition results, the device dynamically adjusts the display options of the illustrations. Users can also manually select display options (e.g., language, theme, font size) from the settings screen.

[0740] User interaction and experience

[0741] 1. Viewing news articles and illustrations

[0742] Users scroll through the list in the news app and select the article they are interested in. When they open the article, the corresponding illustration is displayed along with the details of the news article.

[0743] 2. Emotion-based change

[0744] While the user is browsing the article, the device recognizes the user's emotions in real time and dynamically changes the illustration display style based on the user's emotions. For example, if the user is feeling stressed, the illustration's color scheme will become softer.

[0745] 3. Change settings

[0746] Users can change the display options in the settings screen to suit their preferences, for example, switching from Japanese to English or increasing the font size.

[0747] 4. Use of Emotion History

[0748] The system records the user's emotional history and can provide more customized displays in the future based on the user's emotional patterns.

[0749] As a concrete example, consider the following scenario: When a user browses an "article about an incident" in a news app, the article contains content about an "incident that occurred in Shibuya Ward, Tokyo." The analysis engine extracts the keywords "Shibuya Ward, Tokyo" and "incident" and classifies the category as "incident." The server selects a template for the "incident category" and generates an illustration that combines map and timeline data. It then creates and saves multiple variations (e.g., for dark mode, English version, uppercase version).

[0750] When a user opens this news article on their device, a map and timeline illustration are displayed. At the same time, an emotion engine analyzes the user's facial expression and, if the user expresses sadness, the illustration's color scheme is softened.

[0751] In this way, the present invention not only displays the contents of news articles in a visually easy-to-understand manner, but also dynamically customizes the content according to the user's emotions, thereby enabling more personalized information to be provided.

[0752] The processing flow will be explained below.

[0753] Server (backend) processing

[0754] Step 1:

[0755] The server periodically queries an RSS feed or news API to retrieve new news articles, including basic information such as the article title, body of the article, publication date, and category.

[0756] Step 2:

[0757] The server analyzes the retrieved news articles using natural language processing (NLP) technology to extract key keywords and phrases, such as "Shibuya Ward, Tokyo" and "incident."

[0758] Step 3:

[0759] Based on the extracted keywords and phrases, the server automatically classifies news articles into specific categories (incidents, economics, science, sports, etc.).

[0760] Step 4:

[0761] The server selects an appropriate illustration template for each category, for example, a map or timeline template for an article in the incident category.

[0762] Step 5:

[0763] The server then creates an illustration of the news article based on the selected template, using map information and time-series data to generate a visually easy-to-understand illustration.

[0764] Step 6:

[0765] The server creates one or more illustrated variations for accessibility, including dark mode, different languages ​​(e.g., English, Japanese), different text sizes, etc.

[0766] Step 7:

[0767] The server uses interaction data (camera, microphone, touch operations, etc.) from the user's device to allow the emotion engine to estimate the user's current emotional state (joy, sadness, surprise, anger, etc.).

[0768] Step 8:

[0769] Based on the user's emotional data obtained from the emotion engine, the server selects the optimal illustration variation and dynamically changes the illustration display format. For example, if the user is feeling sad, it will provide an illustration with a softer color scheme.

[0770] Step 9:

[0771] The server stores the generated illustrations and their variations in a database and prepares them for distribution to user terminals.

[0772] Terminal (client-side) processing

[0773] Step 1:

[0774] When a user opens the news app, the device calls the server's API to retrieve the latest news articles and corresponding illustrations.

[0775] Step 2:

[0776] The device temporarily caches received news articles and illustrations to provide smooth display.

[0777] Step 3:

[0778] A user scrolls through the news list and taps on an article they are interested in. The article details screen opens, along with a corresponding illustration.

[0779] Step 4:

[0780] The device collects user interaction data (camera footage, voice input, touch operations, etc.) and sends it to the emotion engine, which analyzes it to recognize the user's emotions.

[0781] Step 5:

[0782] Based on the emotion recognition results, the device dynamically adjusts the display options of the illustrations. Users can also manually select display options (e.g., language, theme, font size) from the settings screen.

[0783] User interaction and experience

[0784] Step 1:

[0785] Users scroll through the news list in a news app, find an article that interests them, and tap on it.

[0786] Step 2:

[0787] When you open an article, you'll see details about the news story along with visually compelling illustrations, such as a map showing the location of the incident and a timeline of chronological events.

[0788] Step 3:

[0789] While the user is browsing the article, the device recognizes the user's emotions in real time and dynamically changes the illustration display style based on the user's emotions. For example, if the user is feeling stressed, the illustration's color scheme will become softer.

[0790] Step 4:

[0791] Users can change the display options in the settings screen to suit their preferences, for example, switching from Japanese to English or increasing the font size.

[0792] Step 5:

[0793] The system records the user's emotional history and can provide more customized displays in the future based on the user's emotional patterns.

[0794] As a concrete example, consider the following scenario: When a user browses an "article about an incident" in a news app, the article contains content about an "incident that occurred in Shibuya Ward, Tokyo." The analysis engine extracts the keywords "Shibuya Ward, Tokyo" and "incident" and classifies the category as "incident." The server selects a template for the "incident category" and generates an illustration that combines map and timeline data. It then creates and saves multiple variations (e.g., for dark mode, English version, uppercase version).

[0795] When a user opens this news article on their device, a map and timeline illustration are displayed. At the same time, an emotion engine analyzes the user's facial expression and, if the user expresses sadness, the illustration's color scheme is softened.

[0796] In this way, the present invention not only displays the contents of news articles in a visually easy-to-understand manner, but also dynamically customizes the content according to the user's emotions, thereby enabling more personalized information to be provided.

[0797] Example 2

[0798] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0799] Conventional news article delivery systems have had difficulty providing information tailored to individual users' emotional states. Furthermore, they were unable to dynamically customize the content of news articles to match the user's emotional state. As a result, the user experience was static, potentially leaving some users dissatisfied.

[0800] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring news articles, means for analyzing the acquired news articles using natural language processing technology and extracting major keywords and categories, means for selecting an appropriate illustration template based on the extracted keywords and categories, means for generating an illustration that visualizes the content of the news article based on the selected template, means for creating one or more illustration variations taking accessibility into consideration, means for collecting user interaction data and recognizing emotions, means for dynamically customizing the illustration variations based on the recognized emotions, and means for saving the generated illustration variations and delivering them to the user terminal. This dynamically customizes the visual display of the news article, enabling more personalized information to be provided based on the user's emotions.

[0801] "Means for obtaining news articles" refers to the function of periodically collecting news articles using a news API or RSS feed.

[0802] "Means of analyzing acquired news articles using natural language processing technology and extracting key keywords and categories" refers to a function that analyzes acquired news articles using natural language processing technology such as morphological analysis and automatically extracts key keywords and categories.

[0803] "Means for selecting an appropriate illustrated template based on extracted keywords and categories" is a function that automatically selects the most appropriate template from multiple pre-prepared illustrated templates based on analyzed keyword and category information.

[0804] "Means for generating illustrations that visualize the contents of a news article based on a selected template" refers to a function that uses a selected illustration template to generate illustrations that represent the contents of a news article in a visually easy-to-understand format, such as a map or timeline.

[0805] "Means for creating one or more variations of illustrations with accessibility in mind" refers to the ability to create variations of the generated illustrations in different formats (e.g., dark mode, different languages, different font sizes) that take into account user visibility and ease of use.

[0806] "Means for collecting user interaction data and recognizing emotions" refers to a function that collects data such as the user's camera footage, voice input, and touch operations, and analyzes it to estimate the user's current emotional state.

[0807] "Means for dynamically customizing illustration variations based on recognized emotions" is a function that selects the optimal illustration variation based on the user's emotional data and changes the color scheme and design of the illustration in real time.

[0808] The "means for saving the generated illustration variations and distributing them to the user's terminal" is a function for saving the generated multiple illustration variations in a database and distributing them to the user's terminal via the Internet.

[0809] MODE FOR CARRYING OUT THE INVENTION

[0810] This invention aims to provide a system for acquiring, analyzing, and illustrating news articles, and to provide a dynamically customized visual display according to the user's emotions. This system uses the following hardware and software:

[0811] Hardware

[0812] Server: Responsible for news article acquisition, analysis, illustration generation, emotion recognition, customization, storage and distribution.

[0813] User devices: Responsible for displaying news articles, collecting and transmitting emotion data, and dynamically changing the display. These devices include smartphones, tablets, and PCs.

[0814] software

[0815] Natural Language Processing (NLP) software: Use libraries such as SpaCy and NLTK to analyze news articles and extract key keywords and categories.

[0816] News API: Uses an API such as Google News API to make an HTTP request to retrieve the latest news articles.

[0817] Illustration generation library: Uses D3.js, Google Maps API, etc. to create visually easy-to-understand illustrations of the contents of news articles.

[0818] Emotion recognition engine: Analyzes user emotions using Microsoft Azure Emotion API, etc.

[0819] Example

[0820] 1. Acquiring and analyzing news articles

[0821] The server periodically retrieves news articles using the Google News API or RSS feeds. The retrieved articles are analyzed using natural language processing technology to extract key keywords and categories. For example, a request like https: / / newsapi.org / v2 / top-headlines?country=jp&apiKey=YOUR_API_KEY is sent.

[0822] 2. Template selection and diagram generation

[0823] Based on the extracted keywords and categories, the server selects an appropriate illustration template. Based on the selected template, it generates illustrations such as maps and timeline data using the Google Maps API and D3.js. For example, if the article is about an incident that occurred in Shibuya Ward, Tokyo, it selects a template in the incident category and illustrates the map and timeline data.

[0824] 3. Accessible Variation Generation

[0825] The server creates multiple variations of the generated illustrations, such as dark mode, different languages ​​(English, Japanese), different font sizes, etc. For example, it applies dark mode styles with CSS and adds different language labels to the illustration data.

[0826] 4. Emotional Data Collection and Recognition

[0827] While the user is browsing a news article, the device collects interaction data from the camera and microphone and sends it to the server. The server's emotion recognition engine analyzes the user's emotional state. For example, the user's emotional state can be recognized as "happiness," "sadness," "surprise," "anger," etc. based on facial expressions and voice analysis.

[0828] 5. Dynamic customization based on emotions

[0829] The server selects the most appropriate illustration variation based on the user's recognized emotion and changes the illustration's color scheme and design in real time. For example, if the user feels sad, it selects an illustration with a soft color scheme.

[0830] 6. Saving and distributing illustrations

[0831] The selected illustration variations are stored in a database and delivered to the user's device via the Internet, using a real-time database such as Firebase to transmit data in real time.

[0832] Specific examples and prompts for the generative AI model

[0833] As a specific example, consider a scenario in which a user views an article about an incident that occurred in Shibuya Ward, Tokyo.

[0834] When a user opens the news app, an article about an incident that occurred in Shibuya Ward, Tokyo, is retrieved from the server. The analysis engine extracts the keywords "Shibuya Ward, Tokyo" and "incident" from the article and classifies it as an "incident" category. The server selects a template that combines a map and a timeline to visualize the news content. The generated variations (e.g., dark mode, English version, uppercase version) are stored in a database.

[0835] When a user opens an article on their device, a map and timeline illustration are displayed. At the same time, the emotion engine analyzes the user's facial expressions and, if the user is expressing sadness, the illustration's color scheme is changed to a softer tone.

[0836] Example prompts to input to a generative AI model:

[0837] A user is viewing an article about an incident that occurred in Shibuya Ward, Tokyo. The article's category is "Incident," and an illustration using map and timeline data should be displayed. If the user's emotion is "Sadness," change the illustration's color scheme to a softer tone.

[0838] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0839] Step 1: Get the news article

[0840] Input: Periodically send requests for news articles using a news API or RSS feed.

[0841] Data processing: The server sends an HTTP request to the API endpoint and receives the returned data in JSON format.

[0842] Output: Data including news article title, body of text, publication date, category, etc.

[0843] Specific operation: Execute a request such as https: / / newsapi.org / v2 / top-headlines?country=jp&apiKey=YOUR_API_KEY.

[0844] Step 2: Preprocessing the news article

[0845] Input: News article data obtained in step 1.

[0846] Data processing: The server analyzes the news articles using natural language processing techniques (e.g., SpaCy, NLTK) to extract key keywords and phrases.

[0847] Output: Extracted keywords and phrases (e.g., "Shibuya Ward, Tokyo" and "incident").

[0848] Specific operation: Performs morphological analysis and extracts important words and phrases in the sentence.

[0849] Step 3: Select a template by category

[0850] Input: Keywords or phrases extracted in step 2.

[0851] Data processing: The server classifies news articles into specific categories based on extracted keywords and phrases, and selects appropriate illustration templates for each category.

[0852] Output: The selected diagram template (e.g., a template for incident categories).

[0853] Specific operation: Based on the analysis results, categorize them into categories such as "incidents," "economy," and "science," and select the corresponding template for each.

[0854] Step 4: Generate the diagram

[0855] Input: The illustrated template selected in step 3 and the content of the news article.

[0856] Data processing: The server generates illustrations that visualize the contents of news articles based on the selected template. Maps and timeline data are created using Google Maps API, D3.js, etc.

[0857] Output: Visualized and illustrated data.

[0858] Specific functions: Display the locations of incidents on a map and create a chronological timeline.

[0859] Step 5: Generate variations with accessibility in mind

[0860] Input: Illustrated data generated in step 4.

[0861] Data processing: The server creates variations of the generated illustrations in different formats (e.g., dark mode, different languages, changed font size).

[0862] Output: Multiple illustrated variations.

[0863] What it does: Changes CSS to apply dark mode styles and adds labels in different languages ​​to illustrated data.

[0864] Step 6: Emotion data collection and emotion recognition

[0865] Input: Interaction data (camera feeds, voice inputs, touch actions, etc.) while the user is viewing an article.

[0866] Data processing: The interaction data collected by the device is sent to the server, where the server's emotion recognition engine analyzes the emotional state.

[0867] Output: User emotion data (e.g., happy, sad, surprised, angry).

[0868] Specific operation: The user's facial expressions and voice are collected through a camera and microphone and analyzed using an emotion recognition model.

[0869] Step 7: Emotion-based illustration customization

[0870] Input: User emotion data recognized in step 6.

[0871] Data processing: The server selects the optimal illustration variation based on the emotion data and dynamically changes the color scheme and design of the illustration.

[0872] Output: Customized illustrated data.

[0873] Specific behavior: If the user expresses sadness, change the color scheme of the illustration to a softer tone.

[0874] Step 8: Save and distribute your diagrams

[0875] Input: Customized diagram data from step 7.

[0876] Data processing: The server stores the customized illustrations in a database and delivers them to the user's device.

[0877] Output: Illustrated data stored on the user's device.

[0878] What it does: Uses a real-time database such as Firebase to send customized diagrams over the internet.

[0879] Step 9: Initial display of news article and illustration

[0880] Input: News article and illustration data distributed in step 8.

[0881] Data processing: The device caches the news article and illustrations it receives, and initially displays the illustrations when the news article is opened.

[0882] Output: News article and illustration displayed on the user's terminal.

[0883] What it does: When you open the news app, it displays the latest news articles and their illustrations.

[0884] Step 10: Customize display options

[0885] Input: User settings changes (language, theme, text size, etc.).

[0886] Data processing: The device dynamically adjusts the display options of the illustration based on user settings changes.

[0887] Output: Customized graphical display.

[0888] Specific operation: The user changes the display options in the settings screen to display the diagram according to their preference.

[0889] (Application example 2)

[0890] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0891] Conventional advertising delivery systems have difficulty customizing ads based on user emotions, limiting their visibility and effectiveness. Furthermore, there is no technology for summarizing news-like information and incorporating it into ads, making it difficult for ads to be interesting to users. Therefore, there is a need for a system that can provide more personalized information by recognizing user emotions in real time and dynamically changing advertising design and content based on those emotions.

[0892] The specification processing by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring a news article, means for analyzing the news article and extracting main keywords and categories, means for selecting an appropriate illustration template based on the extracted category, means for illustrating the content of the news article based on the selected template, means for generating one or more illustration variations, means for saving the generated illustration variations and delivering them to the user terminal, means for recognizing the user's emotions and dynamically changing the display format of the illustration based on the emotions, means for summarizing the news article and incorporating it into the generated advertising content and displaying it, and means for dynamically changing the advertising design based on the user's emotions. This enables dynamic advertising delivery linked to the user's emotions, making it possible to dramatically improve the visibility and effectiveness of advertising.

[0893] A "news article" is a set of documents or information that is reported as news.

[0894] An "advertising distribution system" is a system for distributing advertisements for products and services to users.

[0895] "Emotion recognition" is a technology that analyzes and recognizes a user's emotions from interaction data such as camera and voice.

[0896] An "illustration" is a graphical representation used to make textual information easier to understand visually.

[0897] A "template" is a standard format that serves as the basis for generating diagrams and displaying them.

[0898] "Illustration Variants" are different versions of an illustration that are generated in multiple formats, such as different languages, themes, font sizes, etc.

[0899] "Dynamic change" means making changes automatically in real time according to the situation or conditions.

[0900] "User terminal" refers to a computer device used by a user, such as a smartphone, tablet, or PC.

[0901] "Customization" means changing settings according to the needs and preferences of a particular user.

[0902] A "generative AI model" is an artificial intelligence model that learns from large amounts of data and generates useful output for new data.

[0903] A "prompt sentence" is an instruction sentence that is input to a generative AI model to obtain a specific output.

[0904] The present invention is a dynamic advertising delivery system based on user emotions, and provides technology for customizing advertising content through the acquisition, analysis, and visualization of news articles.

[0905] Server (backend) processing

[0906] The server periodically retrieves news articles through a news API. The retrieved news articles are analyzed using natural language processing (NLP) techniques to extract key keywords and categories. An appropriate illustration template is then selected based on the extracted categories, and the news article is visualized based on the selected template. One or more illustration variations are generated, which may include different languages, themes, and font sizes.

[0907] Next, the emotion engine recognizes the user's emotions. This recognition is performed in real time from interaction information collected using a camera and microphone. Based on the emotion recognition, the display format of the illustrations is dynamically changed, and a summary of the news article is incorporated into the generated advertisement content. The generated advertisement content is then delivered to the user's device in an optimized design. The following specific technologies are used in this series of processes:

[0908] Natural Language Processing (NLP) technology: Keyword analysis of retrieved news articles

[0909] Emotion Recognition Engine: Analyzes user emotions

[0910] Illustration generation engine: Generate illustrations based on keywords

[0911] Database: storing generated illustrations and ad variations

[0912] User terminal (client side) processing

[0913] The user's device displays the news article and corresponding illustrations received from the server. Web technologies such as HTML and CSS are used for the display, allowing for dynamic customization. In addition, the user's device captures the user's emotional data through a camera and microphone and sends it to the server. Based on feedback from the server, the design and color scheme of the advertisement are changed in real time.

[0914] User interaction and experience

[0915] When a user opens the ad display app, they are shown ads that summarize news-like information. For example, an ad for a new smartphone product summarizes and visualizes the latest trends in the smartphone industry. As the user views the ad, the design and color scheme of the ad dynamically changes depending on the user's emotions, providing a more personalized advertising experience.

[0916] Specific examples of prompts are as follows:

[0917] "Summarize the latest smartphone industry trending news and create a visually compelling illustration. If the user is expressing joy, use a brighter color scheme to create a positive impression."

[0918] Using this prompt, the generative AI model generates advertising content, resulting in advertising designs that adapt to the user's emotions.

[0919] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0920] Step 1:

[0921] The server retrieves the latest news articles using the news API. In this step, an API request is sent and JSON-formatted data is received, including basic information such as the article title, body, publication date and time, and category. The input is the request to the news API, and the output is the JSON data of the retrieved news articles.

[0922] Step 2:

[0923] The server analyzes the retrieved news articles and extracts key keywords and categories. In this step, natural language processing (NLP) techniques are used to analyze the article text and extract important keywords and phrases. The input is the text data of the news article, and the output is the extracted keyword and category data.

[0924] Step 3:

[0925] The server selects an appropriate illustration template based on the extracted category. In this step, it selects the illustration template that best suits the extracted category from a predefined template library. The input is the extracted category data, and the output is the selected illustration template.

[0926] Step 4:

[0927] The server then creates an illustration of the news article content based on the selected template. In this step, the text and keywords are converted into a visually easy-to-understand illustration. Specifically, the keywords are arranged in graphs, charts, maps, etc. The input is the news article text and keyword data, and the output is the generated illustration.

[0928] Step 5:

[0929] The server generates one or more variations of the illustration. In this step, the illustration is generated in multiple formats, such as different languages, themes, font sizes, etc. The input is the generated illustration, and the output is each variation of the illustration.

[0930] Step 6:

[0931] The server saves the generated illustration variations and delivers them to the user terminal. In this step, the illustration variations are saved in a database and a link is generated for users to access them. The input is the generated illustration variations, and the output is the link information for the saved illustrations.

[0932] Step 7:

[0933] The terminal displays the received news article and related illustrations. In this step, the news article and illustrations retrieved from the server are displayed on the screen using HTML and CSS. The input is the news article and illustration received from the server, and the output is a display screen that the user can view.

[0934] Step 8:

[0935] The device acquires the user's emotional data through a camera and microphone and sends it to the server. In this step, the device analyzes the user's facial expressions and tone of voice to recognize their emotions. The input is camera video and audio data, and the output is the recognized emotional data.

[0936] Step 9:

[0937] The server uses an emotion engine to recognize the user's emotion and dynamically change the display format of the illustration based on the emotion. In this step, the color scheme and design of the illustration are changed based on the recognized emotion data. The input is the user's emotion data, and the output is a dynamically changed illustration.

[0938] Step 10:

[0939] The server then summarises the news article and incorporates it into the generated advertisement content, and displays it. Specifically, it incorporates the news article summary into the advertisement in a design that adapts to the user's emotions. The input for this step is the news article summary and an emotion-based design template, and the output is optimized advertisement content.

[0940] Step 11:

[0941] The device dynamically changes the ad design based on the user's emotions. In this step, the color scheme and layout of the ad are adjusted in real time to match the user's emotions. The input is the optimized ad data from the server, and the output is the final ad display.

[0942] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0943] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0944] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0945] [Third embodiment]

[0946] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0947] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0948] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0949] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0950] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0951] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0952] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0953] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0954] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0955] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0956] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0957] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0958] This invention is a system that automatically visualizes news articles. It analyzes the content of news articles and generates and distributes appropriate illustrations, allowing readers to easily and quickly understand the information. To implement this system, it is necessary to execute the following steps: news article acquisition, analysis, illustration generation, variation creation, storage, and distribution. The specific flow of each process is explained below.

[0959] Server (backend) processing

[0960] 1. Retrieving news articles

[0961] The server periodically retrieves the latest news articles via RSS feeds or news APIs, capturing information such as the article title, body of text, publication date, and category.

[0962] 2. Preprocessing of news articles

[0963] Natural language processing (NLP) techniques are used to extract key keywords and phrases from retrieved news articles, and based on the results of this analysis, articles are classified into specific categories (e.g., crime, economy, science, sports, etc.).

[0964] 3. Select a template by category

[0965] Depending on the category, the server will select an appropriate illustration template, for example, a map or timeline template for an incident article.

[0966] 4. Generating illustrations

[0967] The server then creates an illustration of the news article based on the selected template, using map information and time series data to generate a visually easy-to-understand illustration.

[0968] 5. Accessibility-aware variation generation

[0969] Create one or more illustrated variations, such as dark mode, different languages, different text sizes, etc.

[0970] 6. Saving and distributing illustrations

[0971] The generated illustrations and their variations are stored in a database and prepared for distribution to user terminals.

[0972] Terminal (client-side) processing

[0973] 1. Reading the news

[0974] When a user opens the news app, the latest news articles are displayed along with corresponding illustrations. The device caches the data retrieved from the server to provide a smooth display.

[0975] 2. Customizing the illustrations

[0976] From the app's settings screen, users can select the display format of the illustrations (e.g., dark mode, language, font size). When the user changes their settings, the device selects and displays the best version of the illustration from the saved variations.

[0977] 3. Display of illustrations

[0978] The most appropriate illustrations will be displayed based on the user's selection, for example, if dark mode is selected, the illustrations will also be visually displayed in dark mode.

[0979] User interaction and experience

[0980] 1. Viewing news articles and illustrations

[0981] Users can browse the news list in the news app and select an article they are interested in. When they open an article, a corresponding illustration is displayed at the same time, allowing them to visually understand the content.

[0982] 2. Change settings

[0983] Users can change the display options in the settings screen to suit their preferences, such as increasing the font size or changing the display language.

[0984] 3. Multilingual support available

[0985] Users can change the language of articles and illustrations by changing the settings, allowing them to be displayed in different languages, which allows for a wide range of readership.

[0986] As a concrete example, consider the following scenario: When a user browses an "article about an incident" in a news app, the article contains content about an "incident that occurred in Shibuya Ward, Tokyo." The analysis engine extracts the keywords "Shibuya Ward, Tokyo" and "incident" and classifies the category as "incident." The server selects a template for the "incident category" and generates an illustration that combines a map and time-series data. Furthermore, it creates multiple variations (e.g., for dark mode, English version, uppercase version) and stores them in a database.

[0987] When a user opens this news article on their device, they will see the map and timeline illustration. If they select dark mode from the settings screen, the settings will be reflected immediately and the illustration will change to the dark mode version.

[0988] In this way, the present invention displays the contents of news articles in a visually easy-to-understand manner, meeting the needs of a wide variety of users.

[0989] The processing flow will be explained below.

[0990] Server (backend) processing

[0991] Step 1:

[0992] The server periodically queries the RSS feed or news API to retrieve new news articles, along with basic information such as the article title, body of the article, publication date, and category.

[0993] Step 2:

[0994] The server analyzes the retrieved news articles using natural language processing (NLP) technology to extract key keywords and phrases, such as "Shibuya Ward, Tokyo" and "incident."

[0995] Step 3:

[0996] The server automatically classifies news articles into specific categories (incidents, economics, science, sports, etc.) based on the extracted keywords and phrases.

[0997] Step 4:

[0998] The server selects an appropriate illustration template for each category, for example, a map and timeline template for articles in the incident category.

[0999] Step 5:

[1000] The server then creates an illustration of the news article based on the selected template, using map information and time-series data to generate a visually easy-to-understand illustration.

[1001] Step 6:

[1002] For accessibility reasons, the server creates one or more variations of the illustration, including dark mode, different languages ​​(e.g., English, Japanese), different font sizes, etc.

[1003] Step 7:

[1004] The server stores the generated illustrations and their variations in a database and prepares them for distribution to user terminals.

[1005] Terminal (client-side) processing

[1006] Step 1:

[1007] When a user opens the news app, the device calls the server's API to retrieve the latest news articles and corresponding illustrations.

[1008] Step 2:

[1009] The device temporarily caches received news articles and illustrations to provide smooth display.

[1010] Step 3:

[1011] A user scrolls through the news list and taps on an article they are interested in. The article details screen opens, along with a corresponding illustration.

[1012] Step 4:

[1013] The device's settings screen allows the user to select display options (language, theme, text size, etc.).

[1014] Step 5:

[1015] Depending on the user's settings, the device retrieves the optimal illustration variation from the server and changes the display format. For example, if dark mode is selected, the illustration will also be displayed in dark mode.

[1016] User interaction and experience

[1017] Step 1:

[1018] Users scroll through the news list in a news app, find an article that interests them, and tap on it.

[1019] Step 2:

[1020] When you open an article, you'll see details about the news story along with visually compelling illustrations, such as a map showing the location of the incident and a timeline of chronological events.

[1021] Step 3:

[1022] The user accesses the settings screen and changes display options, for example, changing the language from English to Japanese or switching to dark mode.

[1023] Step 4:

[1024] Your preferences are respected and articles and illustrations are displayed in the format of your choice, providing the best possible display based on your visual needs and preferences.

[1025] Example 1

[1026] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1027] Traditional news articles are often presented in text only, which means it takes time for readers to quickly and easily understand the information. Furthermore, the lack of visual information makes it difficult to understand particularly complex news content. Furthermore, the inability to customize the content to meet users' visual preferences and needs poses challenges in terms of accessibility and usability.

[1028] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1029] In this invention, the server includes means for acquiring news articles, means for analyzing the acquired news articles and extracting major keywords and categories, means for selecting an appropriate illustration template based on the extracted category, means for illustrating the content of the news article based on the selected template, means for generating multiple variations with different themes, languages, font sizes, etc., and means for saving the generated illustration variations and distributing them to user terminals. This makes it possible to display the content of news articles in a visually easy-to-understand manner and meet the needs of a variety of users.

[1030] A "news article" is a piece of text that describes the latest information or events distributed by a news organization or information provider.

[1031] A "server" is a central computer system that stores, processes, and distributes data.

[1032] A "terminal" is a client device that is directly operated by a user, and includes smartphones, tablets, PCs, etc.

[1033] An "acquisition means" is a process or device for automatically collecting news articles from external sources.

[1034] "Means for analysis" refers to the processing or device that analyzes the content of the acquired news articles using techniques such as natural language processing and extracts key keywords and categories.

[1035] An "illustrated template" is a pre-designed type or format for visually displaying the content of a news article.

[1036] A "means for visualizing" is a process or device for visualizing the content of a news article based on a selected graphical template.

[1037] "Means for generating variations" refers to a process or device for creating multiple formats from a basic illustration, with different themes, languages, font sizes, etc., according to the user's needs.

[1038] The "storing means" refers to a process or device for permanently storing the generated illustration variations in a database or the like.

[1039] The "distribution means" refers to a process or device for transmitting the saved illustration variations to a user terminal.

[1040] The "displaying means" refers to a process or device for displaying the news article and related illustrations received by the user terminal on the screen.

[1041] "Display options" are settings that the user can select to change the display format of the illustration.

[1042] "Different languages" refers to a format in which illustrated variations are generated in multiple languages, such as Japanese and English.

[1043] A "theme" is a style setting that changes the appearance and color of an illustration.

[1044] "Text size" is a setting item that changes the size of the text in the illustration.

[1045] The present invention is a system that automatically visualizes news articles, in which the server, terminal, and user elements work together to analyze the content of the news article, appropriately illustrate the information, generate variations, and deliver them to the user.

[1046] Server Processing

[1047] The server retrieves the latest news articles using an RSS feed or news API. Specifically, the server receives JSON-formatted data from a news provider using an HTTP request, and extracts information such as the article title, body text, publication date and category from that data. For example, the News API or Google News API may be used.

[1048] The server then applies natural language processing (NLP) techniques to the retrieved news articles. It uses Python libraries like Spacy and NLTK to extract key keywords and phrases from the articles and classify them into categories (e.g., news, economics, science, sports, etc.). This analysis involves techniques such as tokenization, named entity recognition (NER), and Term Frequency-Inverse Document Frequency (TF-IDF).

[1049] Based on the analysis results, the server classifies the news articles into specific categories and selects a corresponding illustration template for that category. This template refers to a type of visual display of the news content, such as a map, timeline, or graph. For example, D3.js or a geographic information system is used to create the map, and a library such as Chart.js is used to display the timeline.

[1050] Based on the selected template, the server creates a visual representation of the news article. For example, to display the geographic location of an incident, map data is retrieved from the OpenStreetMap API and plotted. Timeline data is parsed using the Datetime library as appropriate.

[1051] Additionally, the server generates multiple variations of the generated illustrations to improve their accessibility, such as dark mode, different languages ​​(using the Google Translate API), and different font sizes. Variation generation is implemented using CSS media queries and internationalization (i18n) libraries.

[1052] The generated diagrams and their variations are stored in a database such as MongoDB or MySQL, and then a service such as Firebase Cloud Messaging (FCM) is used to send notifications to user devices.

[1053] Terminal handling

[1054] When a user opens the news app, the device retrieves the latest news articles and illustrations from the server, and the retrieved data is cached in local storage using the Room library and SQLite to provide a smooth display.

[1055] Users can choose how the icons are displayed in the app's settings. Options include dark mode, different languages, and font sizes. User settings are saved in SharedPreferences and UserDefaults. When a setting is changed, the icons are instantly updated to reflect that setting.

[1056] The device will display the best illustration variation based on the user's settings. For example, if the user selects dark mode, the device will apply stylesheets and CSS classes to display dark mode illustrations.

[1057] User interaction and experience

[1058] Users can browse through a list of articles within the news app and select the news article they are interested in. When they tap on an article, a related illustration is displayed at the same time, helping them to understand the content intuitively. For example, if they select a news article about an incident that occurred in Shibuya Ward, Tokyo, an illustration will appear that provides geographical and chronological information about the incident.

[1059] By changing the display options in the settings screen, users can set the display format to suit their preferences. They can also easily increase the font size and change the display language, making it possible to meet a variety of needs.

[1060] Example prompt sentence:

[1061] "Visualize a news story about an incident that occurred in Shibuya Ward, Tokyo, and generate an illustration using maps and time series data. Create multiple variations of the illustration, including a dark mode version, English version, and uppercase version."

[1062] In this way, the present invention can display the contents of news articles in a visually easy-to-understand manner and meet the needs of a wide variety of users.

[1063] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1064] Step 1: Get the news article

[1065] The server retrieves news articles from external sources. Specifically, it periodically retrieves the latest news articles using RSS feeds or news APIs (e.g., NewsAPI, Google News API). This process involves sending an HTTP request and receiving a response in JSON format. The input is the endpoint URL of the news API, and the output is JSON data that includes the article title, body text, publication date and time, category, etc.

[1066] Step 2: Preprocessing the news article

[1067] The server analyzes the retrieved news articles. It uses Python natural language processing libraries (e.g., Spacy, NLTK) to extract key keywords and phrases. The input is the text of the news article obtained in step 1, and the output is the extracted keywords and categories. Specifically, it tokenizes the article text and performs named entity recognition (NER) and TF-IDF (Term Frequency-Inverse Document Frequency).

[1068] Step 3: Select a template by category

[1069] The server classifies the news article into a specific category based on the analysis results and selects the corresponding illustration template. The input is the category extracted in step 2, and the output is the illustration template corresponding to the category. For example, for the incident category, a map or timeline template is selected.

[1070] Step 4: Generate the diagram

[1071] The server visualizes the content of the news article based on the selected template. For example, if geographic information is displayed, the OpenStreetMap API is used to plot the locations of incidents. If time series data is included, it is parsed using the Datetime library to generate a timeline. The input is the news article text and the selected template, and the output is the generated illustration. Specific operations include retrieving map data and generating maps in HTML or SVG format.

[1072] Step 5: Generate accessible variations

[1073] The server generates variations of different themes, languages, and font sizes to improve the accessibility of the generated illustrations. The input is the illustration generated in step 4, and the output is multiple variations of the illustration. Specific behaviors include applying dark mode using CSS media queries and changing the illustration language using the Google Translate API.

[1074] Step 6: Save and distribute your diagrams

[1075] The server stores the generated diagrams and their variations in a database, using, for example, MongoDB or MySQL. It also uses a service such as Firebase Cloud Messaging (FCM) to send notifications to user devices. The input is the diagram variations, and the output is the diagram stored in the database and the notification sent. Specifically, the data is saved using an INSERT query, and notifications are sent using the FCM API.

[1076] Step 7: Read the news

[1077] The user opens the news app to view the latest news articles and illustrations. The device caches and displays the data obtained from the server. The input is the news article and illustration data sent from the server, and the output is the news article and illustration displayed on the device screen. Specifically, the data is obtained via an HTTP request, and the cached data is managed using the Room library and SQLite.

[1078] Step 8: Customize your diagram

[1079] The user can select the display format of the illustrations (e.g., dark mode, language, font size) in the app's settings screen. The device saves the user's selection and reflects it in the display. The input is the user's settings, and the output is the updated display format. Specific behavior is to save the settings in SharedPreferences or UserDefaults and update the UI in real time.

[1080] Step 9: Displaying the diagram

[1081] The optimal illustration is displayed on the device based on the user's selection. For example, if dark mode is selected, the CSS class and style sheet are changed to display an illustration for dark mode. The input is the user's settings and illustration data, and the output is the illustration in the applied display format. Specifically, the HTML and CSS are dynamically updated.

[1082] In this way, the present invention can display the contents of news articles in a visually easy-to-understand manner and meet the needs of a wide variety of users.

[1083] (Application example 1)

[1084] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1085] Conventional news article distribution systems lack the ability to generate illustrations to aid visual understanding, and therefore do not meet the needs of brick-and-mortar stores, especially in environments where real-time information provision is required. Furthermore, the lack of multilingual support and interactive display options makes it difficult to effectively provide information to customers with diverse backgrounds. Given these circumstances, there is a need for a system that can visualize and instantly distribute news articles in brick-and-mortar stores, as well as provide multilingual support and customizable display options.

[1086] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1087] In this invention, the server includes means for acquiring news articles, means for analyzing the acquired news articles and extracting major keywords and categories, means for selecting an appropriate illustration template based on the extracted category, means for illustrating the content of the news article based on the selected template, means for generating one or more illustration variations, means for saving the generated illustration variations and distributing them to a user terminal, means for translating the news article into multiple languages, and means for displaying the generated illustration variations on a display in a physical store. This enables visual display of news articles and multilingual support in physical stores, improving the efficiency of providing information to customers.

[1088] "News Article" means a news article obtained from an internet source.

[1089] "Means of acquisition" refers to methods of automatically downloading news articles from the Internet or news feeds.

[1090] The "analysis method" refers to a technique that uses natural language processing technology to analyze the content of acquired news articles and extract key keywords and phrases.

[1091] "Major keywords and categories" refer to words that are important for understanding the content of a news article and classifications based on them.

[1092] The "selection means" is a method for automatically selecting the most suitable diagram template based on the analysis results.

[1093] An "illustrated template" is a predefined format for visually representing the content of a news article.

[1094] "Visualization tools" are techniques that use selected templates to generate graphs and charts to visually display the content of a news article.

[1095] "Illustration Variations" are illustrations generated in multiple formats, such as different display formats, languages, themes, font sizes, etc.

[1096] The "means for storing and distributing to the user terminal" is a technology for storing the generated illustration variations in a database and transmitting them to the user's terminal via a network.

[1097] A "means for translating news articles into multiple languages" is a translation technique for converting the content of a news article into a different language.

[1098] "Means for displaying on a display in a physical store" refers to a method for displaying the generated illustration on a display installed in a physical store.

[1099] "Means for enabling users to select display options" refers to technology that provides an interface that allows users to select their preferred display format, language, and theme.

[1100] "Visual display" refers to presenting information in a format that is easy to understand visually.

[1101] "Multilingual support" refers to the ability to provide information in multiple languages.

[1102] The present invention is a system for automatically visualizing news articles, providing multilingual support and customizable display options. Specific embodiments are described below.

[1103] Server Processing

[1104] The server retrieves, analyzes, illustrates, and distributes news articles using the following hardware and software:

[1105] Hardware: Server

[1106] Software: News API, Google Translate API, Natural Language Processing (NLP) technology, Matplotlib

[1107] 1. Retrieving news articles

[1108] The server periodically retrieves the latest news articles from the Internet using a news API.

[1109] 2. News article analysis

[1110] NLP technology is used to extract key keywords and categories from retrieved news articles. For example, keywords such as "Shibuya Ward, Tokyo" and "incident" are extracted from an article titled "An incident occurred in Shibuya Ward, Tokyo."

[1111] 3. Select an illustrated template

[1112] Based on the extracted category, an appropriate diagram template is automatically selected. For example, for the category "incidents," a map or timeline template is selected.

[1113] 4. Generating illustrations

[1114] Based on the selected template, we generate illustrations that visually represent the content of the news article, using Matplotlib to create maps and timeline graphs.

[1115] 5. Generating illustrated variations

[1116] Generate one or more illustrative variations, such as dark mode, different languages, and different font sizes.

[1117] 6. News article translation

[1118] Use the Google Translate API to translate news articles into other languages, for example, from English to Japanese, or from Japanese to English.

[1119] 7. Saving and distributing illustrated variations

[1120] The generated illustration variations are stored in a database and prepared for distribution to the user's terminal.

[1121] Terminal handling

[1122] The terminal (a display in a physical store or the user's mobile device) displays the received news article and illustrations, which can be customized according to the user's specifications.

[1123] Hardware: Digital signage displays in physical stores, users' smartphones

[1124] Software: News viewing application

[1125] 1. Reading the news

[1126] When a user opens the news app, the latest news articles are displayed along with corresponding illustrations. The device caches the data retrieved from the server to provide a smooth display.

[1127] 2. Customizing the illustrations

[1128] From the app's settings screen, users can select the display format of the illustrations (e.g., dark mode, language, font size). When the user changes their settings, the device selects and displays the best version of the illustration from the saved variations.

[1129] 3. Display of illustrations

[1130] The most appropriate illustrations will be displayed based on the user's selection, for example, if dark mode is selected, the illustrations will also be visually displayed in dark mode.

[1131] Specific examples

[1132] Showing news articles:

[1133] Title: "The incident that occurred in Shibuya Ward, Tokyo"

[1134] Content: "Police are continuing their investigation into the incident that occurred last night in Shibuya Ward, Tokyo..."

[1135] Translated text: "An incident occurred last night in Shibuya, Tokyo. The police are continuing their investigation..."

[1136] Generated visualization image:

[1137] It will be saved with the file name visual_20230405_093000.png and displayed on the digital signage.

[1138] Prompt Sentence Examples

[1139] Use the News API to retrieve the latest news articles and then use the Google Translate API to translate the content into English. Then use Matplotlib to generate an image that visually displays the translated content. The generated image will then be displayed on a digital signage display in the store. Write a Python program that implements this process.

[1140] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1141] Step 1:

[1142] Get news articles

[1143] The server periodically retrieves the latest news articles from the Internet using a news API. The input is a news API request, and the output is the retrieved news article data. Specifically, the server sends an HTTP request to the news API and receives news article data in JSON format as a response.

[1144] Step 2:

[1145] News article analysis

[1146] The server uses natural language processing (NLP) techniques to extract key keywords and phrases from retrieved news articles. The input is the retrieved news article data, and the output is the extracted keywords and categories. Specifically, the server uses an NLP library (e.g., spaCy) to analyze the content of the article and extract important keywords and categories.

[1147] Step 3:

[1148] Select an illustrated template

[1149] The server selects an appropriate illustration template based on the extracted category. The input is the extracted category, and the output is the selected illustration template. Specifically, the server searches for templates corresponding to the category from the template library and selects the most appropriate template.

[1150] Step 4:

[1151] Generating illustrations

[1152] The server visualizes the contents of the news article based on the selected template. The input is the content of the news article and the selected template, and the output is the generated illustration. Specifically, the server uses a graph drawing library such as Matplotlib to create the illustration according to the template.

[1153] Step 5:

[1154] Generate illustrated variations

[1155] The server generates variations of the generated illustration, such as different themes (e.g., dark mode), different languages, and different font sizes. The input is a basic illustration, and the output is multiple variations of the illustration. Specifically, the server creates multiple variations with different themes, languages, and font sizes according to the set parameters.

[1156] Step 6:

[1157] News article translation

[1158] The server uses the Google Translate API to translate the content of news articles into multiple languages. The input is the content of the news article written in the original language, and the output is the translated content of the news article. Specifically, the server sends a translation request to the Google Translate API and receives the translation result as a response.

[1159] Step 7:

[1160] Saving and distributing illustrated variations

[1161] The server saves the generated illustration variations in a database and prepares them for distribution to user devices. The input is the illustration variation, and the output is the reference information for the saved illustration and a status ready for distribution. Specifically, the server saves the generated illustration variations in a database and generates and stores reference information such as a URL.

[1162] Step 8:

[1163] Reading the news

[1164] The user terminal displays the latest news articles and corresponding illustrations. The input is the news article and illustration data delivered from the server, and the output is the news and illustrations displayed on the terminal's display. In concrete terms, the terminal displays the news article and illustrations on the screen based on the received data.

[1165] Step 9:

[1166] Customizing the illustrations

[1167] The user terminal allows the user to set the display format of the illustration. The input is the user's selection, and the output is a customized illustration display based on the selection. In concrete terms, the terminal receives the display setting change from the user and selects and displays the optimal illustration variation from the saved variations.

[1168] Step 10:

[1169] Viewing the illustration

[1170] The user terminal displays the illustration based on the selected display format. The input is customized illustration data, and the output is the illustration displayed on the terminal display. In specific operation, the terminal selects and displays an appropriate illustration variation according to the user's settings.

[1171] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1172] This invention combines an emotion engine with a system that acquires, analyzes, and visualizes news articles, enabling the system to recognize user emotions and dynamically customize the display of illustrations based on those emotions. To implement this system, it is necessary to execute the following steps: news article acquisition, analysis, illustration generation, variation creation, emotion recognition, customization, storage, and distribution. The specific flow of each process is explained below.

[1173] Server (backend) processing

[1174] 1. Retrieving news articles

[1175] The server periodically retrieves the latest news articles via RSS feeds or news APIs, along with basic information such as article title, body of text, publication date, and category.

[1176] 2. Preprocessing of news articles

[1177] The server analyzes the retrieved news articles using natural language processing (NLP) technology to extract key keywords and phrases, such as "Shibuya Ward, Tokyo" and "incident."

[1178] 3. Select a template by category

[1179] Based on the extracted keywords and phrases, the server automatically classifies the news articles into specific categories (e.g., incidents, economics, science, sports, etc.) and selects an appropriate illustration template accordingly.

[1180] 4. Generating illustrations

[1181] The server then creates an illustration of the news article based on the selected template, using map information and time-series data to generate a visually easy-to-understand illustration.

[1182] 5. Accessibility-aware variation generation

[1183] The server creates one or more variations of the illustration, such as dark mode, different languages ​​(e.g., English, Japanese), different text sizes, etc.

[1184] 6. Emotion Recognition by Emotion Engine

[1185] The server uses interaction data (camera, microphone, touch operations, etc.) from the user's device to allow the emotion engine to estimate the user's current emotional state (e.g., joy, sadness, surprise, anger, etc.).

[1186] 7. Emotion-based illustrated customization

[1187] The server selects the optimal illustration variation based on the user's emotional data obtained from the emotion engine and dynamically changes the illustration display format. For example, if the user feels sad, it will provide an illustration with a softer color scheme.

[1188] 8. Saving and distributing illustrations

[1189] The server stores the illustration variations selected based on emotion recognition in a database and delivers them to the user's terminal.

[1190] Terminal (client-side) processing

[1191] 1. Reading the news

[1192] When a user opens the news app, the device calls the server's API to retrieve the latest news articles and corresponding illustrations.

[1193] 2. Initial display of the diagram

[1194] The device temporarily caches the received news article and illustrations, and when the news article is opened, the corresponding illustration is initially displayed.

[1195] 3. Acquiring Emotion Data

[1196] The device collects user interaction data (camera footage, voice input, touch operations, etc.) and sends it to the emotion engine, which analyzes it to recognize the user's emotions.

[1197] 4. Customize display options

[1198] Based on the emotion recognition results, the device dynamically adjusts the display options of the illustrations. Users can also manually select display options (e.g., language, theme, font size) from the settings screen.

[1199] User interaction and experience

[1200] 1. Viewing news articles and illustrations

[1201] Users scroll through the list in the news app and select the article they are interested in. When they open the article, the corresponding illustration is displayed along with the details of the news article.

[1202] 2. Emotion-based change

[1203] While the user is browsing the article, the device recognizes the user's emotions in real time and dynamically changes the illustration display style based on the user's emotions. For example, if the user is feeling stressed, the illustration's color scheme will become softer.

[1204] 3. Change settings

[1205] Users can change the display options in the settings screen to suit their preferences, for example, switching from Japanese to English or increasing the font size.

[1206] 4. Use of Emotion History

[1207] The system records the user's emotional history and can provide more customized displays in the future based on the user's emotional patterns.

[1208] As a concrete example, consider the following scenario: When a user browses an "article about an incident" in a news app, the article contains content about an "incident that occurred in Shibuya Ward, Tokyo." The analysis engine extracts the keywords "Shibuya Ward, Tokyo" and "incident" and classifies the category as "incident." The server selects a template for the "incident category" and generates an illustration that combines map and timeline data. It then creates and saves multiple variations (e.g., for dark mode, English version, uppercase version).

[1209] When a user opens this news article on their device, a map and timeline illustration are displayed. At the same time, an emotion engine analyzes the user's facial expression and, if the user expresses sadness, the illustration's color scheme is softened.

[1210] In this way, the present invention not only displays the contents of news articles in a visually easy-to-understand manner, but also dynamically customizes the content according to the user's emotions, thereby enabling more personalized information to be provided.

[1211] The processing flow will be explained below.

[1212] Server (backend) processing

[1213] Step 1:

[1214] The server periodically queries an RSS feed or news API to retrieve new news articles, including basic information such as the article title, body of the article, publication date, and category.

[1215] Step 2:

[1216] The server analyzes the retrieved news articles using natural language processing (NLP) technology to extract key keywords and phrases, such as "Shibuya Ward, Tokyo" and "incident."

[1217] Step 3:

[1218] Based on the extracted keywords and phrases, the server automatically classifies news articles into specific categories (incidents, economics, science, sports, etc.).

[1219] Step 4:

[1220] The server selects an appropriate illustration template for each category, for example, a map or timeline template for an article in the incident category.

[1221] Step 5:

[1222] The server then creates an illustration of the news article based on the selected template, using map information and time-series data to generate a visually easy-to-understand illustration.

[1223] Step 6:

[1224] The server creates one or more illustrated variations for accessibility, including dark mode, different languages ​​(e.g., English, Japanese), different text sizes, etc.

[1225] Step 7:

[1226] The server uses interaction data (camera, microphone, touch operations, etc.) from the user's device to allow the emotion engine to estimate the user's current emotional state (joy, sadness, surprise, anger, etc.).

[1227] Step 8:

[1228] Based on the user's emotional data obtained from the emotion engine, the server selects the optimal illustration variation and dynamically changes the illustration display format. For example, if the user is feeling sad, it will provide an illustration with a softer color scheme.

[1229] Step 9:

[1230] The server stores the generated illustrations and their variations in a database and prepares them for distribution to user terminals.

[1231] Terminal (client-side) processing

[1232] Step 1:

[1233] When a user opens the news app, the device calls the server's API to retrieve the latest news articles and corresponding illustrations.

[1234] Step 2:

[1235] The device temporarily caches received news articles and illustrations to provide smooth display.

[1236] Step 3:

[1237] A user scrolls through the news list and taps on an article they are interested in. The article details screen opens, along with a corresponding illustration.

[1238] Step 4:

[1239] The device collects user interaction data (camera footage, voice input, touch operations, etc.) and sends it to the emotion engine, which analyzes it to recognize the user's emotions.

[1240] Step 5:

[1241] Based on the emotion recognition results, the device dynamically adjusts the display options of the illustrations. Users can also manually select display options (e.g., language, theme, font size) from the settings screen.

[1242] User interaction and experience

[1243] Step 1:

[1244] Users scroll through the news list in a news app, find an article that interests them, and tap on it.

[1245] Step 2:

[1246] When you open an article, you'll see details about the news story along with visually compelling illustrations, such as a map showing the location of the incident and a timeline of chronological events.

[1247] Step 3:

[1248] While the user is browsing the article, the device recognizes the user's emotions in real time and dynamically changes the illustration display style based on the user's emotions. For example, if the user is feeling stressed, the illustration's color scheme will become softer.

[1249] Step 4:

[1250] Users can change the display options in the settings screen to suit their preferences, for example, switching from Japanese to English or increasing the font size.

[1251] Step 5:

[1252] The system records the user's emotional history and can provide more customized displays in the future based on the user's emotional patterns.

[1253] As a concrete example, consider the following scenario: When a user browses an "article about an incident" in a news app, the article contains content about an "incident that occurred in Shibuya Ward, Tokyo." The analysis engine extracts the keywords "Shibuya Ward, Tokyo" and "incident" and classifies the category as "incident." The server selects a template for the "incident category" and generates an illustration that combines map and timeline data. It then creates and saves multiple variations (e.g., for dark mode, English version, uppercase version).

[1254] When a user opens this news article on their device, a map and timeline illustration are displayed. At the same time, an emotion engine analyzes the user's facial expression and, if the user expresses sadness, the illustration's color scheme is softened.

[1255] In this way, the present invention not only displays the contents of news articles in a visually easy-to-understand manner, but also dynamically customizes the content according to the user's emotions, thereby enabling more personalized information to be provided.

[1256] Example 2

[1257] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1258] Conventional news article delivery systems have had difficulty providing information tailored to individual users' emotional states. Furthermore, they were unable to dynamically customize the content of news articles to match the user's emotional state. As a result, the user experience was static, potentially leaving some users dissatisfied.

[1259] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring news articles, means for analyzing the acquired news articles using natural language processing technology and extracting major keywords and categories, means for selecting an appropriate illustration template based on the extracted keywords and categories, means for generating an illustration that visualizes the content of the news article based on the selected template, means for creating one or more illustration variations taking accessibility into consideration, means for collecting user interaction data and recognizing emotions, means for dynamically customizing the illustration variations based on the recognized emotions, and means for saving the generated illustration variations and delivering them to the user terminal. This dynamically customizes the visual display of the news article, enabling more personalized information to be provided based on the user's emotions.

[1260] "Means for obtaining news articles" refers to the function of periodically collecting news articles using a news API or RSS feed.

[1261] "Means of analyzing acquired news articles using natural language processing technology and extracting key keywords and categories" refers to a function that analyzes acquired news articles using natural language processing technology such as morphological analysis and automatically extracts key keywords and categories.

[1262] "Means for selecting an appropriate illustrated template based on extracted keywords and categories" is a function that automatically selects the most appropriate template from multiple pre-prepared illustrated templates based on analyzed keyword and category information.

[1263] "Means for generating illustrations that visualize the contents of a news article based on a selected template" refers to a function that uses a selected illustration template to generate illustrations that represent the contents of a news article in a visually easy-to-understand format, such as a map or timeline.

[1264] "Means for creating one or more variations of illustrations with accessibility in mind" refers to the ability to create variations of the generated illustrations in different formats (e.g., dark mode, different languages, different font sizes) that take into account user visibility and ease of use.

[1265] "Means for collecting user interaction data and recognizing emotions" refers to a function that collects data such as the user's camera footage, voice input, and touch operations, and analyzes it to estimate the user's current emotional state.

[1266] "Means for dynamically customizing illustration variations based on recognized emotions" is a function that selects the optimal illustration variation based on the user's emotional data and changes the color scheme and design of the illustration in real time.

[1267] The "means for saving the generated illustration variations and distributing them to the user's terminal" is a function for saving the generated multiple illustration variations in a database and distributing them to the user's terminal via the Internet.

[1268] MODE FOR CARRYING OUT THE INVENTION

[1269] This invention aims to provide a system for acquiring, analyzing, and illustrating news articles, and to provide a dynamically customized visual display according to the user's emotions. This system uses the following hardware and software:

[1270] Hardware

[1271] Server: Responsible for news article acquisition, analysis, illustration generation, emotion recognition, customization, storage and distribution.

[1272] User devices: Responsible for displaying news articles, collecting and transmitting emotion data, and dynamically changing the display. These devices include smartphones, tablets, and PCs.

[1273] software

[1274] Natural Language Processing (NLP) software: Use libraries such as SpaCy and NLTK to analyze news articles and extract key keywords and categories.

[1275] News API: Uses an API such as Google News API to make an HTTP request to retrieve the latest news articles.

[1276] Illustration generation library: Uses D3.js, Google Maps API, etc. to create visually easy-to-understand illustrations of the contents of news articles.

[1277] Emotion recognition engine: Analyzes user emotions using Microsoft Azure Emotion API, etc.

[1278] Example

[1279] 1. Acquiring and analyzing news articles

[1280] The server periodically retrieves news articles using the Google News API or RSS feeds. The retrieved articles are analyzed using natural language processing technology to extract key keywords and categories. For example, a request like https: / / newsapi.org / v2 / top-headlines?country=jp&apiKey=YOUR_API_KEY is sent.

[1281] 2. Template selection and diagram generation

[1282] Based on the extracted keywords and categories, the server selects an appropriate illustration template. Based on the selected template, it generates illustrations such as maps and timeline data using the Google Maps API and D3.js. For example, if the article is about an incident that occurred in Shibuya Ward, Tokyo, it selects a template in the incident category and illustrates the map and timeline data.

[1283] 3. Accessible Variation Generation

[1284] The server creates multiple variations of the generated illustrations, such as dark mode, different languages ​​(English, Japanese), different font sizes, etc. For example, it applies dark mode styles with CSS and adds different language labels to the illustration data.

[1285] 4. Emotional Data Collection and Recognition

[1286] While the user is browsing a news article, the device collects interaction data from the camera and microphone and sends it to the server. The server's emotion recognition engine analyzes the user's emotional state. For example, the user's emotional state can be recognized as "happiness," "sadness," "surprise," "anger," etc. based on facial expressions and voice analysis.

[1287] 5. Dynamic customization based on emotions

[1288] The server selects the most appropriate illustration variation based on the user's recognized emotion and changes the illustration's color scheme and design in real time. For example, if the user feels sad, it selects an illustration with a soft color scheme.

[1289] 6. Saving and distributing illustrations

[1290] The selected illustration variations are stored in a database and delivered to the user's device via the Internet, using a real-time database such as Firebase to transmit data in real time.

[1291] Specific examples and prompts for the generative AI model

[1292] As a specific example, consider a scenario in which a user views an article about an incident that occurred in Shibuya Ward, Tokyo.

[1293] When a user opens the news app, an article about an incident that occurred in Shibuya Ward, Tokyo, is retrieved from the server. The analysis engine extracts the keywords "Shibuya Ward, Tokyo" and "incident" from the article and classifies it as an "incident" category. The server selects a template that combines a map and a timeline to visualize the news content. The generated variations (e.g., dark mode, English version, uppercase version) are stored in a database.

[1294] When a user opens an article on their device, a map and timeline illustration are displayed. At the same time, the emotion engine analyzes the user's facial expressions and, if the user is expressing sadness, the illustration's color scheme is changed to a softer tone.

[1295] Example prompts to input to a generative AI model:

[1296] A user is viewing an article about an incident that occurred in Shibuya Ward, Tokyo. The article's category is "Incident," and an illustration using map and timeline data should be displayed. If the user's emotion is "Sadness," change the illustration's color scheme to a softer tone.

[1297] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1298] Step 1: Get the news article

[1299] Input: Periodically send requests for news articles using a news API or RSS feed.

[1300] Data processing: The server sends an HTTP request to the API endpoint and receives the returned data in JSON format.

[1301] Output: Data including news article title, body of text, publication date, category, etc.

[1302] Specific operation: Execute a request such as https: / / newsapi.org / v2 / top-headlines?country=jp&apiKey=YOUR_API_KEY.

[1303] Step 2: Preprocessing the news article

[1304] Input: News article data obtained in step 1.

[1305] Data processing: The server analyzes the news articles using natural language processing techniques (e.g., SpaCy, NLTK) to extract key keywords and phrases.

[1306] Output: Extracted keywords and phrases (e.g., "Shibuya Ward, Tokyo" and "incident").

[1307] Specific operation: Performs morphological analysis and extracts important words and phrases in the sentence.

[1308] Step 3: Select a template by category

[1309] Input: Keywords or phrases extracted in step 2.

[1310] Data processing: The server classifies news articles into specific categories based on extracted keywords and phrases, and selects appropriate illustration templates for each category.

[1311] Output: The selected diagram template (e.g., a template for incident categories).

[1312] Specific operation: Based on the analysis results, categorize them into categories such as "incidents," "economy," and "science," and select the corresponding template for each.

[1313] Step 4: Generate the diagram

[1314] Input: The illustrated template selected in step 3 and the content of the news article.

[1315] Data processing: The server generates illustrations that visualize the contents of news articles based on the selected template. Maps and timeline data are created using Google Maps API, D3.js, etc.

[1316] Output: Visualized and illustrated data.

[1317] Specific functions: Display the locations of incidents on a map and create a chronological timeline.

[1318] Step 5: Generate variations with accessibility in mind

[1319] Input: Illustrated data generated in step 4.

[1320] Data processing: The server creates variations of the generated illustrations in different formats (e.g., dark mode, different languages, changed font size).

[1321] Output: Multiple illustrated variations.

[1322] What it does: Changes CSS to apply dark mode styles and adds labels in different languages ​​to illustrated data.

[1323] Step 6: Emotion data collection and emotion recognition

[1324] Input: Interaction data (camera feeds, voice inputs, touch actions, etc.) while the user is viewing an article.

[1325] Data processing: The interaction data collected by the device is sent to the server, where the server's emotion recognition engine analyzes the emotional state.

[1326] Output: User emotion data (e.g., happy, sad, surprised, angry).

[1327] Specific operation: The user's facial expressions and voice are collected through a camera and microphone and analyzed using an emotion recognition model.

[1328] Step 7: Emotion-based illustration customization

[1329] Input: User emotion data recognized in step 6.

[1330] Data processing: The server selects the optimal illustration variation based on the emotion data and dynamically changes the color scheme and design of the illustration.

[1331] Output: Customized illustrated data.

[1332] Specific behavior: If the user expresses sadness, change the color scheme of the illustration to a softer tone.

[1333] Step 8: Save and distribute your diagrams

[1334] Input: Customized diagram data from step 7.

[1335] Data processing: The server stores the customized illustrations in a database and delivers them to the user's device.

[1336] Output: Illustrated data stored on the user's device.

[1337] What it does: Uses a real-time database such as Firebase to send customized diagrams over the internet.

[1338] Step 9: Initial display of news article and illustration

[1339] Input: News article and illustration data distributed in step 8.

[1340] Data processing: The device caches the news article and illustrations it receives, and initially displays the illustrations when the news article is opened.

[1341] Output: News article and illustration displayed on the user's terminal.

[1342] What it does: When you open the news app, it displays the latest news articles and their illustrations.

[1343] Step 10: Customize display options

[1344] Input: User settings changes (language, theme, text size, etc.).

[1345] Data processing: The device dynamically adjusts the display options of the illustration based on user settings changes.

[1346] Output: Customized graphical display.

[1347] Specific operation: The user changes the display options in the settings screen to display the diagram according to their preference.

[1348] (Application example 2)

[1349] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1350] Conventional advertising delivery systems have difficulty customizing ads based on user emotions, limiting their visibility and effectiveness. Furthermore, there is no technology for summarizing news-like information and incorporating it into ads, making it difficult for ads to be interesting to users. Therefore, there is a need for a system that can provide more personalized information by recognizing user emotions in real time and dynamically changing advertising design and content based on those emotions.

[1351] The specification processing by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring a news article, means for analyzing the news article and extracting main keywords and categories, means for selecting an appropriate illustration template based on the extracted category, means for illustrating the content of the news article based on the selected template, means for generating one or more illustration variations, means for saving the generated illustration variations and delivering them to the user terminal, means for recognizing the user's emotions and dynamically changing the display format of the illustration based on the emotions, means for summarizing the news article and incorporating it into the generated advertising content and displaying it, and means for dynamically changing the advertising design based on the user's emotions. This enables dynamic advertising delivery linked to the user's emotions, making it possible to dramatically improve the visibility and effectiveness of advertising.

[1352] A "news article" is a set of documents or information that is reported as news.

[1353] An "advertising distribution system" is a system for distributing advertisements for products and services to users.

[1354] "Emotion recognition" is a technology that analyzes and recognizes a user's emotions from interaction data such as camera and voice.

[1355] An "illustration" is a graphical representation used to make textual information easier to understand visually.

[1356] A "template" is a standard format that serves as the basis for generating diagrams and displaying them.

[1357] "Illustration Variants" are different versions of an illustration that are generated in multiple formats, such as different languages, themes, font sizes, etc.

[1358] "Dynamic change" means making changes automatically in real time according to the situation or conditions.

[1359] "User terminal" refers to a computer device used by a user, such as a smartphone, tablet, or PC.

[1360] "Customization" means changing settings according to the needs and preferences of a particular user.

[1361] A "generative AI model" is an artificial intelligence model that learns from large amounts of data and generates useful output for new data.

[1362] A "prompt sentence" is an instruction sentence that is input to a generative AI model to obtain a specific output.

[1363] The present invention is a dynamic advertising delivery system based on user emotions, and provides technology for customizing advertising content through the acquisition, analysis, and visualization of news articles.

[1364] Server (backend) processing

[1365] The server periodically retrieves news articles through a news API. The retrieved news articles are analyzed using natural language processing (NLP) techniques to extract key keywords and categories. An appropriate illustration template is then selected based on the extracted categories, and the news article is visualized based on the selected template. One or more illustration variations are generated, which may include different languages, themes, and font sizes.

[1366] Next, the emotion engine recognizes the user's emotions. This recognition is performed in real time from interaction information collected using a camera and microphone. Based on the emotion recognition, the display format of the illustrations is dynamically changed, and a summary of the news article is incorporated into the generated advertisement content. The generated advertisement content is then delivered to the user's device in an optimized design. The following specific technologies are used in this series of processes:

[1367] Natural Language Processing (NLP) technology: Keyword analysis of retrieved news articles

[1368] Emotion Recognition Engine: Analyzes user emotions

[1369] Illustration generation engine: Generate illustrations based on keywords

[1370] Database: storing generated illustrations and ad variations

[1371] User terminal (client side) processing

[1372] The user's device displays the news article and corresponding illustrations received from the server. Web technologies such as HTML and CSS are used for the display, allowing for dynamic customization. In addition, the user's device captures the user's emotional data through a camera and microphone and sends it to the server. Based on feedback from the server, the design and color scheme of the advertisement are changed in real time.

[1373] User interaction and experience

[1374] When a user opens the ad display app, they are shown ads that summarize news-like information. For example, an ad for a new smartphone product summarizes and visualizes the latest trends in the smartphone industry. As the user views the ad, the design and color scheme of the ad dynamically changes depending on the user's emotions, providing a more personalized advertising experience.

[1375] Specific examples of prompts are as follows:

[1376] "Summarize the latest smartphone industry trending news and create a visually compelling illustration. If the user is expressing joy, use a brighter color scheme to create a positive impression."

[1377] Using this prompt, the generative AI model generates advertising content, resulting in advertising designs that adapt to the user's emotions.

[1378] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1379] Step 1:

[1380] The server retrieves the latest news articles using the news API. In this step, an API request is sent and JSON-formatted data is received, including basic information such as the article title, body, publication date and time, and category. The input is the request to the news API, and the output is the JSON data of the retrieved news articles.

[1381] Step 2:

[1382] The server analyzes the retrieved news articles and extracts key keywords and categories. In this step, natural language processing (NLP) techniques are used to analyze the article text and extract important keywords and phrases. The input is the text data of the news article, and the output is the extracted keyword and category data.

[1383] Step 3:

[1384] The server selects an appropriate illustration template based on the extracted category. In this step, it selects the illustration template that best suits the extracted category from a predefined template library. The input is the extracted category data, and the output is the selected illustration template.

[1385] Step 4:

[1386] The server then creates an illustration of the news article content based on the selected template. In this step, the text and keywords are converted into a visually easy-to-understand illustration. Specifically, the keywords are arranged in graphs, charts, maps, etc. The input is the news article text and keyword data, and the output is the generated illustration.

[1387] Step 5:

[1388] The server generates one or more variations of the illustration. In this step, the illustration is generated in multiple formats, such as different languages, themes, font sizes, etc. The input is the generated illustration, and the output is each variation of the illustration.

[1389] Step 6:

[1390] The server saves the generated illustration variations and delivers them to the user terminal. In this step, the illustration variations are saved in a database and a link is generated for users to access them. The input is the generated illustration variations, and the output is the link information for the saved illustrations.

[1391] Step 7:

[1392] The terminal displays the received news article and related illustrations. In this step, the news article and illustrations retrieved from the server are displayed on the screen using HTML and CSS. The input is the news article and illustration received from the server, and the output is a display screen that the user can view.

[1393] Step 8:

[1394] The device acquires the user's emotional data through a camera and microphone and sends it to the server. In this step, the device analyzes the user's facial expressions and tone of voice to recognize their emotions. The input is camera video and audio data, and the output is the recognized emotional data.

[1395] Step 9:

[1396] The server uses an emotion engine to recognize the user's emotion and dynamically change the display format of the illustration based on the emotion. In this step, the color scheme and design of the illustration are changed based on the recognized emotion data. The input is the user's emotion data, and the output is a dynamically changed illustration.

[1397] Step 10:

[1398] The server then summarises the news article and incorporates it into the generated advertisement content, and displays it. Specifically, it incorporates the news article summary into the advertisement in a design that adapts to the user's emotions. The input for this step is the news article summary and an emotion-based design template, and the output is optimized advertisement content.

[1399] Step 11:

[1400] The device dynamically changes the ad design based on the user's emotions. In this step, the color scheme and layout of the ad are adjusted in real time to match the user's emotions. The input is the optimized ad data from the server, and the output is the final ad display.

[1401] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1402] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1403] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1404] [Fourth embodiment]

[1405] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1406] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1407] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1408] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1409] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1410] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1411] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1412] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1413] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1414] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1415] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1416] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1417] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1418] This invention is a system that automatically visualizes news articles. It analyzes the content of news articles and generates and distributes appropriate illustrations, allowing readers to easily and quickly understand the information. To implement this system, it is necessary to execute the following steps: news article acquisition, analysis, illustration generation, variation creation, storage, and distribution. The specific flow of each process is explained below.

[1419] Server (backend) processing

[1420] 1. Retrieving news articles

[1421] The server periodically retrieves the latest news articles via RSS feeds or news APIs, capturing information such as the article title, body of text, publication date, and category.

[1422] 2. Preprocessing of news articles

[1423] Natural language processing (NLP) techniques are used to extract key keywords and phrases from retrieved news articles, and based on the results of this analysis, articles are classified into specific categories (e.g., crime, economy, science, sports, etc.).

[1424] 3. Select a template by category

[1425] Depending on the category, the server will select an appropriate illustration template, for example, a map or timeline template for an incident article.

[1426] 4. Generating illustrations

[1427] The server then creates an illustration of the news article based on the selected template, using map information and time series data to generate a visually easy-to-understand illustration.

[1428] 5. Accessibility-aware variation generation

[1429] Create one or more illustrated variations, such as dark mode, different languages, different text sizes, etc.

[1430] 6. Saving and distributing illustrations

[1431] The generated illustrations and their variations are stored in a database and prepared for distribution to user terminals.

[1432] Terminal (client-side) processing

[1433] 1. Reading the news

[1434] When a user opens the news app, the latest news articles are displayed along with corresponding illustrations. The device caches the data retrieved from the server to provide a smooth display.

[1435] 2. Customizing the illustrations

[1436] From the app's settings screen, users can select the display format of the illustrations (e.g., dark mode, language, font size). When the user changes their settings, the device selects and displays the best version of the illustration from the saved variations.

[1437] 3. Display of illustrations

[1438] The most appropriate illustrations will be displayed based on the user's selection, for example, if dark mode is selected, the illustrations will also be visually displayed in dark mode.

[1439] User interaction and experience

[1440] 1. Viewing news articles and illustrations

[1441] Users can browse the news list in the news app and select an article they are interested in. When they open an article, a corresponding illustration is displayed at the same time, allowing them to visually understand the content.

[1442] 2. Change settings

[1443] Users can change the display options in the settings screen to suit their preferences, such as increasing the font size or changing the display language.

[1444] 3. Multilingual support available

[1445] Users can change the language of articles and illustrations by changing the settings, allowing them to be displayed in different languages, which allows for a wide range of readership.

[1446] As a concrete example, consider the following scenario: When a user browses an "article about an incident" in a news app, the article contains content about an "incident that occurred in Shibuya Ward, Tokyo." The analysis engine extracts the keywords "Shibuya Ward, Tokyo" and "incident" and classifies the category as "incident." The server selects a template for the "incident category" and generates an illustration that combines a map and time-series data. Furthermore, it creates multiple variations (e.g., for dark mode, English version, uppercase version) and stores them in a database.

[1447] When a user opens this news article on their device, they will see the map and timeline illustration. If they select dark mode from the settings screen, the settings will be reflected immediately and the illustration will change to the dark mode version.

[1448] In this way, the present invention displays the contents of news articles in a visually easy-to-understand manner, meeting the needs of a wide variety of users.

[1449] The processing flow will be explained below.

[1450] Server (backend) processing

[1451] Step 1:

[1452] The server periodically queries the RSS feed or news API to retrieve new news articles, along with basic information such as the article title, body of the article, publication date, and category.

[1453] Step 2:

[1454] The server analyzes the retrieved news articles using natural language processing (NLP) technology to extract key keywords and phrases, such as "Shibuya Ward, Tokyo" and "incident."

[1455] Step 3:

[1456] The server automatically classifies news articles into specific categories (incidents, economics, science, sports, etc.) based on the extracted keywords and phrases.

[1457] Step 4:

[1458] The server selects an appropriate illustration template for each category, for example, a map and timeline template for articles in the incident category.

[1459] Step 5:

[1460] The server then creates an illustration of the news article based on the selected template, using map information and time-series data to generate a visually easy-to-understand illustration.

[1461] Step 6:

[1462] For accessibility reasons, the server creates one or more variations of the illustration, including dark mode, different languages ​​(e.g., English, Japanese), different font sizes, etc.

[1463] Step 7:

[1464] The server stores the generated illustrations and their variations in a database and prepares them for distribution to user terminals.

[1465] Terminal (client-side) processing

[1466] Step 1:

[1467] When a user opens the news app, the device calls the server's API to retrieve the latest news articles and corresponding illustrations.

[1468] Step 2:

[1469] The device temporarily caches received news articles and illustrations to provide smooth display.

[1470] Step 3:

[1471] A user scrolls through the news list and taps on an article they are interested in. The article details screen opens, along with a corresponding illustration.

[1472] Step 4:

[1473] The device's settings screen allows the user to select display options (language, theme, text size, etc.).

[1474] Step 5:

[1475] Depending on the user's settings, the device retrieves the optimal illustration variation from the server and changes the display format. For example, if dark mode is selected, the illustration will also be displayed in dark mode.

[1476] User interaction and experience

[1477] Step 1:

[1478] Users scroll through the news list in a news app, find an article that interests them, and tap on it.

[1479] Step 2:

[1480] When you open an article, you'll see details about the news story along with visually compelling illustrations, such as a map showing the location of the incident and a timeline of chronological events.

[1481] Step 3:

[1482] The user accesses the settings screen and changes display options, for example, changing the language from English to Japanese or switching to dark mode.

[1483] Step 4:

[1484] Your preferences are respected and articles and illustrations are displayed in the format of your choice, providing the best possible display based on your visual needs and preferences.

[1485] Example 1

[1486] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1487] Traditional news articles are often presented in text only, which means it takes time for readers to quickly and easily understand the information. Furthermore, the lack of visual information makes it difficult to understand particularly complex news content. Furthermore, the inability to customize the content to meet users' visual preferences and needs poses challenges in terms of accessibility and usability.

[1488] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1489] In this invention, the server includes means for acquiring news articles, means for analyzing the acquired news articles and extracting major keywords and categories, means for selecting an appropriate illustration template based on the extracted category, means for illustrating the content of the news article based on the selected template, means for generating multiple variations with different themes, languages, font sizes, etc., and means for saving the generated illustration variations and distributing them to user terminals. This makes it possible to display the content of news articles in a visually easy-to-understand manner and meet the needs of a variety of users.

[1490] A "news article" is a piece of text that describes the latest information or events distributed by a news organization or information provider.

[1491] A "server" is a central computer system that stores, processes, and distributes data.

[1492] A "terminal" is a client device that is directly operated by a user, and includes smartphones, tablets, PCs, etc.

[1493] An "acquisition means" is a process or device for automatically collecting news articles from external sources.

[1494] "Means for analysis" refers to the processing or device that analyzes the content of the acquired news articles using techniques such as natural language processing and extracts key keywords and categories.

[1495] An "illustrated template" is a pre-designed type or format for visually displaying the content of a news article.

[1496] A "means for visualizing" is a process or device for visualizing the content of a news article based on a selected graphical template.

[1497] "Means for generating variations" refers to a process or device for creating multiple formats from a basic illustration, with different themes, languages, font sizes, etc., according to the user's needs.

[1498] The "storing means" refers to a process or device for permanently storing the generated illustration variations in a database or the like.

[1499] The "distribution means" refers to a process or device for transmitting the saved illustration variations to a user terminal.

[1500] The "displaying means" refers to a process or device for displaying the news article and related illustrations received by the user terminal on the screen.

[1501] "Display options" are settings that the user can select to change the display format of the illustration.

[1502] "Different languages" refers to a format in which illustrated variations are generated in multiple languages, such as Japanese and English.

[1503] A "theme" is a style setting that changes the appearance and color of an illustration.

[1504] "Text size" is a setting item that changes the size of the text in the illustration.

[1505] The present invention is a system that automatically visualizes news articles, in which the server, terminal, and user elements work together to analyze the content of the news article, appropriately illustrate the information, generate variations, and deliver them to the user.

[1506] Server Processing

[1507] The server retrieves the latest news articles using an RSS feed or news API. Specifically, the server receives JSON-formatted data from a news provider using an HTTP request, and extracts information such as the article title, body text, publication date and category from that data. For example, the News API or Google News API may be used.

[1508] The server then applies natural language processing (NLP) techniques to the retrieved news articles. It uses Python libraries like Spacy and NLTK to extract key keywords and phrases from the articles and classify them into categories (e.g., news, economics, science, sports, etc.). This analysis involves techniques such as tokenization, named entity recognition (NER), and Term Frequency-Inverse Document Frequency (TF-IDF).

[1509] Based on the analysis results, the server classifies the news articles into specific categories and selects a corresponding illustration template for that category. This template refers to a type of visual display of the news content, such as a map, timeline, or graph. For example, D3.js or a geographic information system is used to create the map, and a library such as Chart.js is used to display the timeline.

[1510] Based on the selected template, the server creates a visual representation of the news article. For example, to display the geographic location of an incident, map data is retrieved from the OpenStreetMap API and plotted. Timeline data is parsed using the Datetime library as appropriate.

[1511] Additionally, the server generates multiple variations of the generated illustrations to improve their accessibility, such as dark mode, different languages ​​(using the Google Translate API), and different font sizes. Variation generation is implemented using CSS media queries and internationalization (i18n) libraries.

[1512] The generated diagrams and their variations are stored in a database such as MongoDB or MySQL, and then a service such as Firebase Cloud Messaging (FCM) is used to send notifications to user devices.

[1513] Terminal handling

[1514] When a user opens the news app, the device retrieves the latest news articles and illustrations from the server, and the retrieved data is cached in local storage using the Room library and SQLite to provide a smooth display.

[1515] Users can choose how the icons are displayed in the app's settings. Options include dark mode, different languages, and font sizes. User settings are saved in SharedPreferences and UserDefaults. When a setting is changed, the icons are instantly updated to reflect that setting.

[1516] The device will display the best illustration variation based on the user's settings. For example, if the user selects dark mode, the device will apply stylesheets and CSS classes to display dark mode illustrations.

[1517] User interaction and experience

[1518] Users can browse through a list of articles within the news app and select the news article they are interested in. When they tap on an article, a related illustration is displayed at the same time, helping them to understand the content intuitively. For example, if they select a news article about an incident that occurred in Shibuya Ward, Tokyo, an illustration will appear that provides geographical and chronological information about the incident.

[1519] By changing the display options in the settings screen, users can set the display format to suit their preferences. They can also easily increase the font size and change the display language, making it possible to meet a variety of needs.

[1520] Example prompt sentence:

[1521] "Visualize a news story about an incident that occurred in Shibuya Ward, Tokyo, and generate an illustration using maps and time series data. Create multiple variations of the illustration, including a dark mode version, English version, and uppercase version."

[1522] In this way, the present invention can display the contents of news articles in a visually easy-to-understand manner and meet the needs of a wide variety of users.

[1523] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1524] Step 1: Get the news article

[1525] The server retrieves news articles from external sources. Specifically, it periodically retrieves the latest news articles using RSS feeds or news APIs (e.g., NewsAPI, Google News API). This process involves sending an HTTP request and receiving a response in JSON format. The input is the endpoint URL of the news API, and the output is JSON data that includes the article title, body text, publication date and time, category, etc.

[1526] Step 2: Preprocessing the news article

[1527] The server analyzes the retrieved news articles. It uses Python natural language processing libraries (e.g., Spacy, NLTK) to extract key keywords and phrases. The input is the text of the news article obtained in step 1, and the output is the extracted keywords and categories. Specifically, it tokenizes the article text and performs named entity recognition (NER) and TF-IDF (Term Frequency-Inverse Document Frequency).

[1528] Step 3: Select a template by category

[1529] The server classifies the news article into a specific category based on the analysis results and selects the corresponding illustration template. The input is the category extracted in step 2, and the output is the illustration template corresponding to the category. For example, for the incident category, a map or timeline template is selected.

[1530] Step 4: Generate the diagram

[1531] The server visualizes the content of the news article based on the selected template. For example, if geographic information is displayed, the OpenStreetMap API is used to plot the locations of incidents. If time series data is included, it is parsed using the Datetime library to generate a timeline. The input is the news article text and the selected template, and the output is the generated illustration. Specific operations include retrieving map data and generating maps in HTML or SVG format.

[1532] Step 5: Generate accessible variations

[1533] The server generates variations of different themes, languages, and font sizes to improve the accessibility of the generated illustrations. The input is the illustration generated in step 4, and the output is multiple variations of the illustration. Specific behaviors include applying dark mode using CSS media queries and changing the illustration language using the Google Translate API.

[1534] Step 6: Save and distribute your diagrams

[1535] The server stores the generated diagrams and their variations in a database, using, for example, MongoDB or MySQL. It also uses a service such as Firebase Cloud Messaging (FCM) to send notifications to user devices. The input is the diagram variations, and the output is the diagram stored in the database and the notification sent. Specifically, the data is saved using an INSERT query, and notifications are sent using the FCM API.

[1536] Step 7: Read the news

[1537] The user opens the news app to view the latest news articles and illustrations. The device caches and displays the data obtained from the server. The input is the news article and illustration data sent from the server, and the output is the news article and illustration displayed on the device screen. Specifically, the data is obtained via an HTTP request, and the cached data is managed using the Room library and SQLite.

[1538] Step 8: Customize your diagram

[1539] The user can select the display format of the illustrations (e.g., dark mode, language, font size) in the app's settings screen. The device saves the user's selection and reflects it in the display. The input is the user's settings, and the output is the updated display format. Specific behavior is to save the settings in SharedPreferences or UserDefaults and update the UI in real time.

[1540] Step 9: Displaying the diagram

[1541] The optimal illustration is displayed on the device based on the user's selection. For example, if dark mode is selected, the CSS class and style sheet are changed to display an illustration for dark mode. The input is the user's settings and illustration data, and the output is the illustration in the applied display format. Specifically, the HTML and CSS are dynamically updated.

[1542] In this way, the present invention can display the contents of news articles in a visually easy-to-understand manner and meet the needs of a wide variety of users.

[1543] (Application example 1)

[1544] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1545] Conventional news article distribution systems lack the ability to generate illustrations to aid visual understanding, and therefore do not meet the needs of brick-and-mortar stores, especially in environments where real-time information provision is required. Furthermore, the lack of multilingual support and interactive display options makes it difficult to effectively provide information to customers with diverse backgrounds. Given these circumstances, there is a need for a system that can visualize and instantly distribute news articles in brick-and-mortar stores, as well as provide multilingual support and customizable display options.

[1546] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1547] In this invention, the server includes means for acquiring news articles, means for analyzing the acquired news articles and extracting major keywords and categories, means for selecting an appropriate illustration template based on the extracted category, means for illustrating the content of the news article based on the selected template, means for generating one or more illustration variations, means for saving the generated illustration variations and distributing them to a user terminal, means for translating the news article into multiple languages, and means for displaying the generated illustration variations on a display in a physical store. This enables visual display of news articles and multilingual support in physical stores, improving the efficiency of providing information to customers.

[1548] "News Article" means a news article obtained from an internet source.

[1549] "Means of acquisition" refers to methods of automatically downloading news articles from the Internet or news feeds.

[1550] The "analysis method" refers to a technique that uses natural language processing technology to analyze the content of acquired news articles and extract key keywords and phrases.

[1551] "Major keywords and categories" refer to words that are important for understanding the content of a news article and classifications based on them.

[1552] The "selection means" is a method for automatically selecting the most suitable diagram template based on the analysis results.

[1553] An "illustrated template" is a predefined format for visually representing the content of a news article.

[1554] "Visualization tools" are techniques that use selected templates to generate graphs and charts to visually display the content of a news article.

[1555] "Illustration Variations" are illustrations generated in multiple formats, such as different display formats, languages, themes, font sizes, etc.

[1556] The "means for storing and distributing to the user terminal" is a technology for storing the generated illustration variations in a database and transmitting them to the user's terminal via a network.

[1557] A "means for translating news articles into multiple languages" is a translation technique for converting the content of a news article into a different language.

[1558] "Means for displaying on a display in a physical store" refers to a method for displaying the generated illustration on a display installed in a physical store.

[1559] "Means for enabling users to select display options" refers to technology that provides an interface that allows users to select their preferred display format, language, and theme.

[1560] "Visual display" refers to presenting information in a format that is easy to understand visually.

[1561] "Multilingual support" refers to the ability to provide information in multiple languages.

[1562] The present invention is a system for automatically visualizing news articles, providing multilingual support and customizable display options. Specific embodiments are described below.

[1563] Server Processing

[1564] The server retrieves, analyzes, illustrates, and distributes news articles using the following hardware and software:

[1565] Hardware: Server

[1566] Software: News API, Google Translate API, Natural Language Processing (NLP) technology, Matplotlib

[1567] 1. Retrieving news articles

[1568] The server periodically retrieves the latest news articles from the Internet using a news API.

[1569] 2. News article analysis

[1570] NLP technology is used to extract key keywords and categories from retrieved news articles. For example, keywords such as "Shibuya Ward, Tokyo" and "incident" are extracted from an article titled "An incident occurred in Shibuya Ward, Tokyo."

[1571] 3. Select an illustrated template

[1572] Based on the extracted category, an appropriate diagram template is automatically selected. For example, for the category "incidents," a map or timeline template is selected.

[1573] 4. Generating illustrations

[1574] Based on the selected template, we generate illustrations that visually represent the content of the news article, using Matplotlib to create maps and timeline graphs.

[1575] 5. Generating illustrated variations

[1576] Generate one or more illustrative variations, such as dark mode, different languages, and different font sizes.

[1577] 6. News article translation

[1578] Use the Google Translate API to translate news articles into other languages, for example, from English to Japanese, or from Japanese to English.

[1579] 7. Saving and distributing illustrated variations

[1580] The generated illustration variations are stored in a database and prepared for distribution to the user's terminal.

[1581] Terminal handling

[1582] The terminal (a display in a physical store or the user's mobile device) displays the received news article and illustrations, which can be customized according to the user's specifications.

[1583] Hardware: Digital signage displays in physical stores, users' smartphones

[1584] Software: News viewing application

[1585] 1. Reading the news

[1586] When a user opens the news app, the latest news articles are displayed along with corresponding illustrations. The device caches the data retrieved from the server to provide a smooth display.

[1587] 2. Customizing the illustrations

[1588] From the app's settings screen, users can select the display format of the illustrations (e.g., dark mode, language, font size). When the user changes their settings, the device selects and displays the best version of the illustration from the saved variations.

[1589] 3. Display of illustrations

[1590] The most appropriate illustrations will be displayed based on the user's selection, for example, if dark mode is selected, the illustrations will also be visually displayed in dark mode.

[1591] Specific examples

[1592] Showing news articles:

[1593] Title: "The incident that occurred in Shibuya Ward, Tokyo"

[1594] Content: "Police are continuing their investigation into the incident that occurred last night in Shibuya Ward, Tokyo..."

[1595] Translated text: "An incident occurred last night in Shibuya, Tokyo. The police are continuing their investigation..."

[1596] Generated visualization image:

[1597] It will be saved with the file name visual_20230405_093000.png and displayed on the digital signage.

[1598] Prompt Sentence Examples

[1599] Use the News API to retrieve the latest news articles and then use the Google Translate API to translate the content into English. Then use Matplotlib to generate an image that visually displays the translated content. The generated image will then be displayed on a digital signage display in the store. Write a Python program that implements this process.

[1600] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1601] Step 1:

[1602] Get news articles

[1603] The server periodically retrieves the latest news articles from the Internet using a news API. The input is a news API request, and the output is the retrieved news article data. Specifically, the server sends an HTTP request to the news API and receives news article data in JSON format as a response.

[1604] Step 2:

[1605] News article analysis

[1606] The server uses natural language processing (NLP) techniques to extract key keywords and phrases from retrieved news articles. The input is the retrieved news article data, and the output is the extracted keywords and categories. Specifically, the server uses an NLP library (e.g., spaCy) to analyze the content of the article and extract important keywords and categories.

[1607] Step 3:

[1608] Select an illustrated template

[1609] The server selects an appropriate illustration template based on the extracted category. The input is the extracted category, and the output is the selected illustration template. Specifically, the server searches for templates corresponding to the category from the template library and selects the most appropriate template.

[1610] Step 4:

[1611] Generating illustrations

[1612] The server visualizes the contents of the news article based on the selected template. The input is the content of the news article and the selected template, and the output is the generated illustration. Specifically, the server uses a graph drawing library such as Matplotlib to create the illustration according to the template.

[1613] Step 5:

[1614] Generate illustrated variations

[1615] The server generates variations of the generated illustration, such as different themes (e.g., dark mode), different languages, and different font sizes. The input is a basic illustration, and the output is multiple variations of the illustration. Specifically, the server creates multiple variations with different themes, languages, and font sizes according to the set parameters.

[1616] Step 6:

[1617] News article translation

[1618] The server uses the Google Translate API to translate the content of news articles into multiple languages. The input is the content of the news article written in the original language, and the output is the translated content of the news article. Specifically, the server sends a translation request to the Google Translate API and receives the translation result as a response.

[1619] Step 7:

[1620] Saving and distributing illustrated variations

[1621] The server saves the generated illustration variations in a database and prepares them for distribution to user devices. The input is the illustration variation, and the output is the reference information for the saved illustration and a status ready for distribution. Specifically, the server saves the generated illustration variations in a database and generates and stores reference information such as a URL.

[1622] Step 8:

[1623] Reading the news

[1624] The user terminal displays the latest news articles and corresponding illustrations. The input is the news article and illustration data delivered from the server, and the output is the news and illustrations displayed on the terminal's display. In concrete terms, the terminal displays the news article and illustrations on the screen based on the received data.

[1625] Step 9:

[1626] Customizing the illustrations

[1627] The user terminal allows the user to set the display format of the illustration. The input is the user's selection, and the output is a customized illustration display based on the selection. In concrete terms, the terminal receives the display setting change from the user and selects and displays the optimal illustration variation from the saved variations.

[1628] Step 10:

[1629] Viewing the illustration

[1630] The user terminal displays the illustration based on the selected display format. The input is customized illustration data, and the output is the illustration displayed on the terminal display. In specific operation, the terminal selects and displays an appropriate illustration variation according to the user's settings.

[1631] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1632] This invention combines an emotion engine with a system that acquires, analyzes, and visualizes news articles, enabling the system to recognize user emotions and dynamically customize the display of illustrations based on those emotions. To implement this system, it is necessary to execute the following steps: news article acquisition, analysis, illustration generation, variation creation, emotion recognition, customization, storage, and distribution. The specific flow of each process is explained below.

[1633] Server (backend) processing

[1634] 1. Retrieving news articles

[1635] The server periodically retrieves the latest news articles via RSS feeds or news APIs, along with basic information such as article title, body of text, publication date, and category.

[1636] 2. Preprocessing of news articles

[1637] The server analyzes the retrieved news articles using natural language processing (NLP) technology to extract key keywords and phrases, such as "Shibuya Ward, Tokyo" and "incident."

[1638] 3. Select a template by category

[1639] Based on the extracted keywords and phrases, the server automatically classifies the news articles into specific categories (e.g., incidents, economics, science, sports, etc.) and selects an appropriate illustration template accordingly.

[1640] 4. Generating illustrations

[1641] The server then creates an illustration of the news article based on the selected template, using map information and time-series data to generate a visually easy-to-understand illustration.

[1642] 5. Accessibility-aware variation generation

[1643] The server creates one or more variations of the illustration, such as dark mode, different languages ​​(e.g., English, Japanese), different text sizes, etc.

[1644] 6. Emotion Recognition by Emotion Engine

[1645] The server uses interaction data (camera, microphone, touch operations, etc.) from the user's device to allow the emotion engine to estimate the user's current emotional state (e.g., joy, sadness, surprise, anger, etc.).

[1646] 7. Emotion-based illustrated customization

[1647] The server selects the optimal illustration variation based on the user's emotional data obtained from the emotion engine and dynamically changes the illustration display format. For example, if the user feels sad, it will provide an illustration with a softer color scheme.

[1648] 8. Saving and distributing illustrations

[1649] The server stores the illustration variations selected based on emotion recognition in a database and delivers them to the user's terminal.

[1650] Terminal (client-side) processing

[1651] 1. Reading the news

[1652] When a user opens the news app, the device calls the server's API to retrieve the latest news articles and corresponding illustrations.

[1653] 2. Initial display of the diagram

[1654] The device temporarily caches the received news article and illustrations, and when the news article is opened, the corresponding illustration is initially displayed.

[1655] 3. Acquiring Emotion Data

[1656] The device collects user interaction data (camera footage, voice input, touch operations, etc.) and sends it to the emotion engine, which analyzes it to recognize the user's emotions.

[1657] 4. Customize display options

[1658] Based on the emotion recognition results, the device dynamically adjusts the display options of the illustrations. Users can also manually select display options (e.g., language, theme, font size) from the settings screen.

[1659] User interaction and experience

[1660] 1. Viewing news articles and illustrations

[1661] Users scroll through the list in the news app and select the article they are interested in. When they open the article, the corresponding illustration is displayed along with the details of the news article.

[1662] 2. Emotion-based change

[1663] While the user is browsing the article, the device recognizes the user's emotions in real time and dynamically changes the illustration display style based on the user's emotions. For example, if the user is feeling stressed, the illustration's color scheme will become softer.

[1664] 3. Change settings

[1665] Users can change the display options in the settings screen to suit their preferences, for example, switching from Japanese to English or increasing the font size.

[1666] 4. Use of Emotion History

[1667] The system records the user's emotional history and can provide more customized displays in the future based on the user's emotional patterns.

[1668] As a concrete example, consider the following scenario: When a user browses an "article about an incident" in a news app, the article contains content about an "incident that occurred in Shibuya Ward, Tokyo." The analysis engine extracts the keywords "Shibuya Ward, Tokyo" and "incident" and classifies the category as "incident." The server selects a template for the "incident category" and generates an illustration that combines map and timeline data. It then creates and saves multiple variations (e.g., for dark mode, English version, uppercase version).

[1669] When a user opens this news article on their device, a map and timeline illustration are displayed. At the same time, an emotion engine analyzes the user's facial expression and, if the user expresses sadness, the illustration's color scheme is softened.

[1670] In this way, the present invention not only displays the contents of news articles in a visually easy-to-understand manner, but also dynamically customizes the content according to the user's emotions, thereby enabling more personalized information to be provided.

[1671] The processing flow will be explained below.

[1672] Server (backend) processing

[1673] Step 1:

[1674] The server periodically queries an RSS feed or news API to retrieve new news articles, including basic information such as the article title, body of the article, publication date, and category.

[1675] Step 2:

[1676] The server analyzes the retrieved news articles using natural language processing (NLP) technology to extract key keywords and phrases, such as "Shibuya Ward, Tokyo" and "incident."

[1677] Step 3:

[1678] Based on the extracted keywords and phrases, the server automatically classifies news articles into specific categories (incidents, economics, science, sports, etc.).

[1679] Step 4:

[1680] The server selects an appropriate illustration template for each category, for example, a map or timeline template for an article in the incident category.

[1681] Step 5:

[1682] The server then creates an illustration of the news article based on the selected template, using map information and time-series data to generate a visually easy-to-understand illustration.

[1683] Step 6:

[1684] The server creates one or more illustrated variations for accessibility, including dark mode, different languages ​​(e.g., English, Japanese), different text sizes, etc.

[1685] Step 7:

[1686] The server uses interaction data (camera, microphone, touch operations, etc.) from the user's device to allow the emotion engine to estimate the user's current emotional state (joy, sadness, surprise, anger, etc.).

[1687] Step 8:

[1688] Based on the user's emotional data obtained from the emotion engine, the server selects the optimal illustration variation and dynamically changes the illustration display format. For example, if the user is feeling sad, it will provide an illustration with a softer color scheme.

[1689] Step 9:

[1690] The server stores the generated illustrations and their variations in a database and prepares them for distribution to user terminals.

[1691] Terminal (client-side) processing

[1692] Step 1:

[1693] When a user opens the news app, the device calls the server's API to retrieve the latest news articles and corresponding illustrations.

[1694] Step 2:

[1695] The device temporarily caches received news articles and illustrations to provide smooth display.

[1696] Step 3:

[1697] A user scrolls through the news list and taps on an article they are interested in. The article details screen opens, along with a corresponding illustration.

[1698] Step 4:

[1699] The device collects user interaction data (camera footage, voice input, touch operations, etc.) and sends it to the emotion engine, which analyzes it to recognize the user's emotions.

[1700] Step 5:

[1701] Based on the emotion recognition results, the device dynamically adjusts the display options of the illustrations. Users can also manually select display options (e.g., language, theme, font size) from the settings screen.

[1702] User interaction and experience

[1703] Step 1:

[1704] Users scroll through the news list in a news app, find an article that interests them, and tap on it.

[1705] Step 2:

[1706] When you open an article, you'll see details about the news story along with visually compelling illustrations, such as a map showing the location of the incident and a timeline of chronological events.

[1707] Step 3:

[1708] While the user is browsing the article, the device recognizes the user's emotions in real time and dynamically changes the illustration display style based on the user's emotions. For example, if the user is feeling stressed, the illustration's color scheme will become softer.

[1709] Step 4:

[1710] Users can change the display options in the settings screen to suit their preferences, for example, switching from Japanese to English or increasing the font size.

[1711] Step 5:

[1712] The system records the user's emotional history and can provide more customized displays in the future based on the user's emotional patterns.

[1713] As a concrete example, consider the following scenario: When a user browses an "article about an incident" in a news app, the article contains content about an "incident that occurred in Shibuya Ward, Tokyo." The analysis engine extracts the keywords "Shibuya Ward, Tokyo" and "incident" and classifies the category as "incident." The server selects a template for the "incident category" and generates an illustration that combines map and timeline data. It then creates and saves multiple variations (e.g., for dark mode, English version, uppercase version).

[1714] When a user opens this news article on their device, a map and timeline illustration are displayed. At the same time, an emotion engine analyzes the user's facial expression and, if the user expresses sadness, the illustration's color scheme is softened.

[1715] In this way, the present invention not only displays the contents of news articles in a visually easy-to-understand manner, but also dynamically customizes the content according to the user's emotions, thereby enabling more personalized information to be provided.

[1716] Example 2

[1717] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1718] Conventional news article delivery systems have had difficulty providing information tailored to individual users' emotional states. Furthermore, they were unable to dynamically customize the content of news articles to match the user's emotional state. As a result, the user experience was static, potentially leaving some users dissatisfied.

[1719] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring news articles, means for analyzing the acquired news articles using natural language processing technology and extracting major keywords and categories, means for selecting an appropriate illustration template based on the extracted keywords and categories, means for generating an illustration that visualizes the content of the news article based on the selected template, means for creating one or more illustration variations taking accessibility into consideration, means for collecting user interaction data and recognizing emotions, means for dynamically customizing the illustration variations based on the recognized emotions, and means for saving the generated illustration variations and delivering them to the user terminal. This dynamically customizes the visual display of the news article, enabling more personalized information to be provided based on the user's emotions.

[1720] "Means for obtaining news articles" refers to the function of periodically collecting news articles using a news API or RSS feed.

[1721] "Means of analyzing acquired news articles using natural language processing technology and extracting key keywords and categories" refers to a function that analyzes acquired news articles using natural language processing technology such as morphological analysis and automatically extracts key keywords and categories.

[1722] "Means for selecting an appropriate illustrated template based on extracted keywords and categories" is a function that automatically selects the most appropriate template from multiple pre-prepared illustrated templates based on analyzed keyword and category information.

[1723] "Means for generating illustrations that visualize the contents of a news article based on a selected template" refers to a function that uses a selected illustration template to generate illustrations that represent the contents of a news article in a visually easy-to-understand format, such as a map or timeline.

[1724] "Means for creating one or more variations of illustrations with accessibility in mind" refers to the ability to create variations of the generated illustrations in different formats (e.g., dark mode, different languages, different font sizes) that take into account user visibility and ease of use.

[1725] "Means for collecting user interaction data and recognizing emotions" refers to a function that collects data such as the user's camera footage, voice input, and touch operations, and analyzes it to estimate the user's current emotional state.

[1726] "Means for dynamically customizing illustration variations based on recognized emotions" is a function that selects the optimal illustration variation based on the user's emotional data and changes the color scheme and design of the illustration in real time.

[1727] The "means for saving the generated illustration variations and distributing them to the user's terminal" is a function for saving the generated multiple illustration variations in a database and distributing them to the user's terminal via the Internet.

[1728] MODE FOR CARRYING OUT THE INVENTION

[1729] This invention aims to provide a system for acquiring, analyzing, and illustrating news articles, and to provide a dynamically customized visual display according to the user's emotions. This system uses the following hardware and software:

[1730] Hardware

[1731] Server: Responsible for news article acquisition, analysis, illustration generation, emotion recognition, customization, storage and distribution.

[1732] User devices: Responsible for displaying news articles, collecting and transmitting emotion data, and dynamically changing the display. These devices include smartphones, tablets, and PCs.

[1733] software

[1734] Natural Language Processing (NLP) software: Use libraries such as SpaCy and NLTK to analyze news articles and extract key keywords and categories.

[1735] News API: Uses an API such as Google News API to make an HTTP request to retrieve the latest news articles.

[1736] Illustration generation library: Uses D3.js, Google Maps API, etc. to create visually easy-to-understand illustrations of the contents of news articles.

[1737] Emotion recognition engine: Analyzes user emotions using Microsoft Azure Emotion API, etc.

[1738] Example

[1739] 1. Acquiring and analyzing news articles

[1740] The server periodically retrieves news articles using the Google News API or RSS feeds. The retrieved articles are analyzed using natural language processing technology to extract key keywords and categories. For example, a request like https: / / newsapi.org / v2 / top-headlines?country=jp&apiKey=YOUR_API_KEY is sent.

[1741] 2. Template selection and diagram generation

[1742] Based on the extracted keywords and categories, the server selects an appropriate illustration template. Based on the selected template, it generates illustrations such as maps and timeline data using the Google Maps API and D3.js. For example, if the article is about an incident that occurred in Shibuya Ward, Tokyo, it selects a template in the incident category and illustrates the map and timeline data.

[1743] 3. Accessible Variation Generation

[1744] The server creates multiple variations of the generated illustrations, such as dark mode, different languages ​​(English, Japanese), different font sizes, etc. For example, it applies dark mode styles with CSS and adds different language labels to the illustration data.

[1745] 4. Emotional Data Collection and Recognition

[1746] While the user is browsing a news article, the device collects interaction data from the camera and microphone and sends it to the server. The server's emotion recognition engine analyzes the user's emotional state. For example, the user's emotional state can be recognized as "happiness," "sadness," "surprise," "anger," etc. based on facial expressions and voice analysis.

[1747] 5. Dynamic customization based on emotions

[1748] The server selects the most appropriate illustration variation based on the user's recognized emotion and changes the illustration's color scheme and design in real time. For example, if the user feels sad, it selects an illustration with a soft color scheme.

[1749] 6. Saving and distributing illustrations

[1750] The selected illustration variations are stored in a database and delivered to the user's device via the Internet, using a real-time database such as Firebase to transmit data in real time.

[1751] Specific examples and prompts for the generative AI model

[1752] As a specific example, consider a scenario in which a user views an article about an incident that occurred in Shibuya Ward, Tokyo.

[1753] When a user opens the news app, an article about an incident that occurred in Shibuya Ward, Tokyo, is retrieved from the server. The analysis engine extracts the keywords "Shibuya Ward, Tokyo" and "incident" from the article and classifies it as an "incident" category. The server selects a template that combines a map and a timeline to visualize the news content. The generated variations (e.g., dark mode, English version, uppercase version) are stored in a database.

[1754] When a user opens an article on their device, a map and timeline illustration are displayed. At the same time, the emotion engine analyzes the user's facial expressions and, if the user is expressing sadness, the illustration's color scheme is changed to a softer tone.

[1755] Example prompts to input to a generative AI model:

[1756] A user is viewing an article about an incident that occurred in Shibuya Ward, Tokyo. The article's category is "Incident," and an illustration using map and timeline data should be displayed. If the user's emotion is "Sadness," change the illustration's color scheme to a softer tone.

[1757] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1758] Step 1: Get the news article

[1759] Input: Periodically send requests for news articles using a news API or RSS feed.

[1760] Data processing: The server sends an HTTP request to the API endpoint and receives the returned data in JSON format.

[1761] Output: Data including news article title, body of text, publication date, category, etc.

[1762] Specific operation: Execute a request such as https: / / newsapi.org / v2 / top-headlines?country=jp&apiKey=YOUR_API_KEY.

[1763] Step 2: Preprocessing the news article

[1764] Input: News article data obtained in step 1.

[1765] Data processing: The server analyzes the news articles using natural language processing techniques (e.g., SpaCy, NLTK) to extract key keywords and phrases.

[1766] Output: Extracted keywords and phrases (e.g., "Shibuya Ward, Tokyo" and "incident").

[1767] Specific operation: Performs morphological analysis and extracts important words and phrases in the sentence.

[1768] Step 3: Select a template by category

[1769] Input: Keywords or phrases extracted in step 2.

[1770] Data processing: The server classifies news articles into specific categories based on extracted keywords and phrases, and selects appropriate illustration templates for each category.

[1771] Output: The selected diagram template (e.g., a template for incident categories).

[1772] Specific operation: Based on the analysis results, categorize them into categories such as "incidents," "economy," and "science," and select the corresponding template for each.

[1773] Step 4: Generate the diagram

[1774] Input: The illustrated template selected in step 3 and the content of the news article.

[1775] Data processing: The server generates illustrations that visualize the contents of news articles based on the selected template. Maps and timeline data are created using Google Maps API, D3.js, etc.

[1776] Output: Visualized and illustrated data.

[1777] Specific functions: Display the locations of incidents on a map and create a chronological timeline.

[1778] Step 5: Generate variations with accessibility in mind

[1779] Input: Illustrated data generated in step 4.

[1780] Data processing: The server creates variations of the generated illustrations in different formats (e.g., dark mode, different languages, changed font size).

[1781] Output: Multiple illustrated variations.

[1782] What it does: Changes CSS to apply dark mode styles and adds labels in different languages ​​to illustrated data.

[1783] Step 6: Emotion data collection and emotion recognition

[1784] Input: Interaction data (camera feeds, voice inputs, touch actions, etc.) while the user is viewing an article.

[1785] Data processing: The interaction data collected by the device is sent to the server, where the server's emotion recognition engine analyzes the emotional state.

[1786] Output: User emotion data (e.g., happy, sad, surprised, angry).

[1787] Specific operation: The user's facial expressions and voice are collected through a camera and microphone and analyzed using an emotion recognition model.

[1788] Step 7: Emotion-based illustration customization

[1789] Input: User emotion data recognized in step 6.

[1790] Data processing: The server selects the optimal illustration variation based on the emotion data and dynamically changes the color scheme and design of the illustration.

[1791] Output: Customized illustrated data.

[1792] Specific behavior: If the user expresses sadness, change the color scheme of the illustration to a softer tone.

[1793] Step 8: Save and distribute your diagrams

[1794] Input: Customized diagram data from step 7.

[1795] Data processing: The server stores the customized illustrations in a database and delivers them to the user's device.

[1796] Output: Illustrated data stored on the user's device.

[1797] What it does: Uses a real-time database such as Firebase to send customized diagrams over the internet.

[1798] Step 9: Initial display of news article and illustration

[1799] Input: News article and illustration data distributed in step 8.

[1800] Data processing: The device caches the news article and illustrations it receives, and initially displays the illustrations when the news article is opened.

[1801] Output: News article and illustration displayed on the user's terminal.

[1802] What it does: When you open the news app, it displays the latest news articles and their illustrations.

[1803] Step 10: Customize display options

[1804] Input: User settings changes (language, theme, text size, etc.).

[1805] Data processing: The device dynamically adjusts the display options of the illustration based on user settings changes.

[1806] Output: Customized graphical display.

[1807] Specific operation: The user changes the display options in the settings screen to display the diagram according to their preference.

[1808] (Application example 2)

[1809] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1810] Conventional advertising delivery systems have difficulty customizing ads based on user emotions, limiting their visibility and effectiveness. Furthermore, there is no technology for summarizing news-like information and incorporating it into ads, making it difficult for ads to be interesting to users. Therefore, there is a need for a system that can provide more personalized information by recognizing user emotions in real time and dynamically changing advertising design and content based on those emotions.

[1811] The specification processing by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for ...

Claims

1. a means for obtaining news articles; A means for analyzing the retrieved news articles and extracting key keywords and categories; a means for selecting an appropriate illustration template based on the extracted categories; means for illustrating the content of a news article based on a selected template; means for generating one or more graphical variations; A means for storing the generated illustration variations and delivering them to a user terminal; A system including:

2. The user terminal means for displaying the received news articles and associated graphics; means for enabling a user to select display options; means for changing the display format of the illustration based on a user selection; The system of claim 1 .

3. The illustrated variations are generated in multiple formats, including different languages, themes, and font sizes. The system of claim 1 .

4. and means for selecting and displaying an optimal illustration variation based on a user's setting. The system of claim 2.

Images