System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving text data; means for analyzing the text data to extract important keywords or phrases; means for generating or searching for relevant visual elements based on the important keywords or phrases; means for integrating the relevant visual elements to generate an image or video summarizing text content; and means for outputting the generated image or video.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, many people feel tired and stressed when reading long texts and sentences. Particularly in business and educational settings, vast amounts of documents and reports must be read, consuming a great deal of time and effort. Furthermore, people who have difficulty reading characters or who read texts written in a different language face a high hurdle to comprehension. There is a need for a method that can resolve these issues and efficiently input information. [Means for solving the problem]

[0005] This invention is a system including means for receiving text data, means for analyzing the text data to extract important keywords and phrases, means for generating or searching for related visual elements based on the important keywords and phrases, means for integrating the related visual elements to generate an image or video that summarizes the text content, and means for outputting the generated image or video. This system allows users to visually grasp information without reading the text, making it possible to input information efficiently and without stress.

[0006] "Text data" refers to long text information such as meeting materials, reports, novels, etc. that users upload from their terminals.

[0007] "Analysis" refers to the process of using natural language processing (NLP) techniques to segment text data and extract important keywords and phrases.

[0008] "Keywords" refer to words or phrases that have a particularly important meaning within text data and succinctly express its content.

[0009] A "phrase" refers to a meaningful group of multiple words in a sentence.

[0010] "Visual elements" refers to visually comprehensible forms of information such as images, videos, charts, etc.

[0011] A "generative AI model" refers to an artificial intelligence algorithm that automatically generates new images and videos.

[0012] "Tokenization" refers to the process of dividing text data into sentences and paragraphs, and breaking each of these down into smaller, analyzable units.

[0013] "Integration" refers to the process of combining the generated visual elements into a single entity to visually represent the text content.

[0014] A "summary" refers to information that has been condensed from original text data, extracted from the main points, and presented concisely.

[0015] "Output" refers to the process of sending the generated image or video to a user's terminal so that it can be displayed. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The present invention relates to a system for visually summarizing text, and supports users in stress-free comprehension of lengthy text data such as conference materials, reports, and novels.

[0038] System Overview

[0039] The system has the following main functions:

[0040] 1. Receiving text data

[0041] 2. Text Analysis

[0042] 3. Extracting important keywords and phrases

[0043] 4. Visual element generation or search

[0044] 5. Visual Integration

[0045] 6. Output of generated images or videos

[0046] Specific processing flow

[0047] Server-side processing

[0048] Receiving input data

[0049] A user uploads text data such as meeting materials, reports, novels, etc. via a terminal.

[0050] The terminal transmits the document data selected by the user to the server.

[0051] The server receives the uploaded document data.

[0052] Text preprocessing

[0053] The server analyzes the received text data using natural language processing (NLP) technology.

[0054] The server divides the text into sentences and paragraphs and tokenizes each one.

[0055] The server tags each token with a part of speech and extracts key keywords and phrases.

[0056] Analysis for image and video generation

[0057] The server searches a database for relevant visual elements based on the extracted keywords and phrases.

[0058] The server generates new images and videos using generative AI models (e.g., Generative Adversarial Networks, GAN).

[0059] The server applies algorithms to optimize the placement and order of the generated visual elements to match the content of the original text.

[0060] Content Integration

[0061] The server integrates the generated visual elements in the appropriate order to create a summary in a format that allows the user to easily understand the information visually while preserving the story and logical structure of the entire text.

[0062] The server reviews the generated summary and makes any necessary corrections or optimizations.

[0063] Output of conversion results

[0064] The server transmits the generated image or video summary to the user's terminal.

[0065] The terminal displays the received summary to the user, allowing the user to understand the information without stress.

[0066] Specific examples

[0067] Example 1: Novel summary

[0068] The user selects the novel text file from the device and presses the upload button.

[0069] The terminal transmits the text file of the selected novel to the server.

[0070] The server analyzes the received novel data and extracts key scenes and characters.

[0071] The server generates related images and videos based on the extracted information.

[0072] The server integrates images and videos to create a summary based on the story of the novel.

[0073] The server transmits the generated summary to the terminal.

[0074] The terminal displays the received visual summary to the user.

[0075] Users view a visual summary to get an overview of the story.

[0076] Example 2: Summary of business meeting materials

[0077] The user selects a PDF file of business meeting materials from their device and presses the upload button.

[0078] The terminal transmits the selected business conference materials to the server.

[0079] The server analyzes the content of the document and extracts key points and data charts.

[0080] The server generates relevant images and charts based on the extracted information.

[0081] The server integrates the generated images and charts to create a summary of the conference materials.

[0082] The server sends the summary to the terminal.

[0083] The terminal displays the received visual summary to the user.

[0084] Users view a visual summary to grasp the main points of the meeting.

[0085] This system visually summarizes long or complex texts, helping users understand information efficiently and without stress.

[0086] The processing flow will be explained below.

[0087] Step 1:

[0088] The user selects and uploads long text data such as conference materials, reports, novels, etc. from the terminal.

[0089] Step 2:

[0090] The terminal transmits the selected text data to the server.

[0091] Step 3:

[0092] The server receives the uploaded document data.

[0093] Step 4:

[0094] The server analyzes the received text data using natural language processing (NLP) technology.

[0095] Step 5:

[0096] The server breaks the text into sentences and paragraphs and tokenizes each, breaking the text into smaller, analyzable units.

[0097] Step 6:

[0098] The server tags each token with a part of speech tag and extracts key keywords and phrases.

[0099] Step 7:

[0100] The server searches a database for related visual elements (images and videos) based on the extracted keywords and phrases.

[0101] Step 8:

[0102] The server may also generate new images and videos using generative AI models (e.g., Generative Adversarial Networks, GANs).

[0103] Step 9:

[0104] The server applies algorithms to optimize the placement and order of the generated visual elements to match the content of the original text.

[0105] Step 10:

[0106] The server integrates the generated visual elements in the appropriate order to create a summary while preserving the story and logical structure of the entire text.

[0107] Step 11:

[0108] The server reviews the generated summary and makes any necessary corrections or optimizations.

[0109] Step 12:

[0110] The server transmits the generated image or video summary to the user's terminal.

[0111] Step 13:

[0112] The terminal receives the visual summary result sent from the server.

[0113] Step 14:

[0114] The terminal displays the received images and videos to the user, allowing the user to understand the information without stress.

[0115] Step 15:

[0116] Users can view the visual summary and input the necessary information efficiently.

[0117] Example 1

[0118] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0119] There is a problem that it is difficult for users to understand long text data such as meeting materials, reports, and novels efficiently and without stress. In particular, extracting important parts and visually summarizing them requires a great deal of time and effort using conventional methods. Furthermore, if the generation and integration of visual elements is not performed properly, it becomes even more difficult to understand the information.

[0120] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0121] In this invention, the server includes means for receiving text data, means for analyzing the text data to extract important keywords and phrases, means for generating or searching for related visual elements based on the important keywords and phrases, means for integrating the related visual elements to generate an image or video that summarizes the text content, and means for outputting the generated image or video, thereby enabling a user to efficiently understand long text data in a visually summarized format.

[0122] The definition is given below.

[0123] "Text data" refers to data in which information such as meeting materials, reports, and novels is written in text format.

[0124] "Means for receiving" refers to the function by which the server receives text data sent by the user.

[0125] "Means of analysis" refers to the function by which the server uses natural language processing technology to divide and tokenize text data and understand its content.

[0126] "Means for extracting important keywords and phrases" refers to the function of selecting key words and short sentences from analyzed text data.

[0127] "Means for generating or searching for relevant visual elements" refers to the ability to create or search a database for images or videos based on extracted keywords or phrases.

[0128] "Means for integrating visual elements" refers to the function of combining the generated images and videos in an appropriate order and arrangement to visually express the content of the entire text.

[0129] "Means for generating a summarized image or video" refers to a function that produces a summary of a text from integrated visual elements.

[0130] "Means for outputting" refers to a function for transmitting the generated summary result to the user's terminal and displaying it.

[0131] "Natural language processing technology" refers to technology that enables computers to understand, interpret, and manipulate human language.

[0132] A "generative AI model" refers to an algorithm or learning model that uses artificial intelligence to generate new data (e.g., images or videos).

[0133] "Algorithms that optimize placement and order" refer to methods for determining the optimal placement and display order of visual elements to match the content of the original text.

[0134] This invention relates to a visual summarization system that allows users to understand long text data efficiently and without stress. This system analyzes the text data, extracts important keywords and phrases, generates or searches for related visual elements based on them, and integrates these visual elements to generate an image or video that summarizes the text content and provides it to the user.

[0135] System Overview

[0136] The system has the following main functions:

[0137] 1. Receiving text data

[0138] 2. Text Analysis

[0139] 3. Extracting important keywords and phrases

[0140] 4. Visual element generation or search

[0141] 5. Visual Integration

[0142] 6. Output of generated images or videos

[0143] Specifically, this is realized using the following hardware and software.

[0144] Hardware and software used

[0145] server

[0146] Common NLP libraries "SpaCy" or "NLTK" for using natural language processing (NLP) techniques

[0147] "Elasticsearch" as a search engine

[0148] "TensorFlow" or "PyTorch" frameworks for using generative AI models

[0149] OpenCV is an image processing library for optimizing the placement and order of visual elements.

[0150] Terminal

[0151] A computing device having an interface for a user to upload text data and display generated summaries.

[0152] Program processing

[0153] Specific data processing and data calculations are performed between the server, terminal, and user as follows:

[0154] 1. Receiving text data

[0155] Users upload the necessary text data (conference materials, reports, novels, etc.) from their terminals.

[0156] The terminal sends the text data selected by the user to the server using an HTTP POST request.

[0157] The server stores the uploaded text data in a specified directory.

[0158] 2. Text Analysis

[0159] The server analyzes the received text data using natural language processing (NLP) technology. Specifically, it divides the text into sentences and paragraphs, tokenizes each, and tags them by part of speech.

[0160] 3. Extracting important keywords and phrases

[0161] The server extracts key keywords and phrases based on part-of-speech tagged token information, and stores this information in a list format.

[0162] 4. Visual element generation or search

[0163] The server searches the database for relevant visual elements based on the extracted keywords or phrases. If no results are found, it generates new images or videos using a generative AI model (e.g., Generative Adversarial Networks, GAN). The generated visual elements are saved in a temporary directory.

[0164] 5. Visual Integration

[0165] The server then integrates the generated visual elements in the correct order to create a summary that preserves the overall story and logical structure of the text, allowing users to easily understand the information visually. The server then arranges the images according to the storyline, resizes them to the optimal size, and sets the appropriate placement coordinates. The results are then exported as a PDF or video.

[0166] 6. Output of generated images or videos

[0167] The server transmits the generated image or video summary to the user's terminal.

[0168] The terminal receives the response and displays it in the appropriate viewer.

[0169] Specific examples

[0170] Below, we will show the process of summarizing a novel and business conference materials as concrete examples of this system.

[0171] Example 1: Novel summary

[0172] 1. The user selects the novel text file from their device and presses the upload button.

[0173] 2. The terminal sends the text file of the selected novel to the server.

[0174] 3. The server analyzes the received novel data and extracts key scenes and characters.

[0175] 4. The server generates relevant images and videos based on the extracted information.

[0176] 5. The server integrates images and videos to create a summary based on the story of the novel.

[0177] 6. The server sends the generated summary to the terminal.

[0178] 7. The terminal displays the received visual summary to the user.

[0179] Here are some example prompts to input to the generative AI model:

[0180] "Generate a visual summary of the following novel: [Insert novel text here]. Focus on the main scenes and characters."

[0181] Example 2: Summary of business meeting materials

[0182] 1. The user selects the PDF file of the business meeting materials from the device and presses the upload button.

[0183] 2. The terminal transmits the selected business conference materials to the server.

[0184] 3. The server analyzes the content of the document and extracts important points and data charts.

[0185] 4. The server generates relevant images and charts based on the extracted information.

[0186] 5. The server integrates the generated images and charts to create a summary of the meeting materials.

[0187] 6. The server sends the summary to the terminal.

[0188] 7. The terminal displays the received visual summary to the user.

[0189] Here are some example prompts to input to the generative AI model:

[0190] "Create a visual summary of the following business meeting document: [Insert meeting document text here]. Highlight key points and data charts."

[0191] As described above, this system visually summarizes long sentences and complex content, helping users understand information efficiently and without stress.

[0192] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0193] Step 1:

[0194] The user uploads text data.

[0195] Input: Written data such as meeting materials, reports, novels, etc.

[0196] Operation: The user clicks the upload button on the device to upload the selected text data to the device.

[0197] Output: The text data is saved to the terminal.

[0198] Step 2:

[0199] The terminal transmits the text data to the server.

[0200] Input: User-uploaded text data

[0201] Operation: The terminal sends the received text data to the server via an HTTP POST request.

[0202] Output: The server receives the text data.

[0203] Step 3:

[0204] The server analyzes the text data.

[0205] Input: Text data sent from the terminal

[0206] How it works: The server uses a natural language processing (NLP) library to split the text data into sentences and paragraphs, and tokenize each. Specifically, it uses the NLP library "SpaCy" or "NLTK" and the nlp(text) function.

[0207] Output: Segmented and tokenized sentence data.

[0208] Step 4:

[0209] The server extracts important keywords and phrases.

[0210] Input: Segmented and tokenized text data

[0211] How it works: The server tags the segmented and tokenized data and extracts important keywords and phrases. Keywords are extracted based on specific parts of speech (nouns, verbs, adjectives).

[0212] Output: A list of key keywords and phrases.

[0213] Step 5:

[0214] The server generates or retrieves the visual elements.

[0215] Input: Extracted keywords or phrases

[0216] How it works: The server searches a database for relevant visual elements based on keywords or phrases. It uses Elasticsearch to execute the query, and if no results are found, it uses a generative AI model (e.g., Generative Adversarial Networks, GAN) to generate new images or videos. It leverages TensorFlow or PyTorch.

[0217] Output: Generated or retrieved visual elements (images or videos).

[0218] Step 6:

[0219] The server aggregates the visual elements.

[0220] Input: A generated or searched visual element

[0221] How it works: The server orders the images according to the storyline, adjusting their placement and size, and uses OpenCV to integrate the visual elements in an optimized order.

[0222] Output: Optimized and consolidated image or video summary.

[0223] Step 7:

[0224] The server sends the summary to the terminal.

[0225] Input: Optimized and integrated summary (image or video)

[0226] Action: The server generates a summary and sends it as an HTTP response.

[0227] Output: The terminal receives the summary.

[0228] Step 8:

[0229] The terminal displays the summary to the user.

[0230] Input: Abstract (image or video) sent from the server

[0231] Operation: The device displays the received summary in an appropriate viewer, ensuring that the information is visually easy to understand based on the UE / UX design.

[0232] Output: A visual summary that is displayed to the user.

[0233] In this way, the invention visually summarizes complex textual data, allowing users to efficiently grasp the information.

[0234] (Application example 1)

[0235] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0236] The challenge is to efficiently summarize large amounts of text content and present it in a visually easy-to-understand format, allowing users to understand the information without stress. There is a particular need to provide a means to easily grasp the content, even when it is large in volume or highly important.

[0237] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0238] In this invention, the server includes means for receiving text data, means for analyzing the text data to extract important keywords and phrases, means for generating or searching for related visual elements based on the important keywords and phrases, means for interactively displaying the generated images or videos, and means for sharing the generated images or videos with other users or on a social network, thereby enabling efficient summarization of vast amounts of text content and presenting it in a visually easy-to-understand format.

[0239] "Text data" refers to information or content provided in text format, including meeting materials, reports, novels, etc.

[0240] "Analysis" refers to the means and methods for understanding the content of text data and extracting important elements and patterns.

[0241] "Keywords" refer to words or phrases that have particularly important meanings in a text, and are essential elements for summarizing the entire content.

[0242] A "phrase" is a group of words consisting of a series of multiple words, and is an expression that conveys a specific meaning or information.

[0243] A "visual element" is a visually recognizable object, such as an image or video, that is used to visually summarize text.

[0244] "Generation" refers to the process of creating new images or videos, which is done using artificial intelligence models.

[0245] "Searching" is the process of finding relevant data from existing databases and sources.

[0246] "Interactive display" refers to a display method that allows users to manipulate or take action on the displayed information.

[0247] "Sharing" refers to distributing the generated visuals and information to other users or through social networks.

[0248] "Tokenization" is the process of breaking down text data into individual words and phrases, and is a fundamental step in natural language processing.

[0249] A "generative AI model" is an artificial intelligence model that generates new data using technologies such as deep learning, and includes generative adversarial networks (GANs).

[0250] This invention describes a system for efficiently summarizing large amounts of textual content and presenting it in a visually accessible format. This system is particularly applicable to content distribution services and is designed to enable users to quickly understand long-form content such as news articles or novels.

[0251] System Overview

[0252] The system consists of the following major hardware and software components:

[0253] Hardware: Smartphone (iOS or Android), server

[0254] Software: Python, spacy, Summarizer, DeepAI, Pillow

[0255] Program processing flow

[0256] 1. Receiving text data:

[0257] The user enters the URL or text data selected within the content distribution service into the device, which then sends it to the server.

[0258] 2. Text analysis:

[0259] The server analyzes the received text data using natural language processing (NLP) techniques, specifically tokenizing the text using the spacy library and extracting important keywords and phrases.

[0260] 3. Visual element generation or search:

[0261] Based on the extracted keywords and phrases, the server uses generative AI models (e.g., Generative Adversarial Networks, or GANs) to generate new images and videos, and, if necessary, uses DeepAI's API to search for related images from an existing database.

[0262] 4. Interactive display:

[0263] The server generates visual elements and sends them to the user's device, which displays them interactively. Users can view a visual summary and tap on areas of interest to see more detailed information.

[0264] 5. Sharing features:

[0265] The generated visual summary can be shared by the user with other users or on social networks, and the device provides an interface for sharing this information with other users.

[0266] Specific examples

[0267] Example 1: News article summary

[0268] A user enters a news article URL into the app.

[0269] The app sends the URL to a server, which analyzes the article content and generates a summary.

[0270] The generated summary and associated images are sent to the terminal, where the user can view them interactively.

[0271] If desired, the summary can also be shared on social media.

[0272] Example 2: Technical document summary

[0273] The user uploads the text data of the technical document to the app.

[0274] The app sends the text data to a server, which analyzes the document and generates a summary.

[0275] A visual summary containing key points is generated that users can easily check on their smartphone.

[0276] Prompt Sentence Examples

[0277] "Create a summary and visual summary of the news article https: / / example.com / article."

[0278] In this way, the present invention provides a concrete example for efficiently summarizing a huge amount of text content and enabling users to understand it without stress.

[0279] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0280] Step 1:

[0281] The user selects text content of interest within the content distribution service (for example, a news article URL or long text data) and enters it into the device. The device then sends it to the server. The input in this step is the URL or text data, and the output is data sent to the server.

[0282] Step 2:

[0283] The server analyzes the received text data. Specifically, it uses the Python natural language processing library spacy to tokenize the data and extract important keywords and phrases. The input in this step is the received text data, and the output is the extracted keywords and phrases.

[0284] Step 3:

[0285] The server generates new images and videos using generative AI models such as Generative Adversarial Networks (GANs) based on the extracted keywords and phrases. It also searches for related images from a database using DeepAI's API as needed. The input in this step is the extracted keywords and phrases, and the output is the generated or searched visual elements (images and videos).

[0286] Step 4:

[0287] The server integrates the generated visual elements to create an interactive visual summary, and then constructs an interface that allows users to easily manipulate and view the visual summary. The input in this step is the generated visual elements, and the output is an interactive visual summary.

[0288] Step 5:

[0289] The server sends the interactive visual summary to the user's device for the user to review. If necessary, the user can tap on the part of the visual summary that they want to know more about to view the details. The input in this step is the interactive visual summary, and the output is the visual summary displayed on the user's device.

[0290] Step 6:

[0291] The device provides an interface for sharing the generated visual summary with other users or on social networks. Users can easily share their visual summaries with others. The inputs in this step are the visual summary and the sharer information, and the output is the shared visual summary.

[0292] These are the specific processing steps of the system program that realizes this application example, which enables users to efficiently understand large amounts of text content and share visual summaries.

[0293] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0294] The present invention relates to a system that recognizes a user's emotions and visually summarizes text, thereby helping users understand long conference materials, reports, novels, etc. without stress.

[0295] System Overview

[0296] The system has the following main functions:

[0297] 1. Receiving text data

[0298] 2. Text Analysis

[0299] 3. Extracting important keywords and phrases

[0300] 4. Visual element generation or search

[0301] 5. Emotion Recognition by Emotion Engine

[0302] 6. Visual Integration

[0303] 7. Output of generated images or videos

[0304] Specific processing flow

[0305] Server-side processing

[0306] Receiving input data

[0307] A user uploads text data such as meeting materials, reports, novels, etc. via a terminal.

[0308] The terminal transmits the document data selected by the user to the server.

[0309] The server receives the uploaded document data.

[0310] Text preprocessing

[0311] The server analyzes the received text data using natural language processing (NLP) technology.

[0312] The server divides the text into sentences and paragraphs and tokenizes each one.

[0313] The server tags each token with a part of speech and extracts key keywords and phrases.

[0314] Analysis for image and video generation

[0315] The server searches a database for relevant visual elements based on the extracted keywords and phrases.

[0316] The server may also generate new images and videos using generative AI models (e.g., Generative Adversarial Networks, GANs).

[0317] The server applies algorithms to optimize the placement and order of the generated visual elements to match the content of the original text.

[0318] emotion recognition

[0319] The device analyzes the user's facial expressions and reactions as they view visual elements, and uses an emotion engine to recognize the user's emotions.

[0320] The server adjusts the content and style of the generated visual elements based on the recognized user emotion.

[0321] The server changes the format of the summary results to be presented depending on the user's emotions.

[0322] Content Integration

[0323] The server integrates the generated visual elements in the appropriate order to create a summary while preserving the story and logical structure of the entire text.

[0324] The server reviews the generated summary and makes any necessary corrections or optimizations.

[0325] Output of conversion results

[0326] The server transmits the generated image or video summary to the user's terminal.

[0327] The terminal displays the received summary to the user, allowing the user to understand the information without stress.

[0328] Specific examples

[0329] Example 1: Novel summary

[0330] The user selects the novel text file from the device and presses the upload button.

[0331] The terminal transmits the text file of the selected novel to the server.

[0332] The server analyzes the received novel data and extracts key scenes and characters.

[0333] The server generates related images and videos based on the extracted information.

[0334] The server integrates images and videos as a summary according to the story of the novel.

[0335] The server adjusts the content and style based on the perceived user sentiment.

[0336] The server transmits the generated summary to the terminal.

[0337] The terminal displays the received visual summary to the user.

[0338] Users view a visual summary to get an overview of the story.

[0339] Example 2: Summary of business meeting materials

[0340] The user selects a PDF file of business meeting materials from their device and presses the upload button.

[0341] The terminal transmits the selected business conference materials to the server.

[0342] The server analyzes the content of the document and extracts key points and data charts.

[0343] The server generates relevant images and charts based on the extracted information.

[0344] The server integrates the generated images and charts to create a summary of the conference materials.

[0345] The server adjusts the content and style based on the perceived user sentiment.

[0346] The server transmits the generated summary to the terminal.

[0347] The terminal displays the received visual summary to the user.

[0348] Users view a visual summary to grasp the main points of the meeting.

[0349] This system not only visually summarizes long or complex texts, but also adjusts the way information is presented to take the user's emotions into account, thereby deepening their understanding and reducing stress.

[0350] The processing flow will be explained below.

[0351] Step 1:

[0352] The user selects and uploads long text data such as conference materials, reports, novels, etc. from the terminal.

[0353] Step 2:

[0354] The terminal transmits the selected text data to the server.

[0355] Step 3:

[0356] The server receives the uploaded document data.

[0357] Step 4:

[0358] The server analyzes the received text data using natural language processing (NLP) technology.

[0359] Step 5:

[0360] The server breaks the text into sentences and paragraphs and tokenizes each, breaking the text into smaller, analyzable units.

[0361] Step 6:

[0362] The server tags each token with a part of speech tag and extracts key keywords and phrases.

[0363] Step 7:

[0364] The server searches a database for related visual elements (images and videos) based on the extracted keywords and phrases.

[0365] Step 8:

[0366] The server may also generate new images and videos using generative AI models (e.g., Generative Adversarial Networks, GANs).

[0367] Step 9:

[0368] The server applies algorithms to optimize the placement and order of the generated visual elements to match the content of the original text.

[0369] Step 10:

[0370] The device uses cameras and sensors to capture the user's facial expressions and reactions as they view visual elements.

[0371] Step 11:

[0372] The terminal sends the captured data to an emotion engine to analyze the user's emotions.

[0373] Step 12:

[0374] The server receives the analyzed user emotion data and adjusts the content and style of the generated visual elements.

[0375] Step 13:

[0376] The server changes the format of the summary results to be presented depending on the user's emotions.

[0377] Step 14:

[0378] The server integrates the generated visual elements in the appropriate order to create a summary while preserving the story and logical structure of the entire text.

[0379] Step 15:

[0380] The server reviews the generated summary and makes any necessary corrections or optimizations.

[0381] Step 16:

[0382] The server transmits the generated image or video summary to the user's terminal.

[0383] Step 17:

[0384] The terminal receives the visual summary result sent from the server.

[0385] Step 18:

[0386] The terminal displays the received images and videos to the user, allowing the user to understand the information without stress.

[0387] Step 19:

[0388] Users can view the visual summary and input the necessary information efficiently.

[0389] Example 2

[0390] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0391] In today's world, there is a huge amount of written information flying around, and people need to be able to quickly and efficiently retrieve the information they need. However, when users read long documents such as meeting materials, reports, or novels, it takes time and effort to understand and summarize all of the content. Furthermore, if information is presented without taking into account the emotions that individual users experience when comprehending the information, it can lead to a shallow understanding or stress.

[0392] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving text data, means for analyzing the text data and extracting important keywords and phrases, means for generating or searching for related visual elements based on the important keywords and phrases, means for analyzing a user's emotions and adjusting the content and style of the related visual elements, means for integrating the related visual elements and generating an image or video that summarizes the text content, and means for outputting the generated image or video. This enables a user to efficiently understand long sentences or complex information in a visually summarized form, and further, information presented in accordance with the user's emotions deepens understanding and reduces stress.

[0393] The "means for receiving text data" is a function by which the server receives text data uploaded by the user through the terminal.

[0394] The "means for extracting important keywords and phrases" is a function that analyzes received text data and finds the most important keywords and phrases within it based on the internal structure and context.

[0395] "Means for generating or searching for relevant visual elements" refers to a function that searches for appropriate visual elements from a database based on extracted keywords or phrases, or generates new images or videos using a generative AI model.

[0396] "Means for analyzing the user's emotions and adjusting the content and style of the related visual elements" refers to a function that analyzes the user's facial expressions and reactions through the terminal and changes the content and style of the visual elements based on the results.

[0397] "Means for integrating the related visual elements and generating an image or video that summarizes the content of the text" refers to a function that combines the generated visual elements in an appropriate order to create a single image or video that clearly summarizes the content of the original text.

[0398] The "means for outputting the generated image or video" is a function for transmitting the image or video that is the integrated summary result to the user's terminal and displaying it.

[0399] This invention is a visual summarization support system that helps users understand long text data. This system combines natural language processing technology, generative AI models, and emotion recognition technology to analyze text data and generate visual summaries.

[0400] First, a user uploads text data such as meeting materials, reports, or novels using a terminal. The uploaded text data is then sent to a server via the terminal. The server temporarily stores the received text data and uses it for subsequent analysis.

[0401] The server analyzes the text data using natural language processing (NLP) techniques. Examples of NLP tools used include SpaCy and NLTK. The server divides the text data into sentences and paragraphs and tokenizes each. This process may involve using a tokenization algorithm such as the BERT model. The server then tags each token with a part of speech and extracts important keywords and phrases. This process uses techniques such as TF-IDF and Word2Vec.

[0402] The server then searches a database for relevant visual elements based on the extracted keywords or phrases, and generates new images or videos using generative AI models (e.g., Generative Adversarial Networks, GANs) if necessary. The resulting visual elements are then optimized in their placement and order to match the content of the original text.

[0403] Furthermore, the device analyzes the user's facial expressions and reactions when viewing visual elements through the camera, and recognizes the user's emotions using an emotion engine (e.g., OpenCV or Facial Emotion Recognition API).The server adjusts the content and style of the generated visual elements based on the recognized user emotions, and presents optimal information according to the user's emotions.

[0404] The server then integrates the generated visual elements in the correct order to create a summary that preserves the overall story and logical structure of the text. The server then reviews the integrated summary and makes any necessary corrections or optimizations. The generated image or video summary is then sent to the user's device, where it is displayed, allowing the user to easily understand the information.

[0405] Below, examples of prompt sentences are shown as concrete examples of implementing this system.

[0406] Example prompt sentence:

[0407] "The system receives specified text data, analyzes it using natural language processing technology, and extracts important keywords and phrases. It then generates or searches for related visual elements based on the extracted keywords and phrases. It analyzes the user's sentiment and adjusts the content and style of the generated visual elements. Finally, it integrates the related visual elements to generate an image or video that summarizes the text content and outputs it to the user's device."

[0408] The present invention enables a user to efficiently understand long sentences or complex text data through visual summaries, and furthermore, by presenting information according to the user's emotions, the user's understanding is deepened and stress is reduced.

[0409] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0410] Step 1:

[0411] Receiving input data

[0412] A user uploads text data such as meeting materials, reports, novels, etc. via a terminal.

[0413] Input: A text file selected by the user (PDF, TXT, DOCX, etc.)

[0414] Output: A text file temporarily saved on the device

[0415] The terminal sends the selected text data to the server, typically using an HTTP POST request.

[0416] Input: A text file uploaded by the user

[0417] Output: Text file sent to the server

[0418] The server receives the uploaded text data and temporarily stores it.

[0419] Input: Text file sent from the terminal

[0420] Output: Temporarily saved text file

[0421] Step 2:

[0422] Text preprocessing

[0423] The server analyzes the received text data using natural language processing (NLP) techniques, such as SpaCy and NLTK.

[0424] Input: Saved text data

[0425] Output: Tokenized data split into sentences and paragraphs

[0426] The server splits the text into sentences and paragraphs and tokenizes each of them, using a model such as BERT.

[0427] Input: Data for the entire sentence

[0428] Output: An array of tokens

[0429] The server tags each token with a part of speech and extracts key keywords and phrases, using TF-IDF and Word2Vec.

[0430] Input: Tokenized data

[0431] Output: Part-of-speech tagged tokens and key keywords and phrases

[0432] Step 3:

[0433] Extracting important keywords and phrases

[0434] The server compiles the extracted keywords and phrases and ranks them by importance.

[0435] Input: Part-of-speech tagged tokens and extracted keywords and phrases

[0436] Output: A list of keywords or phrases ranked by importance

[0437] Step 4:

[0438] Analysis for image and video generation

[0439] The server searches a database for relevant visual elements based on the extracted keywords or phrases, using an image search API (e.g., a general image search API).

[0440] Input: Ranking keyword or phrase

[0441] Output: A list of related visual elements

[0442] The server may also generate new images and videos using generative AI models (e.g., Generative Adversarial Networks, GANs).

[0443] Input: Keyword or phrase

[0444] Output: The new image or video generated

[0445] The server applies algorithms to optimize the placement and order of the generated visual elements to match the content of the original text.

[0446] Input: A list of visual elements

[0447] Output: properly positioned and ordered visual elements

[0448] Step 5:

[0449] emotion recognition

[0450] The device analyzes the user's facial expressions and reactions when viewing visual elements through the camera and recognizes the user's emotions using an emotion engine (e.g., a general emotion recognition API).

[0451] Input: Camera video data

[0452] Output: Recognized user emotion data

[0453] The server adjusts the content and style of the generated visual elements based on the recognized user emotion.

[0454] Input: User emotion data and visual elements

[0455] Output: Adjusted visual elements

[0456] The server changes the format of the summary results to be presented depending on the user's emotions.

[0457] Input: User emotion data and summary results

[0458] Output: Summarization results according to emotions

[0459] Step 6:

[0460] Content Integration

[0461] The server integrates the generated visual elements in the appropriate order to create a summary while preserving the story and logical structure of the entire text.

[0462] Input: Optimized visual elements

[0463] Output: Integrated summary image or video

[0464] The server reviews the generated summary and makes any necessary corrections or optimizations.

[0465] Input: Synthesized summary image or video

[0466] Output: revised and optimized summary

[0467] Step 7:

[0468] Output of conversion results

[0469] The server sends the generated image or video summary to the user's device, typically using an HTTP POST request.

[0470] Input: Corrected and optimized summary

[0471] Output: Summary sent to the user's terminal

[0472] The terminal displays the received summary to the user, allowing the user to understand the information without stress.

[0473] Input: Submitted Abstract

[0474] Output: The summary displayed to the user

[0475] This allows users to efficiently understand long or complex textual data through visual summaries.In addition, the way information is presented is adapted according to the user's emotions, deepening understanding and enabling information acquisition with less stress.

[0476] (Application example 2)

[0477] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0478] In today's world, people are bombarded with a vast amount of information, and in the field of advertising in particular, highly expressive content and content that takes into account the recipient's emotions are required to attract attention. However, conventional ad generation methods have difficulty dynamically changing content in response to user emotions, resulting in the inability to provide effective ads. To solve this problem, technology is needed that can recognize user emotions in real time and generate ads by combining appropriate visual elements and text content.

[0479] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0480] In this invention, the server includes means for receiving text data, means for analyzing the text data and extracting important keywords and phrases, means for generating or searching for related visual elements based on the important keywords and phrases, means for recognizing a user's emotions, means for adjusting the content and style of the generated visual elements based on the user's emotions, and means for outputting the generated images or videos, thereby enabling effective advertisement generation according to the user's emotions in real time.

[0481] "Text data" refers to information in text format provided by users, including content such as advertising copy and product descriptions.

[0482] "Analysis" is the process of breaking down text data into sentences and paragraphs and extracting important keywords and phrases.

[0483] "Keywords and phrases" refer to words and expressions that are particularly important in text data and express the gist of the information.

[0484] "Visual elements" refer to elements such as images and videos used to visually convey information to users.

[0485] "Integration" refers to the process of combining visual elements generated or retrieved based on extracted keywords or phrases and displaying them in a cohesive form.

[0486] A "generative AI model" refers to a model that uses artificial intelligence technology to generate new images and videos.

[0487] "Emotion recognition" is the process of detecting and classifying emotions from a user's facial expressions and reactions.

[0488] "Adjustment" is the procedure of dynamically changing the content and style of visual elements based on the user's emotions.

[0489] "Output" refers to the processing of the generated visual elements and integrated advertising content for display to the user.

[0490] The present invention is applied to a system that recognizes a user's emotions and visually summarizes long texts. The system for implementing the present invention is realized by linking a server and a user's terminal.

[0491] System configuration

[0492] The system includes the following main components:

[0493] 1. Server:

[0494] Includes software that implements natural language processing (NLP) techniques to analyze text and extract important keywords and phrases.

[0495] Includes software that implements generative AI models (e.g., Generative Adversarial Networks, GAN) to generate or search for relevant visual elements.

[0496] It includes an emotion engine that recognizes user emotions and adjusts the content and style of visual elements.

[0497] It includes software for generating and outputting a summary result format according to the user's emotions.

[0498] 2. On the user's device:

[0499] Includes software that provides an interface for users to input textual data. Example: smartphone applications.

[0500] It includes a camera for detecting the user's facial expressions and reactions in real time, and software for analyzing facial expressions, e.g., OpenCV library.

[0501] Includes software for displaying the visual summary received from the server to the user. Example: smartphone display.

[0502] Processing flow

[0503] 1. Receiving input data:

[0504] A user inputs text data such as advertising copy and product descriptions using a smartphone application.

[0505] 2. Text Analysis:

[0506] The server analyzes the received text data and extracts important keywords and phrases using natural language processing techniques (e.g., the TextBlob library).

[0507] 3. Visual element generation:

[0508] Based on the extracted keywords and phrases, relevant visual elements (images and videos) are generated or searched for, using a generative AI model (e.g., GAN).

[0509] 4. Emotion recognition:

[0510] While the user is viewing the advertisement on the device, the camera detects facial expressions in real time and analyzes the user's emotions using an emotion engine, which uses the OpenCV library.

[0511] 5. Content Adjustment:

[0512] Dynamically adjust the content and style of visual elements based on recognized user emotions.

[0513] 6. Output:

[0514] The final visual summary is sent to the user's terminal and displayed.

[0515] Specific examples

[0516] For example, suppose a user enters the following ad copy for a newly released cosmetic product: "New cosmetics! The secret to making your skin glow is here." The server analyzes this text and extracts key keywords and phrases (e.g., "new release," "cosmetics," "the secret to glowing"). It then uses a generative AI model to generate visual elements that match these keywords. When a user views an ad, the device's camera detects the user's facial expression of interest and selects visual elements and short video clips that best match that emotion to display the ad. This series of processes enables effective ad generation.

[0517] Prompt Sentence Examples

[0518] "If the user has an interested expression, the system will generate the most resonant visuals and short video clips to pair with this ad copy to create an ad."

[0519] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0520] Step 1:

[0521] A user uses a smartphone application to input text data such as advertising copy or product descriptions. As input, the user enters text into a text box and presses the "Send" button. As output, the device sends this text data to a server.

[0522] Step 2:

[0523] The server analyzes the received text data. As input, the server receives text data. To process the data, it uses natural language processing techniques (e.g., the TextBlob library) to split the data into sentences and paragraphs, tokenize them, and extract important keywords and phrases. As output, it generates a list of these keywords and phrases.

[0524] Step 3:

[0525] The server generates or searches for relevant visual elements based on the extracted keywords or phrases. As input, it receives a list of keywords and uses a generative AI model (e.g., GAN) to generate or search for appropriate images or videos from a database. As output, it obtains a list of generated or searched visual elements.

[0526] Step 4:

[0527] The device detects the user's facial expressions and reactions in real time and analyzes the user's emotions using an emotion engine. As input, the device's camera captures real-time video and applies facial recognition and emotion recognition algorithms (e.g., OpenCV). As data processing, the user's facial expression data is analyzed to identify the type of emotion (e.g., joy, surprise, interest). As output, emotion data is obtained as the analysis result.

[0528] Step 5:

[0529] The server adjusts the content and style of the generated visual elements based on the recognized user emotion. As input, the server receives emotion data and a list of visual elements. As data processing, it selects the visual elements that best fit the emotion data and dynamically adjusts their content and style. As output, it obtains the adjusted visual elements.

[0530] Step 6:

[0531] The server sends the final generated visual summary to the user's device, which displays it to the user. As input, it receives the adjusted visual elements and integrates them with the advertising copy. As data processing, it generates the final advertising content. As output, it displays the advertisement on the device display and allows the user to view the visual summary.

[0532] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0533] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0534] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0535] [Second embodiment]

[0536] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0537] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0538] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0539] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0540] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0541] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0542] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0543] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0544] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0545] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0546] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0547] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0548] The present invention relates to a system for visually summarizing text, and supports users in stress-free comprehension of lengthy text data such as conference materials, reports, and novels.

[0549] System Overview

[0550] The system has the following main functions:

[0551] 1. Receiving text data

[0552] 2. Text Analysis

[0553] 3. Extracting important keywords and phrases

[0554] 4. Visual element generation or search

[0555] 5. Visual Integration

[0556] 6. Output of generated images or videos

[0557] Specific processing flow

[0558] Server-side processing

[0559] Receiving input data

[0560] A user uploads text data such as meeting materials, reports, novels, etc. via a terminal.

[0561] The terminal transmits the document data selected by the user to the server.

[0562] The server receives the uploaded document data.

[0563] Text preprocessing

[0564] The server analyzes the received text data using natural language processing (NLP) technology.

[0565] The server divides the text into sentences and paragraphs and tokenizes each one.

[0566] The server tags each token with a part of speech and extracts key keywords and phrases.

[0567] Analysis for image and video generation

[0568] The server searches a database for relevant visual elements based on the extracted keywords and phrases.

[0569] The server generates new images and videos using generative AI models (e.g., Generative Adversarial Networks, GAN).

[0570] The server applies algorithms to optimize the placement and order of the generated visual elements to match the content of the original text.

[0571] Content Integration

[0572] The server integrates the generated visual elements in the appropriate order to create a summary in a format that allows the user to easily understand the information visually while preserving the story and logical structure of the entire text.

[0573] The server reviews the generated summary and makes any necessary corrections or optimizations.

[0574] Output of conversion results

[0575] The server transmits the generated image or video summary to the user's terminal.

[0576] The terminal displays the received summary to the user, allowing the user to understand the information without stress.

[0577] Specific examples

[0578] Example 1: Novel summary

[0579] The user selects the novel text file from the device and presses the upload button.

[0580] The terminal transmits the text file of the selected novel to the server.

[0581] The server analyzes the received novel data and extracts key scenes and characters.

[0582] The server generates related images and videos based on the extracted information.

[0583] The server integrates images and videos to create a summary based on the story of the novel.

[0584] The server transmits the generated summary to the terminal.

[0585] The terminal displays the received visual summary to the user.

[0586] Users view a visual summary to get an overview of the story.

[0587] Example 2: Summary of business meeting materials

[0588] The user selects a PDF file of business meeting materials from their device and presses the upload button.

[0589] The terminal transmits the selected business conference materials to the server.

[0590] The server analyzes the content of the document and extracts key points and data charts.

[0591] The server generates relevant images and charts based on the extracted information.

[0592] The server integrates the generated images and charts to create a summary of the conference materials.

[0593] The server sends the summary to the terminal.

[0594] The terminal displays the received visual summary to the user.

[0595] Users view a visual summary to grasp the main points of the meeting.

[0596] This system visually summarizes long or complex texts, helping users understand information efficiently and without stress.

[0597] The processing flow will be explained below.

[0598] Step 1:

[0599] The user selects and uploads long text data such as conference materials, reports, novels, etc. from the terminal.

[0600] Step 2:

[0601] The terminal transmits the selected text data to the server.

[0602] Step 3:

[0603] The server receives the uploaded document data.

[0604] Step 4:

[0605] The server analyzes the received text data using natural language processing (NLP) technology.

[0606] Step 5:

[0607] The server breaks the text into sentences and paragraphs and tokenizes each, breaking the text into smaller, analyzable units.

[0608] Step 6:

[0609] The server tags each token with a part of speech tag and extracts key keywords and phrases.

[0610] Step 7:

[0611] The server searches a database for related visual elements (images and videos) based on the extracted keywords and phrases.

[0612] Step 8:

[0613] The server may also generate new images and videos using generative AI models (e.g., Generative Adversarial Networks, GANs).

[0614] Step 9:

[0615] The server applies algorithms to optimize the placement and order of the generated visual elements to match the content of the original text.

[0616] Step 10:

[0617] The server integrates the generated visual elements in the appropriate order to create a summary while preserving the story and logical structure of the entire text.

[0618] Step 11:

[0619] The server reviews the generated summary and makes any necessary corrections or optimizations.

[0620] Step 12:

[0621] The server transmits the generated image or video summary to the user's terminal.

[0622] Step 13:

[0623] The terminal receives the visual summary result sent from the server.

[0624] Step 14:

[0625] The terminal displays the received images and videos to the user, allowing the user to understand the information without stress.

[0626] Step 15:

[0627] Users can view the visual summary and input the necessary information efficiently.

[0628] Example 1

[0629] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0630] There is a problem that it is difficult for users to understand long text data such as meeting materials, reports, and novels efficiently and without stress. In particular, extracting important parts and visually summarizing them requires a great deal of time and effort using conventional methods. Furthermore, if the generation and integration of visual elements is not performed properly, it becomes even more difficult to understand the information.

[0631] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0632] In this invention, the server includes means for receiving text data, means for analyzing the text data to extract important keywords and phrases, means for generating or searching for related visual elements based on the important keywords and phrases, means for integrating the related visual elements to generate an image or video that summarizes the text content, and means for outputting the generated image or video, thereby enabling a user to efficiently understand long text data in a visually summarized format.

[0633] The definition is given below.

[0634] "Text data" refers to data in which information such as meeting materials, reports, and novels is written in text format.

[0635] "Means for receiving" refers to the function by which the server receives text data sent by the user.

[0636] "Means of analysis" refers to the function by which the server uses natural language processing technology to divide and tokenize text data and understand its content.

[0637] "Means for extracting important keywords and phrases" refers to the function of selecting key words and short sentences from analyzed text data.

[0638] "Means for generating or searching for relevant visual elements" refers to the ability to create or search a database for images or videos based on extracted keywords or phrases.

[0639] "Means for integrating visual elements" refers to the function of combining the generated images and videos in an appropriate order and arrangement to visually express the content of the entire text.

[0640] "Means for generating a summarized image or video" refers to a function that produces a summary of a text from integrated visual elements.

[0641] "Means for outputting" refers to a function for transmitting the generated summary result to the user's terminal and displaying it.

[0642] "Natural language processing technology" refers to technology that enables computers to understand, interpret, and manipulate human language.

[0643] A "generative AI model" refers to an algorithm or learning model that uses artificial intelligence to generate new data (e.g., images or videos).

[0644] "Algorithms that optimize placement and order" refer to methods for determining the optimal placement and display order of visual elements to match the content of the original text.

[0645] This invention relates to a visual summarization system that allows users to understand long text data efficiently and without stress. This system analyzes the text data, extracts important keywords and phrases, generates or searches for related visual elements based on them, and integrates these visual elements to generate an image or video that summarizes the text content and provides it to the user.

[0646] System Overview

[0647] The system has the following main functions:

[0648] 1. Receiving text data

[0649] 2. Text Analysis

[0650] 3. Extracting important keywords and phrases

[0651] 4. Visual element generation or search

[0652] 5. Visual Integration

[0653] 6. Output of generated images or videos

[0654] Specifically, this is realized using the following hardware and software.

[0655] Hardware and software used

[0656] server

[0657] Common NLP libraries "SpaCy" or "NLTK" for using natural language processing (NLP) techniques

[0658] "Elasticsearch" as a search engine

[0659] "TensorFlow" or "PyTorch" frameworks for using generative AI models

[0660] OpenCV is an image processing library for optimizing the placement and order of visual elements.

[0661] Terminal

[0662] A computing device having an interface for a user to upload text data and display generated summaries.

[0663] Program processing

[0664] Specific data processing and data calculations are performed between the server, terminal, and user as follows:

[0665] 1. Receiving text data

[0666] Users upload the necessary text data (conference materials, reports, novels, etc.) from their terminals.

[0667] The terminal sends the text data selected by the user to the server using an HTTP POST request.

[0668] The server stores the uploaded text data in a specified directory.

[0669] 2. Text Analysis

[0670] The server analyzes the received text data using natural language processing (NLP) technology. Specifically, it divides the text into sentences and paragraphs, tokenizes each, and tags them by part of speech.

[0671] 3. Extracting important keywords and phrases

[0672] The server extracts key keywords and phrases based on part-of-speech tagged token information, and stores this information in a list format.

[0673] 4. Visual element generation or search

[0674] The server searches the database for relevant visual elements based on the extracted keywords or phrases. If no results are found, it generates new images or videos using a generative AI model (e.g., Generative Adversarial Networks, GAN). The generated visual elements are saved in a temporary directory.

[0675] 5. Visual Integration

[0676] The server then integrates the generated visual elements in the correct order to create a summary that preserves the overall story and logical structure of the text, allowing users to easily understand the information visually. The server then arranges the images according to the storyline, resizes them to the optimal size, and sets the appropriate placement coordinates. The results are then exported as a PDF or video.

[0677] 6. Output of generated images or videos

[0678] The server transmits the generated image or video summary to the user's terminal.

[0679] The terminal receives the response and displays it in the appropriate viewer.

[0680] Specific examples

[0681] Below, we will show the process of summarizing a novel and business conference materials as concrete examples of this system.

[0682] Example 1: Novel summary

[0683] 1. The user selects the novel text file from their device and presses the upload button.

[0684] 2. The terminal sends the text file of the selected novel to the server.

[0685] 3. The server analyzes the received novel data and extracts key scenes and characters.

[0686] 4. The server generates relevant images and videos based on the extracted information.

[0687] 5. The server integrates images and videos to create a summary based on the story of the novel.

[0688] 6. The server sends the generated summary to the terminal.

[0689] 7. The terminal displays the received visual summary to the user.

[0690] Here are some example prompts to input to the generative AI model:

[0691] "Generate a visual summary of the following novel: [Insert novel text here]. Focus on the main scenes and characters."

[0692] Example 2: Summary of business meeting materials

[0693] 1. The user selects the PDF file of the business meeting materials from the device and presses the upload button.

[0694] 2. The terminal transmits the selected business conference materials to the server.

[0695] 3. The server analyzes the content of the document and extracts important points and data charts.

[0696] 4. The server generates relevant images and charts based on the extracted information.

[0697] 5. The server integrates the generated images and charts to create a summary of the meeting materials.

[0698] 6. The server sends the summary to the terminal.

[0699] 7. The terminal displays the received visual summary to the user.

[0700] Here are some example prompts to input to the generative AI model:

[0701] "Create a visual summary of the following business meeting document: [Insert meeting document text here]. Highlight key points and data charts."

[0702] As described above, this system visually summarizes long sentences and complex content, helping users understand information efficiently and without stress.

[0703] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0704] Step 1:

[0705] The user uploads text data.

[0706] Input: Written data such as meeting materials, reports, novels, etc.

[0707] Operation: The user clicks the upload button on the device to upload the selected text data to the device.

[0708] Output: The text data is saved to the terminal.

[0709] Step 2:

[0710] The terminal transmits the text data to the server.

[0711] Input: User-uploaded text data

[0712] Operation: The terminal sends the received text data to the server via an HTTP POST request.

[0713] Output: The server receives the text data.

[0714] Step 3:

[0715] The server analyzes the text data.

[0716] Input: Text data sent from the terminal

[0717] How it works: The server uses a natural language processing (NLP) library to split the text data into sentences and paragraphs, and tokenize each. Specifically, it uses the NLP library "SpaCy" or "NLTK" and the nlp(text) function.

[0718] Output: Segmented and tokenized sentence data.

[0719] Step 4:

[0720] The server extracts important keywords and phrases.

[0721] Input: Segmented and tokenized text data

[0722] How it works: The server tags the segmented and tokenized data and extracts important keywords and phrases. Keywords are extracted based on specific parts of speech (nouns, verbs, adjectives).

[0723] Output: A list of key keywords and phrases.

[0724] Step 5:

[0725] The server generates or retrieves the visual elements.

[0726] Input: Extracted keywords or phrases

[0727] How it works: The server searches a database for relevant visual elements based on keywords or phrases. It uses Elasticsearch to execute the query, and if no results are found, it uses a generative AI model (e.g., Generative Adversarial Networks, GAN) to generate new images or videos. It leverages TensorFlow or PyTorch.

[0728] Output: Generated or retrieved visual elements (images or videos).

[0729] Step 6:

[0730] The server aggregates the visual elements.

[0731] Input: A generated or searched visual element

[0732] How it works: The server orders the images according to the storyline, adjusting their placement and size, and uses OpenCV to integrate the visual elements in an optimized order.

[0733] Output: Optimized and consolidated image or video summary.

[0734] Step 7:

[0735] The server sends the summary to the terminal.

[0736] Input: Optimized and integrated summary (image or video)

[0737] Action: The server generates a summary and sends it as an HTTP response.

[0738] Output: The terminal receives the summary.

[0739] Step 8:

[0740] The terminal displays the summary to the user.

[0741] Input: Abstract (image or video) sent from the server

[0742] Operation: The device displays the received summary in an appropriate viewer, ensuring that the information is visually easy to understand based on the UE / UX design.

[0743] Output: A visual summary that is displayed to the user.

[0744] In this way, the invention visually summarizes complex textual data, allowing users to efficiently grasp the information.

[0745] (Application example 1)

[0746] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0747] The challenge is to efficiently summarize large amounts of text content and present it in a visually easy-to-understand format, allowing users to understand the information without stress. There is a particular need to provide a means to easily grasp the content, even when it is large in volume or highly important.

[0748] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0749] In this invention, the server includes means for receiving text data, means for analyzing the text data to extract important keywords and phrases, means for generating or searching for related visual elements based on the important keywords and phrases, means for interactively displaying the generated images or videos, and means for sharing the generated images or videos with other users or on a social network, thereby enabling efficient summarization of vast amounts of text content and presenting it in a visually easy-to-understand format.

[0750] "Text data" refers to information or content provided in text format, including meeting materials, reports, novels, etc.

[0751] "Analysis" refers to the means and methods for understanding the content of text data and extracting important elements and patterns.

[0752] "Keywords" refer to words or phrases that have particularly important meanings in a text, and are essential elements for summarizing the entire content.

[0753] A "phrase" is a group of words consisting of a series of multiple words, and is an expression that conveys a specific meaning or information.

[0754] A "visual element" is a visually recognizable object, such as an image or video, that is used to visually summarize text.

[0755] "Generation" refers to the process of creating new images or videos, which is done using artificial intelligence models.

[0756] "Searching" is the process of finding relevant data from existing databases and sources.

[0757] "Interactive display" refers to a display method that allows users to manipulate or take action on the displayed information.

[0758] "Sharing" refers to distributing the generated visuals and information to other users or through social networks.

[0759] "Tokenization" is the process of breaking down text data into individual words and phrases, and is a fundamental step in natural language processing.

[0760] A "generative AI model" is an artificial intelligence model that generates new data using technologies such as deep learning, and includes generative adversarial networks (GANs).

[0761] This invention describes a system for efficiently summarizing large amounts of textual content and presenting it in a visually accessible format. This system is particularly applicable to content distribution services and is designed to enable users to quickly understand long-form content such as news articles or novels.

[0762] System Overview

[0763] The system consists of the following major hardware and software components:

[0764] Hardware: Smartphone (iOS or Android), server

[0765] Software: Python, spacy, Summarizer, DeepAI, Pillow

[0766] Program processing flow

[0767] 1. Receiving text data:

[0768] The user enters the URL or text data selected within the content distribution service into the device, which then sends it to the server.

[0769] 2. Text analysis:

[0770] The server analyzes the received text data using natural language processing (NLP) techniques, specifically tokenizing the text using the spacy library and extracting important keywords and phrases.

[0771] 3. Visual element generation or search:

[0772] Based on the extracted keywords and phrases, the server uses generative AI models (e.g., Generative Adversarial Networks, or GANs) to generate new images and videos, and, if necessary, uses DeepAI's API to search for related images from an existing database.

[0773] 4. Interactive display:

[0774] The server generates visual elements and sends them to the user's device, which displays them interactively. Users can view a visual summary and tap on areas of interest to see more detailed information.

[0775] 5. Sharing features:

[0776] The generated visual summary can be shared by the user with other users or on social networks, and the device provides an interface for sharing this information with other users.

[0777] Specific examples

[0778] Example 1: News article summary

[0779] A user enters a news article URL into the app.

[0780] The app sends the URL to a server, which analyzes the article content and generates a summary.

[0781] The generated summary and associated images are sent to the terminal, where the user can view them interactively.

[0782] If desired, the summary can also be shared on social media.

[0783] Example 2: Technical document summary

[0784] The user uploads the text data of the technical document to the app.

[0785] The app sends the text data to a server, which analyzes the document and generates a summary.

[0786] A visual summary containing key points is generated that users can easily check on their smartphone.

[0787] Prompt Sentence Examples

[0788] "Create a summary and visual summary of the news article https: / / example.com / article."

[0789] In this way, the present invention provides a concrete example for efficiently summarizing a huge amount of text content and enabling users to understand it without stress.

[0790] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0791] Step 1:

[0792] The user selects text content of interest within the content distribution service (for example, a news article URL or long text data) and enters it into the device. The device then sends it to the server. The input in this step is the URL or text data, and the output is data sent to the server.

[0793] Step 2:

[0794] The server analyzes the received text data. Specifically, it uses the Python natural language processing library spacy to tokenize the data and extract important keywords and phrases. The input in this step is the received text data, and the output is the extracted keywords and phrases.

[0795] Step 3:

[0796] The server generates new images and videos using generative AI models such as Generative Adversarial Networks (GANs) based on the extracted keywords and phrases. It also searches for related images from a database using DeepAI's API as needed. The input in this step is the extracted keywords and phrases, and the output is the generated or searched visual elements (images and videos).

[0797] Step 4:

[0798] The server integrates the generated visual elements to create an interactive visual summary, and then constructs an interface that allows users to easily manipulate and view the visual summary. The input in this step is the generated visual elements, and the output is an interactive visual summary.

[0799] Step 5:

[0800] The server sends the interactive visual summary to the user's device for the user to review. If necessary, the user can tap on the part of the visual summary that they want to know more about to view the details. The input in this step is the interactive visual summary, and the output is the visual summary displayed on the user's device.

[0801] Step 6:

[0802] The device provides an interface for sharing the generated visual summary with other users or on social networks. Users can easily share their visual summaries with others. The inputs in this step are the visual summary and the sharer information, and the output is the shared visual summary.

[0803] These are the specific processing steps of the system program that realizes this application example, which enables users to efficiently understand large amounts of text content and share visual summaries.

[0804] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0805] The present invention relates to a system that recognizes a user's emotions and visually summarizes text, thereby helping users understand long conference materials, reports, novels, etc. without stress.

[0806] System Overview

[0807] The system has the following main functions:

[0808] 1. Receiving text data

[0809] 2. Text Analysis

[0810] 3. Extracting important keywords and phrases

[0811] 4. Visual element generation or search

[0812] 5. Emotion Recognition by Emotion Engine

[0813] 6. Visual Integration

[0814] 7. Output of generated images or videos

[0815] Specific processing flow

[0816] Server-side processing

[0817] Receiving input data

[0818] A user uploads text data such as meeting materials, reports, novels, etc. via a terminal.

[0819] The terminal transmits the document data selected by the user to the server.

[0820] The server receives the uploaded document data.

[0821] Text preprocessing

[0822] The server analyzes the received text data using natural language processing (NLP) technology.

[0823] The server divides the text into sentences and paragraphs and tokenizes each one.

[0824] The server tags each token with a part of speech and extracts key keywords and phrases.

[0825] Analysis for image and video generation

[0826] The server searches a database for relevant visual elements based on the extracted keywords and phrases.

[0827] The server may also generate new images and videos using generative AI models (e.g., Generative Adversarial Networks, GANs).

[0828] The server applies algorithms to optimize the placement and order of the generated visual elements to match the content of the original text.

[0829] emotion recognition

[0830] The device analyzes the user's facial expressions and reactions as they view visual elements, and uses an emotion engine to recognize the user's emotions.

[0831] The server adjusts the content and style of the generated visual elements based on the recognized user emotion.

[0832] The server changes the format of the summary results to be presented depending on the user's emotions.

[0833] Content Integration

[0834] The server integrates the generated visual elements in the appropriate order to create a summary while preserving the story and logical structure of the entire text.

[0835] The server reviews the generated summary and makes any necessary corrections or optimizations.

[0836] Output of conversion results

[0837] The server transmits the generated image or video summary to the user's terminal.

[0838] The terminal displays the received summary to the user, allowing the user to understand the information without stress.

[0839] Specific examples

[0840] Example 1: Novel summary

[0841] The user selects the novel text file from the device and presses the upload button.

[0842] The terminal transmits the text file of the selected novel to the server.

[0843] The server analyzes the received novel data and extracts key scenes and characters.

[0844] The server generates related images and videos based on the extracted information.

[0845] The server integrates images and videos as a summary according to the story of the novel.

[0846] The server adjusts the content and style based on the perceived user sentiment.

[0847] The server transmits the generated summary to the terminal.

[0848] The terminal displays the received visual summary to the user.

[0849] Users view a visual summary to get an overview of the story.

[0850] Example 2: Summary of business meeting materials

[0851] The user selects a PDF file of business meeting materials from their device and presses the upload button.

[0852] The terminal transmits the selected business conference materials to the server.

[0853] The server analyzes the content of the document and extracts key points and data charts.

[0854] The server generates relevant images and charts based on the extracted information.

[0855] The server integrates the generated images and charts to create a summary of the conference materials.

[0856] The server adjusts the content and style based on the perceived user sentiment.

[0857] The server transmits the generated summary to the terminal.

[0858] The terminal displays the received visual summary to the user.

[0859] Users view a visual summary to grasp the main points of the meeting.

[0860] This system not only visually summarizes long or complex texts, but also adjusts the way information is presented to take the user's emotions into account, thereby deepening their understanding and reducing stress.

[0861] The processing flow will be explained below.

[0862] Step 1:

[0863] The user selects and uploads long text data such as conference materials, reports, novels, etc. from the terminal.

[0864] Step 2:

[0865] The terminal transmits the selected text data to the server.

[0866] Step 3:

[0867] The server receives the uploaded document data.

[0868] Step 4:

[0869] The server analyzes the received text data using natural language processing (NLP) technology.

[0870] Step 5:

[0871] The server breaks the text into sentences and paragraphs and tokenizes each, breaking the text into smaller, analyzable units.

[0872] Step 6:

[0873] The server tags each token with a part of speech tag and extracts key keywords and phrases.

[0874] Step 7:

[0875] The server searches a database for related visual elements (images and videos) based on the extracted keywords and phrases.

[0876] Step 8:

[0877] The server may also generate new images and videos using generative AI models (e.g., Generative Adversarial Networks, GANs).

[0878] Step 9:

[0879] The server applies algorithms to optimize the placement and order of the generated visual elements to match the content of the original text.

[0880] Step 10:

[0881] The device uses cameras and sensors to capture the user's facial expressions and reactions as they view visual elements.

[0882] Step 11:

[0883] The terminal sends the captured data to an emotion engine to analyze the user's emotions.

[0884] Step 12:

[0885] The server receives the analyzed user emotion data and adjusts the content and style of the generated visual elements.

[0886] Step 13:

[0887] The server changes the format of the summary results to be presented depending on the user's emotions.

[0888] Step 14:

[0889] The server integrates the generated visual elements in the appropriate order to create a summary while preserving the story and logical structure of the entire text.

[0890] Step 15:

[0891] The server reviews the generated summary and makes any necessary corrections or optimizations.

[0892] Step 16:

[0893] The server transmits the generated image or video summary to the user's terminal.

[0894] Step 17:

[0895] The terminal receives the visual summary result sent from the server.

[0896] Step 18:

[0897] The terminal displays the received images and videos to the user, allowing the user to understand the information without stress.

[0898] Step 19:

[0899] Users can view the visual summary and input the necessary information efficiently.

[0900] Example 2

[0901] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0902] In today's world, there is a huge amount of written information flying around, and people need to be able to quickly and efficiently retrieve the information they need. However, when users read long documents such as meeting materials, reports, or novels, it takes time and effort to understand and summarize all of the content. Furthermore, if information is presented without taking into account the emotions that individual users experience when comprehending the information, it can lead to a shallow understanding or stress.

[0903] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving text data, means for analyzing the text data and extracting important keywords and phrases, means for generating or searching for related visual elements based on the important keywords and phrases, means for analyzing a user's emotions and adjusting the content and style of the related visual elements, means for integrating the related visual elements and generating an image or video that summarizes the text content, and means for outputting the generated image or video. This enables a user to efficiently understand long sentences or complex information in a visually summarized form, and further, information presented in accordance with the user's emotions deepens understanding and reduces stress.

[0904] The "means for receiving text data" is a function by which the server receives text data uploaded by the user through the terminal.

[0905] The "means for extracting important keywords and phrases" is a function that analyzes received text data and finds the most important keywords and phrases within it based on the internal structure and context.

[0906] "Means for generating or searching for relevant visual elements" refers to a function that searches for appropriate visual elements from a database based on extracted keywords or phrases, or generates new images or videos using a generative AI model.

[0907] "Means for analyzing the user's emotions and adjusting the content and style of the related visual elements" refers to a function that analyzes the user's facial expressions and reactions through the terminal and changes the content and style of the visual elements based on the results.

[0908] "Means for integrating the related visual elements and generating an image or video that summarizes the content of the text" refers to a function that combines the generated visual elements in an appropriate order to create a single image or video that clearly summarizes the content of the original text.

[0909] The "means for outputting the generated image or video" is a function for transmitting the image or video that is the integrated summary result to the user's terminal and displaying it.

[0910] This invention is a visual summarization support system that helps users understand long text data. This system combines natural language processing technology, generative AI models, and emotion recognition technology to analyze text data and generate visual summaries.

[0911] First, a user uploads text data such as meeting materials, reports, or novels using a terminal. The uploaded text data is then sent to a server via the terminal. The server temporarily stores the received text data and uses it for subsequent analysis.

[0912] The server analyzes the text data using natural language processing (NLP) techniques. Examples of NLP tools used include SpaCy and NLTK. The server divides the text data into sentences and paragraphs and tokenizes each. This process may involve using a tokenization algorithm such as the BERT model. The server then tags each token with a part of speech and extracts important keywords and phrases. This process uses techniques such as TF-IDF and Word2Vec.

[0913] The server then searches a database for relevant visual elements based on the extracted keywords or phrases, and generates new images or videos using generative AI models (e.g., Generative Adversarial Networks, GANs) if necessary. The resulting visual elements are then optimized in their placement and order to match the content of the original text.

[0914] Furthermore, the device analyzes the user's facial expressions and reactions when viewing visual elements through the camera, and recognizes the user's emotions using an emotion engine (e.g., OpenCV or Facial Emotion Recognition API).The server adjusts the content and style of the generated visual elements based on the recognized user emotions, and presents optimal information according to the user's emotions.

[0915] The server then integrates the generated visual elements in the correct order to create a summary that preserves the overall story and logical structure of the text. The server then reviews the integrated summary and makes any necessary corrections or optimizations. The generated image or video summary is then sent to the user's device, where it is displayed, allowing the user to easily understand the information.

[0916] Below, examples of prompt sentences are shown as concrete examples of implementing this system.

[0917] Example prompt sentence:

[0918] "The system receives specified text data, analyzes it using natural language processing technology, and extracts important keywords and phrases. It then generates or searches for related visual elements based on the extracted keywords and phrases. It analyzes the user's sentiment and adjusts the content and style of the generated visual elements. Finally, it integrates the related visual elements to generate an image or video that summarizes the text content and outputs it to the user's device."

[0919] The present invention enables a user to efficiently understand long sentences or complex text data through visual summaries, and furthermore, by presenting information according to the user's emotions, the user's understanding is deepened and stress is reduced.

[0920] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0921] Step 1:

[0922] Receiving input data

[0923] A user uploads text data such as meeting materials, reports, novels, etc. via a terminal.

[0924] Input: A text file selected by the user (PDF, TXT, DOCX, etc.)

[0925] Output: A text file temporarily saved on the device

[0926] The terminal sends the selected text data to the server, typically using an HTTP POST request.

[0927] Input: A text file uploaded by the user

[0928] Output: Text file sent to the server

[0929] The server receives the uploaded text data and temporarily stores it.

[0930] Input: Text file sent from the terminal

[0931] Output: Temporarily saved text file

[0932] Step 2:

[0933] Text preprocessing

[0934] The server analyzes the received text data using natural language processing (NLP) techniques, such as SpaCy and NLTK.

[0935] Input: Saved text data

[0936] Output: Tokenized data split into sentences and paragraphs

[0937] The server splits the text into sentences and paragraphs and tokenizes each of them, using a model such as BERT.

[0938] Input: Data for the entire sentence

[0939] Output: An array of tokens

[0940] The server tags each token with a part of speech and extracts key keywords and phrases, using TF-IDF and Word2Vec.

[0941] Input: Tokenized data

[0942] Output: Part-of-speech tagged tokens and key keywords and phrases

[0943] Step 3:

[0944] Extracting important keywords and phrases

[0945] The server compiles the extracted keywords and phrases and ranks them by importance.

[0946] Input: Part-of-speech tagged tokens and extracted keywords and phrases

[0947] Output: A list of keywords or phrases ranked by importance

[0948] Step 4:

[0949] Analysis for image and video generation

[0950] The server searches a database for relevant visual elements based on the extracted keywords or phrases, using an image search API (e.g., a general image search API).

[0951] Input: Ranking keyword or phrase

[0952] Output: A list of related visual elements

[0953] The server may also generate new images and videos using generative AI models (e.g., Generative Adversarial Networks, GANs).

[0954] Input: Keyword or phrase

[0955] Output: The new image or video generated

[0956] The server applies algorithms to optimize the placement and order of the generated visual elements to match the content of the original text.

[0957] Input: A list of visual elements

[0958] Output: properly positioned and ordered visual elements

[0959] Step 5:

[0960] emotion recognition

[0961] The device analyzes the user's facial expressions and reactions when viewing visual elements through the camera and recognizes the user's emotions using an emotion engine (e.g., a general emotion recognition API).

[0962] Input: Camera video data

[0963] Output: Recognized user emotion data

[0964] The server adjusts the content and style of the generated visual elements based on the recognized user emotion.

[0965] Input: User emotion data and visual elements

[0966] Output: Adjusted visual elements

[0967] The server changes the format of the summary results to be presented depending on the user's emotions.

[0968] Input: User emotion data and summary results

[0969] Output: Summarization results according to emotions

[0970] Step 6:

[0971] Content Integration

[0972] The server integrates the generated visual elements in the appropriate order to create a summary while preserving the story and logical structure of the entire text.

[0973] Input: Optimized visual elements

[0974] Output: Integrated summary image or video

[0975] The server reviews the generated summary and makes any necessary corrections or optimizations.

[0976] Input: Synthesized summary image or video

[0977] Output: revised and optimized summary

[0978] Step 7:

[0979] Output of conversion results

[0980] The server sends the generated image or video summary to the user's device, typically using an HTTP POST request.

[0981] Input: Corrected and optimized summary

[0982] Output: Summary sent to the user's terminal

[0983] The terminal displays the received summary to the user, allowing the user to understand the information without stress.

[0984] Input: Submitted Abstract

[0985] Output: The summary displayed to the user

[0986] This allows users to efficiently understand long or complex textual data through visual summaries.In addition, the way information is presented is adapted according to the user's emotions, deepening understanding and enabling information acquisition with less stress.

[0987] (Application example 2)

[0988] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0989] In today's world, people are bombarded with a vast amount of information, and in the field of advertising in particular, highly expressive content and content that takes into account the recipient's emotions are required to attract attention. However, conventional ad generation methods have difficulty dynamically changing content in response to user emotions, resulting in the inability to provide effective ads. To solve this problem, technology is needed that can recognize user emotions in real time and generate ads by combining appropriate visual elements and text content.

[0990] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0991] In this invention, the server includes means for receiving text data, means for analyzing the text data and extracting important keywords and phrases, means for generating or searching for related visual elements based on the important keywords and phrases, means for recognizing a user's emotions, means for adjusting the content and style of the generated visual elements based on the user's emotions, and means for outputting the generated images or videos, thereby enabling effective advertisement generation according to the user's emotions in real time.

[0992] "Text data" refers to information in text format provided by users, including content such as advertising copy and product descriptions.

[0993] "Analysis" is the process of breaking down text data into sentences and paragraphs and extracting important keywords and phrases.

[0994] "Keywords and phrases" refer to words and expressions that are particularly important in text data and express the gist of the information.

[0995] "Visual elements" refer to elements such as images and videos used to visually convey information to users.

[0996] "Integration" refers to the process of combining visual elements generated or retrieved based on extracted keywords or phrases and displaying them in a cohesive form.

[0997] A "generative AI model" refers to a model that uses artificial intelligence technology to generate new images and videos.

[0998] "Emotion recognition" is the process of detecting and classifying emotions from a user's facial expressions and reactions.

[0999] "Adjustment" is the procedure of dynamically changing the content and style of visual elements based on the user's emotions.

[1000] "Output" refers to the processing of the generated visual elements and integrated advertising content for display to the user.

[1001] The present invention is applied to a system that recognizes a user's emotions and visually summarizes long texts. The system for implementing the present invention is realized by linking a server and a user's terminal.

[1002] System configuration

[1003] The system includes the following main components:

[1004] 1. Server:

[1005] Includes software that implements natural language processing (NLP) techniques to analyze text and extract important keywords and phrases.

[1006] Includes software that implements generative AI models (e.g., Generative Adversarial Networks, GAN) to generate or search for relevant visual elements.

[1007] It includes an emotion engine that recognizes user emotions and adjusts the content and style of visual elements.

[1008] It includes software for generating and outputting a summary result format according to the user's emotions.

[1009] 2. On the user's device:

[1010] Includes software that provides an interface for users to input textual data. Example: smartphone applications.

[1011] It includes a camera for detecting the user's facial expressions and reactions in real time, and software for analyzing facial expressions, e.g., OpenCV library.

[1012] Includes software for displaying the visual summary received from the server to the user. Example: smartphone display.

[1013] Processing flow

[1014] 1. Receiving input data:

[1015] A user inputs text data such as advertising copy and product descriptions using a smartphone application.

[1016] 2. Text Analysis:

[1017] The server analyzes the received text data and extracts important keywords and phrases using natural language processing techniques (e.g., the TextBlob library).

[1018] 3. Visual element generation:

[1019] Based on the extracted keywords and phrases, relevant visual elements (images and videos) are generated or searched for, using a generative AI model (e.g., GAN).

[1020] 4. Emotion recognition:

[1021] While the user is viewing the advertisement on the device, the camera detects facial expressions in real time and analyzes the user's emotions using an emotion engine, which uses the OpenCV library.

[1022] 5. Content Adjustment:

[1023] Dynamically adjust the content and style of visual elements based on recognized user emotions.

[1024] 6. Output:

[1025] The final visual summary is sent to the user's terminal and displayed.

[1026] Specific examples

[1027] For example, suppose a user enters the following ad copy for a newly released cosmetic product: "New cosmetics! The secret to making your skin glow is here." The server analyzes this text and extracts key keywords and phrases (e.g., "new release," "cosmetics," "the secret to glowing"). It then uses a generative AI model to generate visual elements that match these keywords. When a user views an ad, the device's camera detects the user's facial expression of interest and selects visual elements and short video clips that best match that emotion to display the ad. This series of processes enables effective ad generation.

[1028] Prompt Sentence Examples

[1029] "If the user has an interested expression, the system will generate the most resonant visuals and short video clips to pair with this ad copy to create an ad."

[1030] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1031] Step 1:

[1032] A user uses a smartphone application to input text data such as advertising copy or product descriptions. As input, the user enters text into a text box and presses the "Send" button. As output, the device sends this text data to a server.

[1033] Step 2:

[1034] The server analyzes the received text data. As input, the server receives text data. To process the data, it uses natural language processing techniques (e.g., the TextBlob library) to split the data into sentences and paragraphs, tokenize them, and extract important keywords and phrases. As output, it generates a list of these keywords and phrases.

[1035] Step 3:

[1036] The server generates or searches for relevant visual elements based on the extracted keywords or phrases. As input, it receives a list of keywords and uses a generative AI model (e.g., GAN) to generate or search for appropriate images or videos from a database. As output, it obtains a list of generated or searched visual elements.

[1037] Step 4:

[1038] The device detects the user's facial expressions and reactions in real time and analyzes the user's emotions using an emotion engine. As input, the device's camera captures real-time video and applies facial recognition and emotion recognition algorithms (e.g., OpenCV). As data processing, the user's facial expression data is analyzed to identify the type of emotion (e.g., joy, surprise, interest). As output, emotion data is obtained as the analysis result.

[1039] Step 5:

[1040] The server adjusts the content and style of the generated visual elements based on the recognized user emotion. As input, the server receives emotion data and a list of visual elements. As data processing, it selects the visual elements that best fit the emotion data and dynamically adjusts their content and style. As output, it obtains the adjusted visual elements.

[1041] Step 6:

[1042] The server sends the final generated visual summary to the user's device, which displays it to the user. As input, it receives the adjusted visual elements and integrates them with the advertising copy. As data processing, it generates the final advertising content. As output, it displays the advertisement on the device display and allows the user to view the visual summary.

[1043] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1044] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1045] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1046] [Third embodiment]

[1047] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1048] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1049] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1050] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1051] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1052] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1053] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1054] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1055] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1056] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1057] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1058] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1059] The present invention relates to a system for visually summarizing text, and supports users in stress-free comprehension of lengthy text data such as conference materials, reports, and novels.

[1060] System Overview

[1061] The system has the following main functions:

[1062] 1. Receiving text data

[1063] 2. Text Analysis

[1064] 3. Extracting important keywords and phrases

[1065] 4. Visual element generation or search

[1066] 5. Visual Integration

[1067] 6. Output of generated images or videos

[1068] Specific processing flow

[1069] Server-side processing

[1070] Receiving input data

[1071] A user uploads text data such as meeting materials, reports, novels, etc. via a terminal.

[1072] The terminal transmits the document data selected by the user to the server.

[1073] The server receives the uploaded document data.

[1074] Text preprocessing

[1075] The server analyzes the received text data using natural language processing (NLP) technology.

[1076] The server divides the text into sentences and paragraphs and tokenizes each one.

[1077] The server tags each token with a part of speech and extracts key keywords and phrases.

[1078] Analysis for image and video generation

[1079] The server searches a database for relevant visual elements based on the extracted keywords and phrases.

[1080] The server generates new images and videos using generative AI models (e.g., Generative Adversarial Networks, GAN).

[1081] The server applies algorithms to optimize the placement and order of the generated visual elements to match the content of the original text.

[1082] Content Integration

[1083] The server integrates the generated visual elements in the appropriate order to create a summary in a format that allows the user to easily understand the information visually while preserving the story and logical structure of the entire text.

[1084] The server reviews the generated summary and makes any necessary corrections or optimizations.

[1085] Output of conversion results

[1086] The server transmits the generated image or video summary to the user's terminal.

[1087] The terminal displays the received summary to the user, allowing the user to understand the information without stress.

[1088] Specific examples

[1089] Example 1: Novel summary

[1090] The user selects the novel text file from the device and presses the upload button.

[1091] The terminal transmits the text file of the selected novel to the server.

[1092] The server analyzes the received novel data and extracts key scenes and characters.

[1093] The server generates related images and videos based on the extracted information.

[1094] The server integrates images and videos to create a summary based on the story of the novel.

[1095] The server transmits the generated summary to the terminal.

[1096] The terminal displays the received visual summary to the user.

[1097] Users view a visual summary to get an overview of the story.

[1098] Example 2: Summary of business meeting materials

[1099] The user selects a PDF file of business meeting materials from their device and presses the upload button.

[1100] The terminal transmits the selected business conference materials to the server.

[1101] The server analyzes the content of the document and extracts key points and data charts.

[1102] The server generates relevant images and charts based on the extracted information.

[1103] The server integrates the generated images and charts to create a summary of the conference materials.

[1104] The server sends the summary to the terminal.

[1105] The terminal displays the received visual summary to the user.

[1106] Users view a visual summary to grasp the main points of the meeting.

[1107] This system visually summarizes long or complex texts, helping users understand information efficiently and without stress.

[1108] The processing flow will be explained below.

[1109] Step 1:

[1110] The user selects and uploads long text data such as conference materials, reports, novels, etc. from the terminal.

[1111] Step 2:

[1112] The terminal transmits the selected text data to the server.

[1113] Step 3:

[1114] The server receives the uploaded document data.

[1115] Step 4:

[1116] The server analyzes the received text data using natural language processing (NLP) technology.

[1117] Step 5:

[1118] The server breaks the text into sentences and paragraphs and tokenizes each, breaking the text into smaller, analyzable units.

[1119] Step 6:

[1120] The server tags each token with a part of speech tag and extracts key keywords and phrases.

[1121] Step 7:

[1122] The server searches a database for related visual elements (images and videos) based on the extracted keywords and phrases.

[1123] Step 8:

[1124] The server may also generate new images and videos using generative AI models (e.g., Generative Adversarial Networks, GANs).

[1125] Step 9:

[1126] The server applies algorithms to optimize the placement and order of the generated visual elements to match the content of the original text.

[1127] Step 10:

[1128] The server integrates the generated visual elements in the appropriate order to create a summary while preserving the story and logical structure of the entire text.

[1129] Step 11:

[1130] The server reviews the generated summary and makes any necessary corrections or optimizations.

[1131] Step 12:

[1132] The server transmits the generated image or video summary to the user's terminal.

[1133] Step 13:

[1134] The terminal receives the visual summary result sent from the server.

[1135] Step 14:

[1136] The terminal displays the received images and videos to the user, allowing the user to understand the information without stress.

[1137] Step 15:

[1138] Users can view the visual summary and input the necessary information efficiently.

[1139] Example 1

[1140] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1141] There is a problem that it is difficult for users to understand long text data such as meeting materials, reports, and novels efficiently and without stress. In particular, extracting important parts and visually summarizing them requires a great deal of time and effort using conventional methods. Furthermore, if the generation and integration of visual elements is not performed properly, it becomes even more difficult to understand the information.

[1142] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1143] In this invention, the server includes means for receiving text data, means for analyzing the text data to extract important keywords and phrases, means for generating or searching for related visual elements based on the important keywords and phrases, means for integrating the related visual elements to generate an image or video that summarizes the text content, and means for outputting the generated image or video, thereby enabling a user to efficiently understand long text data in a visually summarized format.

[1144] The definition is given below.

[1145] "Text data" refers to data in which information such as meeting materials, reports, and novels is written in text format.

[1146] "Means for receiving" refers to the function by which the server receives text data sent by the user.

[1147] "Means of analysis" refers to the function by which the server uses natural language processing technology to divide and tokenize text data and understand its content.

[1148] "Means for extracting important keywords and phrases" refers to the function of selecting key words and short sentences from analyzed text data.

[1149] "Means for generating or searching for relevant visual elements" refers to the ability to create or search a database for images or videos based on extracted keywords or phrases.

[1150] "Means for integrating visual elements" refers to the function of combining the generated images and videos in an appropriate order and arrangement to visually express the content of the entire text.

[1151] "Means for generating a summarized image or video" refers to a function that produces a summary of a text from integrated visual elements.

[1152] "Means for outputting" refers to a function for transmitting the generated summary result to the user's terminal and displaying it.

[1153] "Natural language processing technology" refers to technology that enables computers to understand, interpret, and manipulate human language.

[1154] A "generative AI model" refers to an algorithm or learning model that uses artificial intelligence to generate new data (e.g., images or videos).

[1155] "Algorithms that optimize placement and order" refer to methods for determining the optimal placement and display order of visual elements to match the content of the original text.

[1156] This invention relates to a visual summarization system that allows users to understand long text data efficiently and without stress. This system analyzes the text data, extracts important keywords and phrases, generates or searches for related visual elements based on them, and integrates these visual elements to generate an image or video that summarizes the text content and provides it to the user.

[1157] System Overview

[1158] The system has the following main functions:

[1159] 1. Receiving text data

[1160] 2. Text Analysis

[1161] 3. Extracting important keywords and phrases

[1162] 4. Visual element generation or search

[1163] 5. Visual Integration

[1164] 6. Output of generated images or videos

[1165] Specifically, this is realized using the following hardware and software.

[1166] Hardware and software used

[1167] server

[1168] Common NLP libraries "SpaCy" or "NLTK" for using natural language processing (NLP) techniques

[1169] "Elasticsearch" as a search engine

[1170] "TensorFlow" or "PyTorch" frameworks for using generative AI models

[1171] OpenCV is an image processing library for optimizing the placement and order of visual elements.

[1172] Terminal

[1173] A computing device having an interface for a user to upload text data and display generated summaries.

[1174] Program processing

[1175] Specific data processing and data calculations are performed between the server, terminal, and user as follows:

[1176] 1. Receiving text data

[1177] Users upload the necessary text data (conference materials, reports, novels, etc.) from their terminals.

[1178] The terminal sends the text data selected by the user to the server using an HTTP POST request.

[1179] The server stores the uploaded text data in a specified directory.

[1180] 2. Text Analysis

[1181] The server analyzes the received text data using natural language processing (NLP) technology. Specifically, it divides the text into sentences and paragraphs, tokenizes each, and tags them by part of speech.

[1182] 3. Extracting important keywords and phrases

[1183] The server extracts key keywords and phrases based on part-of-speech tagged token information, and stores this information in a list format.

[1184] 4. Visual element generation or search

[1185] The server searches the database for relevant visual elements based on the extracted keywords or phrases. If no results are found, it generates new images or videos using a generative AI model (e.g., Generative Adversarial Networks, GAN). The generated visual elements are saved in a temporary directory.

[1186] 5. Visual Integration

[1187] The server then integrates the generated visual elements in the correct order to create a summary that preserves the overall story and logical structure of the text, allowing users to easily understand the information visually. The server then arranges the images according to the storyline, resizes them to the optimal size, and sets the appropriate placement coordinates. The results are then exported as a PDF or video.

[1188] 6. Output of generated images or videos

[1189] The server transmits the generated image or video summary to the user's terminal.

[1190] The terminal receives the response and displays it in the appropriate viewer.

[1191] Specific examples

[1192] Below, we will show the process of summarizing a novel and business conference materials as concrete examples of this system.

[1193] Example 1: Novel summary

[1194] 1. The user selects the novel text file from their device and presses the upload button.

[1195] 2. The terminal sends the text file of the selected novel to the server.

[1196] 3. The server analyzes the received novel data and extracts key scenes and characters.

[1197] 4. The server generates relevant images and videos based on the extracted information.

[1198] 5. The server integrates images and videos to create a summary based on the story of the novel.

[1199] 6. The server sends the generated summary to the terminal.

[1200] 7. The terminal displays the received visual summary to the user.

[1201] Here are some example prompts to input to the generative AI model:

[1202] "Generate a visual summary of the following novel: [Insert novel text here]. Focus on the main scenes and characters."

[1203] Example 2: Summary of business meeting materials

[1204] 1. The user selects the PDF file of the business meeting materials from the device and presses the upload button.

[1205] 2. The terminal transmits the selected business conference materials to the server.

[1206] 3. The server analyzes the content of the document and extracts important points and data charts.

[1207] 4. The server generates relevant images and charts based on the extracted information.

[1208] 5. The server integrates the generated images and charts to create a summary of the meeting materials.

[1209] 6. The server sends the summary to the terminal.

[1210] 7. The terminal displays the received visual summary to the user.

[1211] Here are some example prompts to input to the generative AI model:

[1212] "Create a visual summary of the following business meeting document: [Insert meeting document text here]. Highlight key points and data charts."

[1213] As described above, this system visually summarizes long sentences and complex content, helping users understand information efficiently and without stress.

[1214] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1215] Step 1:

[1216] The user uploads text data.

[1217] Input: Written data such as meeting materials, reports, novels, etc.

[1218] Operation: The user clicks the upload button on the device to upload the selected text data to the device.

[1219] Output: The text data is saved to the terminal.

[1220] Step 2:

[1221] The terminal transmits the text data to the server.

[1222] Input: User-uploaded text data

[1223] Operation: The terminal sends the received text data to the server via an HTTP POST request.

[1224] Output: The server receives the text data.

[1225] Step 3:

[1226] The server analyzes the text data.

[1227] Input: Text data sent from the terminal

[1228] How it works: The server uses a natural language processing (NLP) library to split the text data into sentences and paragraphs, and tokenize each. Specifically, it uses the NLP library "SpaCy" or "NLTK" and the nlp(text) function.

[1229] Output: Segmented and tokenized sentence data.

[1230] Step 4:

[1231] The server extracts important keywords and phrases.

[1232] Input: Segmented and tokenized text data

[1233] How it works: The server tags the segmented and tokenized data and extracts important keywords and phrases. Keywords are extracted based on specific parts of speech (nouns, verbs, adjectives).

[1234] Output: A list of key keywords and phrases.

[1235] Step 5:

[1236] The server generates or retrieves the visual elements.

[1237] Input: Extracted keywords or phrases

[1238] How it works: The server searches a database for relevant visual elements based on keywords or phrases. It uses Elasticsearch to execute the query, and if no results are found, it uses a generative AI model (e.g., Generative Adversarial Networks, GAN) to generate new images or videos. It leverages TensorFlow or PyTorch.

[1239] Output: Generated or retrieved visual elements (images or videos).

[1240] Step 6:

[1241] The server aggregates the visual elements.

[1242] Input: A generated or searched visual element

[1243] How it works: The server orders the images according to the storyline, adjusting their placement and size, and uses OpenCV to integrate the visual elements in an optimized order.

[1244] Output: Optimized and consolidated image or video summary.

[1245] Step 7:

[1246] The server sends the summary to the terminal.

[1247] Input: Optimized and integrated summary (image or video)

[1248] Action: The server generates a summary and sends it as an HTTP response.

[1249] Output: The terminal receives the summary.

[1250] Step 8:

[1251] The terminal displays the summary to the user.

[1252] Input: Abstract (image or video) sent from the server

[1253] Operation: The device displays the received summary in an appropriate viewer, ensuring that the information is visually easy to understand based on the UE / UX design.

[1254] Output: A visual summary that is displayed to the user.

[1255] In this way, the invention visually summarizes complex textual data, allowing users to efficiently grasp the information.

[1256] (Application example 1)

[1257] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1258] The challenge is to efficiently summarize large amounts of text content and present it in a visually easy-to-understand format, allowing users to understand the information without stress. There is a particular need to provide a means to easily grasp the content, even when it is large in volume or highly important.

[1259] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1260] In this invention, the server includes means for receiving text data, means for analyzing the text data to extract important keywords and phrases, means for generating or searching for related visual elements based on the important keywords and phrases, means for interactively displaying the generated images or videos, and means for sharing the generated images or videos with other users or on a social network, thereby enabling efficient summarization of vast amounts of text content and presenting it in a visually easy-to-understand format.

[1261] "Text data" refers to information or content provided in text format, including meeting materials, reports, novels, etc.

[1262] "Analysis" refers to the means and methods for understanding the content of text data and extracting important elements and patterns.

[1263] "Keywords" refer to words or phrases that have particularly important meanings in a text, and are essential elements for summarizing the entire content.

[1264] A "phrase" is a group of words consisting of a series of multiple words, and is an expression that conveys a specific meaning or information.

[1265] A "visual element" is a visually recognizable object, such as an image or video, that is used to visually summarize text.

[1266] "Generation" refers to the process of creating new images or videos, which is done using artificial intelligence models.

[1267] "Searching" is the process of finding relevant data from existing databases and sources.

[1268] "Interactive display" refers to a display method that allows users to manipulate or take action on the displayed information.

[1269] "Sharing" refers to distributing the generated visuals and information to other users or through social networks.

[1270] "Tokenization" is the process of breaking down text data into individual words and phrases, and is a fundamental step in natural language processing.

[1271] A "generative AI model" is an artificial intelligence model that generates new data using technologies such as deep learning, and includes generative adversarial networks (GANs).

[1272] This invention describes a system for efficiently summarizing large amounts of textual content and presenting it in a visually accessible format. This system is particularly applicable to content distribution services and is designed to enable users to quickly understand long-form content such as news articles or novels.

[1273] System Overview

[1274] The system consists of the following major hardware and software components:

[1275] Hardware: Smartphone (iOS or Android), server

[1276] Software: Python, spacy, Summarizer, DeepAI, Pillow

[1277] Program processing flow

[1278] 1. Receiving text data:

[1279] The user enters the URL or text data selected within the content distribution service into the device, which then sends it to the server.

[1280] 2. Text analysis:

[1281] The server analyzes the received text data using natural language processing (NLP) techniques, specifically tokenizing the text using the spacy library and extracting important keywords and phrases.

[1282] 3. Visual element generation or search:

[1283] Based on the extracted keywords and phrases, the server uses generative AI models (e.g., Generative Adversarial Networks, or GANs) to generate new images and videos, and, if necessary, uses DeepAI's API to search for related images from an existing database.

[1284] 4. Interactive display:

[1285] The server generates visual elements and sends them to the user's device, which displays them interactively. Users can view a visual summary and tap on areas of interest to see more detailed information.

[1286] 5. Sharing features:

[1287] The generated visual summary can be shared by the user with other users or on social networks, and the device provides an interface for sharing this information with other users.

[1288] Specific examples

[1289] Example 1: News article summary

[1290] A user enters a news article URL into the app.

[1291] The app sends the URL to a server, which analyzes the article content and generates a summary.

[1292] The generated summary and associated images are sent to the terminal, where the user can view them interactively.

[1293] If desired, the summary can also be shared on social media.

[1294] Example 2: Technical document summary

[1295] The user uploads the text data of the technical document to the app.

[1296] The app sends the text data to a server, which analyzes the document and generates a summary.

[1297] A visual summary containing key points is generated that users can easily check on their smartphone.

[1298] Prompt Sentence Examples

[1299] "Create a summary and visual summary of the news article https: / / example.com / article."

[1300] In this way, the present invention provides a concrete example for efficiently summarizing a huge amount of text content and enabling users to understand it without stress.

[1301] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1302] Step 1:

[1303] The user selects text content of interest within the content distribution service (for example, a news article URL or long text data) and enters it into the device. The device then sends it to the server. The input in this step is the URL or text data, and the output is data sent to the server.

[1304] Step 2:

[1305] The server analyzes the received text data. Specifically, it uses the Python natural language processing library spacy to tokenize the data and extract important keywords and phrases. The input in this step is the received text data, and the output is the extracted keywords and phrases.

[1306] Step 3:

[1307] The server generates new images and videos using generative AI models such as Generative Adversarial Networks (GANs) based on the extracted keywords and phrases. It also searches for related images from a database using DeepAI's API as needed. The input in this step is the extracted keywords and phrases, and the output is the generated or searched visual elements (images and videos).

[1308] Step 4:

[1309] The server integrates the generated visual elements to create an interactive visual summary, and then constructs an interface that allows users to easily manipulate and view the visual summary. The input in this step is the generated visual elements, and the output is an interactive visual summary.

[1310] Step 5:

[1311] The server sends the interactive visual summary to the user's device for the user to review. If necessary, the user can tap on the part of the visual summary that they want to know more about to view the details. The input in this step is the interactive visual summary, and the output is the visual summary displayed on the user's device.

[1312] Step 6:

[1313] The device provides an interface for sharing the generated visual summary with other users or on social networks. Users can easily share their visual summaries with others. The inputs in this step are the visual summary and the sharer information, and the output is the shared visual summary.

[1314] These are the specific processing steps of the system program that realizes this application example, which enables users to efficiently understand large amounts of text content and share visual summaries.

[1315] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1316] The present invention relates to a system that recognizes a user's emotions and visually summarizes text, thereby helping users understand long conference materials, reports, novels, etc. without stress.

[1317] System Overview

[1318] The system has the following main functions:

[1319] 1. Receiving text data

[1320] 2. Text Analysis

[1321] 3. Extracting important keywords and phrases

[1322] 4. Visual element generation or search

[1323] 5. Emotion Recognition by Emotion Engine

[1324] 6. Visual Integration

[1325] 7. Output of generated images or videos

[1326] Specific processing flow

[1327] Server-side processing

[1328] Receiving input data

[1329] A user uploads text data such as meeting materials, reports, novels, etc. via a terminal.

[1330] The terminal transmits the document data selected by the user to the server.

[1331] The server receives the uploaded document data.

[1332] Text preprocessing

[1333] The server analyzes the received text data using natural language processing (NLP) technology.

[1334] The server divides the text into sentences and paragraphs and tokenizes each one.

[1335] The server tags each token with a part of speech and extracts key keywords and phrases.

[1336] Analysis for image and video generation

[1337] The server searches a database for relevant visual elements based on the extracted keywords and phrases.

[1338] The server may also generate new images and videos using generative AI models (e.g., Generative Adversarial Networks, GANs).

[1339] The server applies algorithms to optimize the placement and order of the generated visual elements to match the content of the original text.

[1340] emotion recognition

[1341] The device analyzes the user's facial expressions and reactions as they view visual elements, and uses an emotion engine to recognize the user's emotions.

[1342] The server adjusts the content and style of the generated visual elements based on the recognized user emotion.

[1343] The server changes the format of the summary results to be presented depending on the user's emotions.

[1344] Content Integration

[1345] The server integrates the generated visual elements in the appropriate order to create a summary while preserving the story and logical structure of the entire text.

[1346] The server reviews the generated summary and makes any necessary corrections or optimizations.

[1347] Output of conversion results

[1348] The server transmits the generated image or video summary to the user's terminal.

[1349] The terminal displays the received summary to the user, allowing the user to understand the information without stress.

[1350] Specific examples

[1351] Example 1: Novel summary

[1352] The user selects the novel text file from the device and presses the upload button.

[1353] The terminal transmits the text file of the selected novel to the server.

[1354] The server analyzes the received novel data and extracts key scenes and characters.

[1355] The server generates related images and videos based on the extracted information.

[1356] The server integrates images and videos as a summary according to the story of the novel.

[1357] The server adjusts the content and style based on the perceived user sentiment.

[1358] The server transmits the generated summary to the terminal.

[1359] The terminal displays the received visual summary to the user.

[1360] Users view a visual summary to get an overview of the story.

[1361] Example 2: Summary of business meeting materials

[1362] The user selects a PDF file of business meeting materials from their device and presses the upload button.

[1363] The terminal transmits the selected business conference materials to the server.

[1364] The server analyzes the content of the document and extracts key points and data charts.

[1365] The server generates relevant images and charts based on the extracted information.

[1366] The server integrates the generated images and charts to create a summary of the conference materials.

[1367] The server adjusts the content and style based on the perceived user sentiment.

[1368] The server transmits the generated summary to the terminal.

[1369] The terminal displays the received visual summary to the user.

[1370] Users view a visual summary to grasp the main points of the meeting.

[1371] This system not only visually summarizes long or complex texts, but also adjusts the way information is presented to take the user's emotions into account, thereby deepening their understanding and reducing stress.

[1372] The processing flow will be explained below.

[1373] Step 1:

[1374] The user selects and uploads long text data such as conference materials, reports, novels, etc. from the terminal.

[1375] Step 2:

[1376] The terminal transmits the selected text data to the server.

[1377] Step 3:

[1378] The server receives the uploaded document data.

[1379] Step 4:

[1380] The server analyzes the received text data using natural language processing (NLP) technology.

[1381] Step 5:

[1382] The server breaks the text into sentences and paragraphs and tokenizes each, breaking the text into smaller, analyzable units.

[1383] Step 6:

[1384] The server tags each token with a part of speech tag and extracts key keywords and phrases.

[1385] Step 7:

[1386] The server searches a database for related visual elements (images and videos) based on the extracted keywords and phrases.

[1387] Step 8:

[1388] The server may also generate new images and videos using generative AI models (e.g., Generative Adversarial Networks, GANs).

[1389] Step 9:

[1390] The server applies algorithms to optimize the placement and order of the generated visual elements to match the content of the original text.

[1391] Step 10:

[1392] The device uses cameras and sensors to capture the user's facial expressions and reactions as they view visual elements.

[1393] Step 11:

[1394] The terminal sends the captured data to an emotion engine to analyze the user's emotions.

[1395] Step 12:

[1396] The server receives the analyzed user emotion data and adjusts the content and style of the generated visual elements.

[1397] Step 13:

[1398] The server changes the format of the summary results to be presented depending on the user's emotions.

[1399] Step 14:

[1400] The server integrates the generated visual elements in the appropriate order to create a summary while preserving the story and logical structure of the entire text.

[1401] Step 15:

[1402] The server reviews the generated summary and makes any necessary corrections or optimizations.

[1403] Step 16:

[1404] The server transmits the generated image or video summary to the user's terminal.

[1405] Step 17:

[1406] The terminal receives the visual summary result sent from the server.

[1407] Step 18:

[1408] The terminal displays the received images and videos to the user, allowing the user to understand the information without stress.

[1409] Step 19:

[1410] Users can view the visual summary and input the necessary information efficiently.

[1411] Example 2

[1412] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1413] In today's world, there is a huge amount of written information flying around, and people need to be able to quickly and efficiently retrieve the information they need. However, when users read long documents such as meeting materials, reports, or novels, it takes time and effort to understand and summarize all of the content. Furthermore, if information is presented without taking into account the emotions that individual users experience when comprehending the information, it can lead to a shallow understanding or stress.

[1414] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving text data, means for analyzing the text data and extracting important keywords and phrases, means for generating or searching for related visual elements based on the important keywords and phrases, means for analyzing a user's emotions and adjusting the content and style of the related visual elements, means for integrating the related visual elements and generating an image or video that summarizes the text content, and means for outputting the generated image or video. This enables a user to efficiently understand long sentences or complex information in a visually summarized form, and further, information presented in accordance with the user's emotions deepens understanding and reduces stress.

[1415] The "means for receiving text data" is a function by which the server receives text data uploaded by the user through the terminal.

[1416] The "means for extracting important keywords and phrases" is a function that analyzes received text data and finds the most important keywords and phrases within it based on the internal structure and context.

[1417] "Means for generating or searching for relevant visual elements" refers to a function that searches for appropriate visual elements from a database based on extracted keywords or phrases, or generates new images or videos using a generative AI model.

[1418] "Means for analyzing the user's emotions and adjusting the content and style of the related visual elements" refers to a function that analyzes the user's facial expressions and reactions through the terminal and changes the content and style of the visual elements based on the results.

[1419] "Means for integrating the related visual elements and generating an image or video that summarizes the content of the text" refers to a function that combines the generated visual elements in an appropriate order to create a single image or video that clearly summarizes the content of the original text.

[1420] The "means for outputting the generated image or video" is a function for transmitting the image or video that is the integrated summary result to the user's terminal and displaying it.

[1421] This invention is a visual summarization support system that helps users understand long text data. This system combines natural language processing technology, generative AI models, and emotion recognition technology to analyze text data and generate visual summaries.

[1422] First, a user uploads text data such as meeting materials, reports, or novels using a terminal. The uploaded text data is then sent to a server via the terminal. The server temporarily stores the received text data and uses it for subsequent analysis.

[1423] The server analyzes the text data using natural language processing (NLP) techniques. Examples of NLP tools used include SpaCy and NLTK. The server divides the text data into sentences and paragraphs and tokenizes each. This process may involve using a tokenization algorithm such as the BERT model. The server then tags each token with a part of speech and extracts important keywords and phrases. This process uses techniques such as TF-IDF and Word2Vec.

[1424] The server then searches a database for relevant visual elements based on the extracted keywords or phrases, and generates new images or videos using generative AI models (e.g., Generative Adversarial Networks, GANs) if necessary. The resulting visual elements are then optimized in their placement and order to match the content of the original text.

[1425] Furthermore, the device analyzes the user's facial expressions and reactions when viewing visual elements through the camera, and recognizes the user's emotions using an emotion engine (e.g., OpenCV or Facial Emotion Recognition API).The server adjusts the content and style of the generated visual elements based on the recognized user emotions, and presents optimal information according to the user's emotions.

[1426] The server then integrates the generated visual elements in the correct order to create a summary that preserves the overall story and logical structure of the text. The server then reviews the integrated summary and makes any necessary corrections or optimizations. The generated image or video summary is then sent to the user's device, where it is displayed, allowing the user to easily understand the information.

[1427] Below, examples of prompt sentences are shown as concrete examples of implementing this system.

[1428] Example prompt sentence:

[1429] "The system receives specified text data, analyzes it using natural language processing technology, and extracts important keywords and phrases. It then generates or searches for related visual elements based on the extracted keywords and phrases. It analyzes the user's sentiment and adjusts the content and style of the generated visual elements. Finally, it integrates the related visual elements to generate an image or video that summarizes the text content and outputs it to the user's device."

[1430] The present invention enables a user to efficiently understand long sentences or complex text data through visual summaries, and furthermore, by presenting information according to the user's emotions, the user's understanding is deepened and stress is reduced.

[1431] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1432] Step 1:

[1433] Receiving input data

[1434] A user uploads text data such as meeting materials, reports, novels, etc. via a terminal.

[1435] Input: A text file selected by the user (PDF, TXT, DOCX, etc.)

[1436] Output: A text file temporarily saved on the device

[1437] The terminal sends the selected text data to the server, typically using an HTTP POST request.

[1438] Input: A text file uploaded by the user

[1439] Output: Text file sent to the server

[1440] The server receives the uploaded text data and temporarily stores it.

[1441] Input: Text file sent from the terminal

[1442] Output: Temporarily saved text file

[1443] Step 2:

[1444] Text preprocessing

[1445] The server analyzes the received text data using natural language processing (NLP) techniques, such as SpaCy and NLTK.

[1446] Input: Saved text data

[1447] Output: Tokenized data split into sentences and paragraphs

[1448] The server splits the text into sentences and paragraphs and tokenizes each of them, using a model such as BERT.

[1449] Input: Data for the entire sentence

[1450] Output: An array of tokens

[1451] The server tags each token with a part of speech and extracts key keywords and phrases, using TF-IDF and Word2Vec.

[1452] Input: Tokenized data

[1453] Output: Part-of-speech tagged tokens and key keywords and phrases

[1454] Step 3:

[1455] Extracting important keywords and phrases

[1456] The server compiles the extracted keywords and phrases and ranks them by importance.

[1457] Input: Part-of-speech tagged tokens and extracted keywords and phrases

[1458] Output: A list of keywords or phrases ranked by importance

[1459] Step 4:

[1460] Analysis for image and video generation

[1461] The server searches a database for relevant visual elements based on the extracted keywords or phrases, using an image search API (e.g., a general image search API).

[1462] Input: Ranking keyword or phrase

[1463] Output: A list of related visual elements

[1464] The server may also generate new images and videos using generative AI models (e.g., Generative Adversarial Networks, GANs).

[1465] Input: Keyword or phrase

[1466] Output: The new image or video generated

[1467] The server applies algorithms to optimize the placement and order of the generated visual elements to match the content of the original text.

[1468] Input: A list of visual elements

[1469] Output: properly positioned and ordered visual elements

[1470] Step 5:

[1471] emotion recognition

[1472] The device analyzes the user's facial expressions and reactions when viewing visual elements through the camera and recognizes the user's emotions using an emotion engine (e.g., a general emotion recognition API).

[1473] Input: Camera video data

[1474] Output: Recognized user emotion data

[1475] The server adjusts the content and style of the generated visual elements based on the recognized user emotion.

[1476] Input: User emotion data and visual elements

[1477] Output: Adjusted visual elements

[1478] The server changes the format of the summary results to be presented depending on the user's emotions.

[1479] Input: User emotion data and summary results

[1480] Output: Summarization results according to emotions

[1481] Step 6:

[1482] Content Integration

[1483] The server integrates the generated visual elements in the appropriate order to create a summary while preserving the story and logical structure of the entire text.

[1484] Input: Optimized visual elements

[1485] Output: Integrated summary image or video

[1486] The server reviews the generated summary and makes any necessary corrections or optimizations.

[1487] Input: Synthesized summary image or video

[1488] Output: revised and optimized summary

[1489] Step 7:

[1490] Output of conversion results

[1491] The server sends the generated image or video summary to the user's device, typically using an HTTP POST request.

[1492] Input: Corrected and optimized summary

[1493] Output: Summary sent to the user's terminal

[1494] The terminal displays the received summary to the user, allowing the user to understand the information without stress.

[1495] Input: Submitted Abstract

[1496] Output: The summary displayed to the user

[1497] This allows users to efficiently understand long or complex textual data through visual summaries.In addition, the way information is presented is adapted according to the user's emotions, deepening understanding and enabling information acquisition with less stress.

[1498] (Application example 2)

[1499] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1500] In today's world, people are bombarded with a vast amount of information, and in the field of advertising in particular, highly expressive content and content that takes into account the recipient's emotions are required to attract attention. However, conventional ad generation methods have difficulty dynamically changing content in response to user emotions, resulting in the inability to provide effective ads. To solve this problem, technology is needed that can recognize user emotions in real time and generate ads by combining appropriate visual elements and text content.

[1501] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1502] In this invention, the server includes means for receiving text data, means for analyzing the text data and extracting important keywords and phrases, means for generating or searching for related visual elements based on the important keywords and phrases, means for recognizing a user's emotions, means for adjusting the content and style of the generated visual elements based on the user's emotions, and means for outputting the generated images or videos, thereby enabling effective advertisement generation according to the user's emotions in real time.

[1503] "Text data" refers to information in text format provided by users, including content such as advertising copy and product descriptions.

[1504] "Analysis" is the process of breaking down text data into sentences and paragraphs and extracting important keywords and phrases.

[1505] "Keywords and phrases" refer to words and expressions that are particularly important in text data and express the gist of the information.

[1506] "Visual elements" refer to elements such as images and videos used to visually convey information to users.

[1507] "Integration" refers to the process of combining visual elements generated or retrieved based on extracted keywords or phrases and displaying them in a cohesive form.

[1508] A "generative AI model" refers to a model that uses artificial intelligence technology to generate new images and videos.

[1509] "Emotion recognition" is the process of detecting and classifying emotions from a user's facial expressions and reactions.

[1510] "Adjustment" is the procedure of dynamically changing the content and style of visual elements based on the user's emotions.

[1511] "Output" refers to the processing of the generated visual elements and integrated advertising content for display to the user.

[1512] The present invention is applied to a system that recognizes a user's emotions and visually summarizes long texts. The system for implementing the present invention is realized by linking a server and a user's terminal.

[1513] System configuration

[1514] The system includes the following main components:

[1515] 1. Server:

[1516] Includes software that implements natural language processing (NLP) techniques to analyze text and extract important keywords and phrases.

[1517] Includes software that implements generative AI models (e.g., Generative Adversarial Networks, GAN) to generate or search for relevant visual elements.

[1518] It includes an emotion engine that recognizes user emotions and adjusts the content and style of visual elements.

[1519] It includes software for generating and outputting a summary result format according to the user's emotions.

[1520] 2. On the user's device:

[1521] Includes software that provides an interface for users to input textual data. Example: smartphone applications.

[1522] It includes a camera for detecting the user's facial expressions and reactions in real time, and software for analyzing facial expressions, e.g., OpenCV library.

[1523] Includes software for displaying the visual summary received from the server to the user. Example: smartphone display.

[1524] Processing flow

[1525] 1. Receiving input data:

[1526] A user inputs text data such as advertising copy and product descriptions using a smartphone application.

[1527] 2. Text Analysis:

[1528] The server analyzes the received text data and extracts important keywords and phrases using natural language processing techniques (e.g., the TextBlob library).

[1529] 3. Visual element generation:

[1530] Based on the extracted keywords and phrases, relevant visual elements (images and videos) are generated or searched for, using a generative AI model (e.g., GAN).

[1531] 4. Emotion recognition:

[1532] While the user is viewing the advertisement on the device, the camera detects facial expressions in real time and analyzes the user's emotions using an emotion engine, which uses the OpenCV library.

[1533] 5. Content Adjustment:

[1534] Dynamically adjust the content and style of visual elements based on recognized user emotions.

[1535] 6. Output:

[1536] The final visual summary is sent to the user's terminal and displayed.

[1537] Specific examples

[1538] For example, suppose a user enters the following ad copy for a newly released cosmetic product: "New cosmetics! The secret to making your skin glow is here." The server analyzes this text and extracts key keywords and phrases (e.g., "new release," "cosmetics," "the secret to glowing"). It then uses a generative AI model to generate visual elements that match these keywords. When a user views an ad, the device's camera detects the user's facial expression of interest and selects visual elements and short video clips that best match that emotion to display the ad. This series of processes enables effective ad generation.

[1539] Prompt Sentence Examples

[1540] "If the user has an interested expression, the system will generate the most resonant visuals and short video clips to pair with this ad copy to create an ad."

[1541] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1542] Step 1:

[1543] A user uses a smartphone application to input text data such as advertising copy or product descriptions. As input, the user enters text into a text box and presses the "Send" button. As output, the device sends this text data to a server.

[1544] Step 2:

[1545] The server analyzes the received text data. As input, the server receives text data. To process the data, it uses natural language processing techniques (e.g., the TextBlob library) to split the data into sentences and paragraphs, tokenize them, and extract important keywords and phrases. As output, it generates a list of these keywords and phrases.

[1546] Step 3:

[1547] The server generates or searches for relevant visual elements based on the extracted keywords or phrases. As input, it receives a list of keywords and uses a generative AI model (e.g., GAN) to generate or search for appropriate images or videos from a database. As output, it obtains a list of generated or searched visual elements.

[1548] Step 4:

[1549] The device detects the user's facial expressions and reactions in real time and analyzes the user's emotions using an emotion engine. As input, the device's camera captures real-time video and applies facial recognition and emotion recognition algorithms (e.g., OpenCV). As data processing, the user's facial expression data is analyzed to identify the type of emotion (e.g., joy, surprise, interest). As output, emotion data is obtained as the analysis result.

[1550] Step 5:

[1551] The server adjusts the content and style of the generated visual elements based on the recognized user emotion. As input, the server receives emotion data and a list of visual elements. As data processing, it selects the visual elements that best fit the emotion data and dynamically adjusts their content and style. As output, it obtains the adjusted visual elements.

[1552] Step 6:

[1553] The server sends the final generated visual summary to the user's device, which displays it to the user. As input, it receives the adjusted visual elements and integrates them with the advertising copy. As data processing, it generates the final advertising content. As output, it displays the advertisement on the device display and allows the user to view the visual summary.

[1554] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1555] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1556] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1557] [Fourth embodiment]

[1558] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1559] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1560] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1561] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1562] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1563] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1564] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1565] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1566] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1567] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1568] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1569] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1570] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1571] The present invention relates to a system for visually summarizing text, and supports users in stress-free comprehension of lengthy text data such as conference materials, reports, and novels.

[1572] System Overview

[1573] The system has the following main functions:

[1574] 1. Receiving text data

[1575] 2. Text Analysis

[1576] 3. Extracting important keywords and phrases

[1577] 4. Visual element generation or search

[1578] 5. Visual Integration

[1579] 6. Output of generated images or videos

[1580] Specific processing flow

[1581] Server-side processing

[1582] Receiving input data

[1583] A user uploads text data such as meeting materials, reports, novels, etc. via a terminal.

[1584] The terminal transmits the document data selected by the user to the server.

[1585] The server receives the uploaded document data.

[1586] Text preprocessing

[1587] The server analyzes the received text data using natural language processing (NLP) technology.

[1588] The server divides the text into sentences and paragraphs and tokenizes each one.

[1589] The server tags each token with a part of speech and extracts key keywords and phrases.

[1590] Analysis for image and video generation

[1591] The server searches a database for relevant visual elements based on the extracted keywords and phrases.

[1592] The server generates new images and videos using generative AI models (e.g., Generative Adversarial Networks, GAN).

[1593] The server applies algorithms to optimize the placement and order of the generated visual elements to match the content of the original text.

[1594] Content Integration

[1595] The server integrates the generated visual elements in the appropriate order to create a summary in a format that allows the user to easily understand the information visually while preserving the story and logical structure of the entire text.

[1596] The server reviews the generated summary and makes any necessary corrections or optimizations.

[1597] Output of conversion results

[1598] The server transmits the generated image or video summary to the user's terminal.

[1599] The terminal displays the received summary to the user, allowing the user to understand the information without stress.

[1600] Specific examples

[1601] Example 1: Novel summary

[1602] The user selects the novel text file from the device and presses the upload button.

[1603] The terminal transmits the text file of the selected novel to the server.

[1604] The server analyzes the received novel data and extracts key scenes and characters.

[1605] The server generates related images and videos based on the extracted information.

[1606] The server integrates images and videos to create a summary based on the story of the novel.

[1607] The server transmits the generated summary to the terminal.

[1608] The terminal displays the received visual summary to the user.

[1609] Users view a visual summary to get an overview of the story.

[1610] Example 2: Summary of business meeting materials

[1611] The user selects a PDF file of business meeting materials from their device and presses the upload button.

[1612] The terminal transmits the selected business conference materials to the server.

[1613] The server analyzes the content of the document and extracts key points and data charts.

[1614] The server generates relevant images and charts based on the extracted information.

[1615] The server integrates the generated images and charts to create a summary of the conference materials.

[1616] The server sends the summary to the terminal.

[1617] The terminal displays the received visual summary to the user.

[1618] Users view a visual summary to grasp the main points of the meeting.

[1619] This system visually summarizes long or complex texts, helping users understand information efficiently and without stress.

[1620] The processing flow will be explained below.

[1621] Step 1:

[1622] The user selects and uploads long text data such as conference materials, reports, novels, etc. from the terminal.

[1623] Step 2:

[1624] The terminal transmits the selected text data to the server.

[1625] Step 3:

[1626] The server receives the uploaded document data.

[1627] Step 4:

[1628] The server analyzes the received text data using natural language processing (NLP) technology.

[1629] Step 5:

[1630] The server breaks the text into sentences and paragraphs and tokenizes each, breaking the text into smaller, analyzable units.

[1631] Step 6:

[1632] The server tags each token with a part of speech tag and extracts key keywords and phrases.

[1633] Step 7:

[1634] The server searches a database for related visual elements (images and videos) based on the extracted keywords and phrases.

[1635] Step 8:

[1636] The server may also generate new images and videos using generative AI models (e.g., Generative Adversarial Networks, GANs).

[1637] Step 9:

[1638] The server applies algorithms to optimize the placement and order of the generated visual elements to match the content of the original text.

[1639] Step 10:

[1640] The server integrates the generated visual elements in the appropriate order to create a summary while preserving the story and logical structure of the entire text.

[1641] Step 11:

[1642] The server reviews the generated summary and makes any necessary corrections or optimizations.

[1643] Step 12:

[1644] The server transmits the generated image or video summary to the user's terminal.

[1645] Step 13:

[1646] The terminal receives the visual summary result sent from the server.

[1647] Step 14:

[1648] The terminal displays the received images and videos to the user, allowing the user to understand the information without stress.

[1649] Step 15:

[1650] Users can view the visual summary and input the necessary information efficiently.

[1651] Example 1

[1652] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1653] There is a problem that it is difficult for users to understand long text data such as meeting materials, reports, and novels efficiently and without stress. In particular, extracting important parts and visually summarizing them requires a great deal of time and effort using conventional methods. Furthermore, if the generation and integration of visual elements is not performed properly, it becomes even more difficult to understand the information.

[1654] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1655] In this invention, the server includes means for receiving text data, means for analyzing the text data to extract important keywords and phrases, means for generating or searching for related visual elements based on the important keywords and phrases, means for integrating the related visual elements to generate an image or video that summarizes the text content, and means for outputting the generated image or video, thereby enabling a user to efficiently understand long text data in a visually summarized format.

[1656] The definition is given below.

[1657] "Text data" refers to data in which information such as meeting materials, reports, and novels is written in text format.

[1658] "Means for receiving" refers to the function by which the server receives text data sent by the user.

[1659] "Means of analysis" refers to the function by which the server uses natural language processing technology to divide and tokenize text data and understand its content.

[1660] "Means for extracting important keywords and phrases" refers to the function of selecting key words and short sentences from analyzed text data.

[1661] "Means for generating or searching for relevant visual elements" refers to the ability to create or search a database for images or videos based on extracted keywords or phrases.

[1662] "Means for integrating visual elements" refers to the function of combining the generated images and videos in an appropriate order and arrangement to visually express the content of the entire text.

[1663] "Means for generating a summarized image or video" refers to a function that produces a summary of a text from integrated visual elements.

[1664] "Means for outputting" refers to a function for transmitting the generated summary result to the user's terminal and displaying it.

[1665] "Natural language processing technology" refers to technology that enables computers to understand, interpret, and manipulate human language.

[1666] A "generative AI model" refers to an algorithm or learning model that uses artificial intelligence to generate new data (e.g., images or videos).

[1667] "Algorithms that optimize placement and order" refer to methods for determining the optimal placement and display order of visual elements to match the content of the original text.

[1668] This invention relates to a visual summarization system that allows users to understand long text data efficiently and without stress. This system analyzes the text data, extracts important keywords and phrases, generates or searches for related visual elements based on them, and integrates these visual elements to generate an image or video that summarizes the text content and provides it to the user.

[1669] System Overview

[1670] The system has the following main functions:

[1671] 1. Receiving text data

[1672] 2. Text Analysis

[1673] 3. Extracting important keywords and phrases

[1674] 4. Visual element generation or search

[1675] 5. Visual Integration

[1676] 6. Output of generated images or videos

[1677] Specifically, this is realized using the following hardware and software.

[1678] Hardware and software used

[1679] server

[1680] Common NLP libraries "SpaCy" or "NLTK" for using natural language processing (NLP) techniques

[1681] "Elasticsearch" as a search engine

[1682] "TensorFlow" or "PyTorch" frameworks for using generative AI models

[1683] OpenCV is an image processing library for optimizing the placement and order of visual elements.

[1684] Terminal

[1685] A computing device having an interface for a user to upload text data and display generated summaries.

[1686] Program processing

[1687] Specific data processing and data calculations are performed between the server, terminal, and user as follows:

[1688] 1. Receiving text data

[1689] Users upload the necessary text data (conference materials, reports, novels, etc.) from their terminals.

[1690] The terminal sends the text data selected by the user to the server using an HTTP POST request.

[1691] The server stores the uploaded text data in a specified directory.

[1692] 2. Text Analysis

[1693] The server analyzes the received text data using natural language processing (NLP) technology. Specifically, it divides the text into sentences and paragraphs, tokenizes each, and tags them by part of speech.

[1694] 3. Extracting important keywords and phrases

[1695] The server extracts key keywords and phrases based on part-of-speech tagged token information, and stores this information in a list format.

[1696] 4. Visual element generation or search

[1697] The server searches the database for relevant visual elements based on the extracted keywords or phrases. If no results are found, it generates new images or videos using a generative AI model (e.g., Generative Adversarial Networks, GAN). The generated visual elements are saved in a temporary directory.

[1698] 5. Visual Integration

[1699] The server then integrates the generated visual elements in the correct order to create a summary that preserves the overall story and logical structure of the text, allowing users to easily understand the information visually. The server then arranges the images according to the storyline, resizes them to the optimal size, and sets the appropriate placement coordinates. The results are then exported as a PDF or video.

[1700] 6. Output of generated images or videos

[1701] The server transmits the generated image or video summary to the user's terminal.

[1702] The terminal receives the response and displays it in the appropriate viewer.

[1703] Specific examples

[1704] Below, we will show the process of summarizing a novel and business conference materials as concrete examples of this system.

[1705] Example 1: Novel summary

[1706] 1. The user selects the novel text file from their device and presses the upload button.

[1707] 2. The terminal sends the text file of the selected novel to the server.

[1708] 3. The server analyzes the received novel data and extracts key scenes and characters.

[1709] 4. The server generates relevant images and videos based on the extracted information.

[1710] 5. The server integrates images and videos to create a summary based on the story of the novel.

[1711] 6. The server sends the generated summary to the terminal.

[1712] 7. The terminal displays the received visual summary to the user.

[1713] Here are some example prompts to input to the generative AI model:

[1714] "Generate a visual summary of the following novel: [Insert novel text here]. Focus on the main scenes and characters."

[1715] Example 2: Summary of business meeting materials

[1716] 1. The user selects the PDF file of the business meeting materials from the device and presses the upload button.

[1717] 2. The terminal transmits the selected business conference materials to the server.

[1718] 3. The server analyzes the content of the document and extracts important points and data charts.

[1719] 4. The server generates relevant images and charts based on the extracted information.

[1720] 5. The server integrates the generated images and charts to create a summary of the meeting materials.

[1721] 6. The server sends the summary to the terminal.

[1722] 7. The terminal displays the received visual summary to the user.

[1723] Here are some example prompts to input to the generative AI model:

[1724] "Create a visual summary of the following business meeting document: [Insert meeting document text here]. Highlight key points and data charts."

[1725] As described above, this system visually summarizes long sentences and complex content, helping users understand information efficiently and without stress.

[1726] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1727] Step 1:

[1728] The user uploads text data.

[1729] Input: Written data such as meeting materials, reports, novels, etc.

[1730] Operation: The user clicks the upload button on the device to upload the selected text data to the device.

[1731] Output: The text data is saved to the terminal.

[1732] Step 2:

[1733] The terminal transmits the text data to the server.

[1734] Input: User-uploaded text data

[1735] Operation: The terminal sends the received text data to the server via an HTTP POST request.

[1736] Output: The server receives the text data.

[1737] Step 3:

[1738] The server analyzes the text data.

[1739] Input: Text data sent from the terminal

[1740] How it works: The server uses a natural language processing (NLP) library to split the text data into sentences and paragraphs, and tokenize each. Specifically, it uses the NLP library "SpaCy" or "NLTK" and the nlp(text) function.

[1741] Output: Segmented and tokenized sentence data.

[1742] Step 4:

[1743] The server extracts important keywords and phrases.

[1744] Input: Segmented and tokenized text data

[1745] How it works: The server tags the segmented and tokenized data and extracts important keywords and phrases. Keywords are extracted based on specific parts of speech (nouns, verbs, adjectives).

[1746] Output: A list of key keywords and phrases.

[1747] Step 5:

[1748] The server generates or retrieves the visual elements.

[1749] Input: Extracted keywords or phrases

[1750] How it works: The server searches a database for relevant visual elements based on keywords or phrases. It uses Elasticsearch to execute the query, and if no results are found, it uses a generative AI model (e.g., Generative Adversarial Networks, GAN) to generate new images or videos. It leverages TensorFlow or PyTorch.

[1751] Output: Generated or retrieved visual elements (images or videos).

[1752] Step 6:

[1753] The server aggregates the visual elements.

[1754] Input: A generated or searched visual element

[1755] How it works: The server orders the images according to the storyline, adjusting their placement and size, and uses OpenCV to integrate the visual elements in an optimized order.

[1756] Output: Optimized and consolidated image or video summary.

[1757] Step 7:

[1758] The server sends the summary to the terminal.

[1759] Input: Optimized and integrated summary (image or video)

[1760] Action: The server generates a summary and sends it as an HTTP response.

[1761] Output: The terminal receives the summary.

[1762] Step 8:

[1763] The terminal displays the summary to the user.

[1764] Input: Abstract (image or video) sent from the server

[1765] Operation: The device displays the received summary in an appropriate viewer, ensuring that the information is visually easy to understand based on the UE / UX design.

[1766] Output: A visual summary that is displayed to the user.

[1767] In this way, the invention visually summarizes complex textual data, allowing users to efficiently grasp the information.

[1768] (Application example 1)

[1769] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1770] The challenge is to efficiently summarize large amounts of text content and present it in a visually easy-to-understand format, allowing users to understand the information without stress. There is a particular need to provide a means to easily grasp the content, even when it is large in volume or highly important.

[1771] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1772] In this invention, the server includes means for receiving text data, means for analyzing the text data to extract important keywords and phrases, means for generating or searching for related visual elements based on the important keywords and phrases, means for interactively displaying the generated images or videos, and means for sharing the generated images or videos with other users or on a social network, thereby enabling efficient summarization of vast amounts of text content and presenting it in a visually easy-to-understand format.

[1773] "Text data" refers to information or content provided in text format, including meeting materials, reports, novels, etc.

[1774] "Analysis" refers to the means and methods for understanding the content of text data and extracting important elements and patterns.

[1775] "Keywords" refer to words or phrases that have particularly important meanings in a text, and are essential elements for summarizing the entire content.

[1776] A "phrase" is a group of words consisting of a series of multiple words, and is an expression that conveys a specific meaning or information.

[1777] A "visual element" is a visually recognizable object, such as an image or video, that is used to visually summarize text.

[1778] "Generation" refers to the process of creating new images or videos, which is done using artificial intelligence models.

[1779] "Searching" is the process of finding relevant data from existing databases and sources.

[1780] "Interactive display" refers to a display method that allows users to manipulate or take action on the displayed information.

[1781] "Sharing" refers to distributing the generated visuals and information to other users or through social networks.

[1782] "Tokenization" is the process of breaking down text data into individual words and phrases, and is a fundamental step in natural language processing.

[1783] A "generative AI model" is an artificial intelligence model that generates new data using technologies such as deep learning, and includes generative adversarial networks (GANs).

[1784] This invention describes a system for efficiently summarizing large amounts of textual content and presenting it in a visually accessible format. This system is particularly applicable to content distribution services and is designed to enable users to quickly understand long-form content such as news articles or novels.

[1785] System Overview

[1786] The system consists of the following major hardware and software components:

[1787] Hardware: Smartphone (iOS or Android), server

[1788] Software: Python, spacy, Summarizer, DeepAI, Pillow

[1789] Program processing flow

[1790] 1. Receiving text data:

[1791] The user enters the URL or text data selected within the content distribution service into the device, which then sends it to the server.

[1792] 2. Text analysis:

[1793] The server analyzes the received text data using natural language processing (NLP) techniques, specifically tokenizing the text using the spacy library and extracting important keywords and phrases.

[1794] 3. Visual element generation or search:

[1795] Based on the extracted keywords and phrases, the server uses generative AI models (e.g., Generative Adversarial Networks, or GANs) to generate new images and videos, and, if necessary, uses DeepAI's API to search for related images from an existing database.

[1796] 4. Interactive display:

[1797] The server generates visual elements and sends them to the user's device, which displays them interactively. Users can view a visual summary and tap on areas of interest to see more detailed information.

[1798] 5. Sharing features:

[1799] The generated visual summary can be shared by the user with other users or on social networks, and the device provides an interface for sharing this information with other users.

[1800] Specific examples

[1801] Example 1: News article summary

[1802] A user enters a news article URL into the app.

[1803] The app sends the URL to a server, which analyzes the article content and generates a summary.

[1804] The generated summary and associated images are sent to the terminal, where the user can view them interactively.

[1805] If desired, the summary can also be shared on social media.

[1806] Example 2: Technical document summary

[1807] The user uploads the text data of the technical document to the app.

[1808] The app sends the text data to a server, which analyzes the document and generates a summary.

[1809] A visual summary containing key points is generated that users can easily check on their smartphone.

[1810] Prompt Sentence Examples

[1811] "Create a summary and visual summary of the news article https: / / example.com / article."

[1812] In this way, the present invention provides a concrete example for efficiently summarizing a huge amount of text content and enabling users to understand it without stress.

[1813] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1814] Step 1:

[1815] The user selects text content of interest within the content distribution service (for example, a news article URL or long text data) and enters it into the device. The device then sends it to the server. The input in this step is the URL or text data, and the output is data sent to the server.

[1816] Step 2:

[1817] The server analyzes the received text data. Specifically, it uses the Python natural language processing library spacy to tokenize the data and extract important keywords and phrases. The input in this step is the received text data, and the output is the extracted keywords and phrases.

[1818] Step 3:

[1819] The server generates new images and videos using generative AI models such as Generative Adversarial Networks (GANs) based on the extracted keywords and phrases. It also searches for related images from a database using DeepAI's API as needed. The input in this step is the extracted keywords and phrases, and the output is the generated or searched visual elements (images and videos).

[1820] Step 4:

[1821] The server integrates the generated visual elements to create an interactive visual summary, and then constructs an interface that allows users to easily manipulate and view the visual summary. The input in this step is the generated visual elements, and the output is an interactive visual summary.

[1822] Step 5:

[1823] The server sends the interactive visual summary to the user's device for the user to review. If necessary, the user can tap on the part of the visual summary that they want to know more about to view the details. The input in this step is the interactive visual summary, and the output is the visual summary displayed on the user's device.

[1824] Step 6:

[1825] The device provides an interface for sharing the generated visual summary with other users or on social networks. Users can easily share their visual summaries with others. The inputs in this step are the visual summary and the sharer information, and the output is the shared visual summary.

[1826] These are the specific processing steps of the system program that realizes this application example, which enables users to efficiently understand large amounts of text content and share visual summaries.

[1827] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1828] The present invention relates to a system that recognizes a user's emotions and visually summarizes text, thereby helping users understand long conference materials, reports, novels, etc. without stress.

[1829] System Overview

[1830] The system has the following main functions:

[1831] 1. Receiving text data

[1832] 2. Text Analysis

[1833] 3. Extracting important keywords and phrases

[1834] 4. Visual element generation or search

[1835] 5. Emotion Recognition by Emotion Engine

[1836] 6. Visual Integration

[1837] 7. Output of generated images or videos

[1838] Specific processing flow

[1839] Server-side processing

[1840] Receiving input data

[1841] A user uploads text data such as meeting materials, reports, novels, etc. via a terminal.

[1842] The terminal transmits the document data selected by the user to the server.

[1843] The server receives the uploaded document data.

[1844] Text preprocessing

[1845] The server analyzes the received text data using natural language processing (NLP) technology.

[1846] The server divides the text into sentences and paragraphs and tokenizes each one.

[1847] The server tags each token with a part of speech and extracts key keywords and phrases.

[1848] Analysis for image and video generation

[1849] The server searches a database for relevant visual elements based on the extracted keywords and phrases.

[1850] The server may also generate new images and videos using generative AI models (e.g., Generative Adversarial Networks, GANs).

[1851] The server applies algorithms to optimize the placement and order of the generated visual elements to match the content of the original text.

[1852] emotion recognition

[1853] The device analyzes the user's facial expressions and reactions as they view visual elements, and uses an emotion engine to recognize the user's emotions.

[1854] The server adjusts the content and style of the generated visual elements based on the recognized user emotion.

[1855] The server changes the format of the summary results to be presented depending on the user's emotions.

[1856] Content Integration

[1857] The server integrates the generated visual elements in the appropriate order to create a summary while preserving the story and logical structure of the entire text.

[1858] The server reviews the generated summary and makes any necessary corrections or optimizations.

[1859] Output of conversion results

[1860] The server transmits the generated image or video summary to the user's terminal.

[1861] The terminal displays the received summary to the user, allowing the user to understand the information without stress.

[1862] Specific examples

[1863] Example 1: Novel summary

[1864] The user selects the novel text file from the device and presses the upload button.

[1865] The terminal transmits the text file of the selected novel to the server.

[1866] The server analyzes the received novel data and extracts key scenes and characters.

[1867] The server generates related images and videos based on the extracted information.

[1868] The server integrates images and videos as a summary according to the story of the novel.

[1869] The server adjusts the content and style based on the perceived user sentiment.

[1870] The server transmits the generated summary to the terminal.

[1871] The terminal displays the received visual summary to the user.

[1872] Users view a visual summary to get an overview of the story.

[1873] Example 2: Summary of business meeting materials

[1874] The user selects a PDF file of business meeting materials from their device and presses the upload button.

[1875] The terminal transmits the selected business conference materials to the server.

[1876] The server analyzes the content of the document and extracts key points and data charts.

[1877] The server generates relevant images and charts based on the extracted information.

[1878] The server integrates the generated images and charts to create a summary of the conference materials.

[1879] The server adjusts the content and style based on the perceived user sentiment.

[1880] The server transmits the generated summary to the terminal.

[1881] The terminal displays the received visual summary to the user.

[1882] Users view a visual summary to grasp the main points of the meeting.

[1883] This system not only visually summarizes long or complex texts, but also adjusts the way information is presented to take the user's emotions into account, thereby deepening their understanding and reducing stress.

[1884] The processing flow will be explained below.

[1885] Step 1:

[1886] The user selects and uploads long text data such as conference materials, reports, novels, etc. from the terminal.

[1887] Step 2:

[1888] The terminal transmits the selected text data to the server.

[1889] Step 3:

[1890] The server receives the uploaded document data.

[1891] Step 4:

[1892] The server analyzes the received text data using natural language processing (NLP) technology.

[1893] Step 5:

[1894] The server breaks the text into sentences and paragraphs and tokenizes each, breaking the text into smaller, analyzable units.

[1895] Step 6:

[1896] The server tags each token with a part of speech tag and extracts key keywords and phrases.

[1897] Step 7:

[1898] The server searches a database for related visual elements (images and videos) based on the extracted keywords and phrases.

[1899] Step 8:

[1900] The server may also generate new images and videos using generative AI models (e.g., Generative Adversarial Networks, GANs).

[1901] Step 9:

[1902] The server applies algorithms to optimize the placement and order of the generated visual elements to match the content of the original text.

[1903] Step 10:

[1904] The device uses cameras and sensors to capture the user's facial expressions and reactions as they view visual elements.

[1905] Step 11:

[1906] The terminal sends the captured data to an emotion engine to analyze the user's emotions.

[1907] Step 12:

[1908] The server receives the analyzed user emotion data and adjusts the content and style of the generated visual elements.

[1909] Step 13:

[1910] The server changes the format of the summary results to be presented depending on the user's emotions.

[1911] Step 14:

[1912] The server integrates the generated visual elements in the appropriate order to create a summary while preserving the story and logical structure of the entire text.

[1913] Step 15:

[1914] The server reviews the generated summary and makes any necessary corrections or optimizations.

[1915] Step 16:

[1916] The server transmits the generated image or video summary to the user's terminal.

[1917] Step 17:

[1918] The terminal receives the visual summary result sent from the server.

[1919] Step 18:

[1920] The terminal displays the received images and videos to the user, allowing the user to understand the information without stress.

[1921] Step 19:

[1922] Users can view the visual summary and input the necessary information efficiently.

[1923] Example 2

[1924] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1925] In today's world, there is a huge amount of written information flying around, and people need to be able to quickly and efficiently retrieve the information they need. However, when users read long documents such as meeting materials, reports, or novels, it takes time and effort to understand and summarize all of the content. Furthermore, if information is presented without taking into account the emotions that individual users experience when comprehending the information, it can lead to a shallow understanding or stress.

[1926] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving text data, means for analyzing the text data and extracting important keywords and phrases, means for generating or searching for related visual elements based on the important keywords and phrases, means for analyzing a user's emotions and adjusting the content and style of the related visual elements, means for integrating the related visual elements and generating an image or video that summarizes the text content, and means for outputting the generated image or video. This enables a user to efficiently understand long sentences or complex information in a visually summarized form, and further, information presented in accordance with the user's emotions deepens understanding and reduces stress.

[1927] The "means for receiving text data" is a function by which the server receives text data uploaded by the user through the terminal.

[1928] The "means for extracting important keywords and phrases" is a function that analyzes received text data and finds the most important keywords and phrases within it based on the internal structure and context.

[1929] "Means for generating or searching for relevant visual elements" refers to a function that searches for appropriate visual elements from a database based on extracted keywords or phrases, or generates new images or videos using a generative AI model.

[1930] "Means for analyzing the user's emotions and adjusting the content and style of the related visual elements" refers to a function that analyzes the user's facial expressions and reactions through the terminal and changes the content and style of the visual elements based on the results.

[1931] "Means for integrating the related visual elements and generating an image or video that summarizes the content of the text" refers to a function that combines the generated visual elements in an appropriate order to create a single image or video that clearly summarizes the content of the original text.

[1932] The "means for outputting the generated image or video" is a function for transmitting the image or video that is the integrated summary result to the user's terminal and displaying it.

[1933] This invention is a visual summarization support system that helps users understand long text data. This system combines natural language processing technology, generative AI models, and emotion recognition technology to analyze text data and generate visual summaries.

[1934] First, a user uploads text data such as meeting materials, reports, or novels using a terminal. The uploaded text data is then sent to a server via the terminal. The server temporarily stores the received text data and uses it for subsequent analysis.

[1935] The server analyzes the text data using natural language processing (NLP) techniques. Examples of NLP tools used include SpaCy and NLTK. The server divides the text data into sentences and paragraphs and tokenizes each. This process may involve using a tokenization algorithm such as the BERT model. The server then tags each token with a part of speech and extracts important keywords and phrases. This process uses techniques such as TF-IDF and Word2Vec.

[1936] The server then searches a database for relevant visual elements based on the extracted keywords or phrases, and generates new images or videos using generative AI models (e.g., Generative Adversarial Networks, GANs) if necessary. The resulting visual elements are then optimized in their placement and order to match the content of the original text.

[1937] Furthermore, the device analyzes the user's facial expressions and reactions when viewing visual elements through the camera, and recognizes the user's emotions using an emotion engine (e.g., OpenCV or Facial Emotion Recognition API).The server adjusts the content and style of the generated visual elements based on the recognized user emotions, and presents optimal information according to the user's emotions.

[1938] The server then integrates the generated visual elements in the correct order to create a summary that preserves the overall story and logical structure of the text. The server then reviews the integrated summary and makes any necessary corrections or optimizations. The generated image or video summary is then sent to the user's device, where it is displayed, allowing the user to easily understand the information.

[1939] Below, examples of prompt sentences are shown as concrete examples of implementing this system.

[1940] Example prompt sentence:

[1941] "The system receives specified text data, analyzes it using natural language processing technology, and extracts important keywords and phrases. It then generates or searches for related visual elements based on the extracted keywords and phrases. It analyzes the user's sentiment and adjusts the content and style of the generated visual elements. Finally, it integrates the related visual elements to generate an image or video that summarizes the text content and outputs it to the user's device."

[1942] The present invention enables a user to efficiently understand long sentences or complex text data through visual summaries, and furthermore, by presenting information according to the user's emotions, the user's understanding is deepened and stress is reduced.

[1943] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1944] Step 1:

[1945] Receiving input data

[1946] A user uploads text data such as meeting materials, reports, novels, etc. via a terminal.

[1947] Input: A text file selected by the user (PDF, TXT, DOCX, etc.)

[1948] Output: A text file temporarily saved on the device

[1949] The terminal sends the selected text data to the server, typically using an HTTP POST request.

[1950] Input: A text file uploaded by the user

[1951] Output: Text file sent to the server

[1952] The server receives the uploaded text data and temporarily stores it.

[1953] Input: Text file sent from the terminal

[1954] Output: Temporarily saved text file

[1955] Step 2:

[1956] Text preprocessing

[1957] The server analyzes the received text data using natural language processing (NLP) techniques, such as SpaCy and NLTK.

[1958] Input: Saved text data

[1959] Output: Tokenized data split into sentences and paragraphs

[1960] The server splits the text into sentences and paragraphs and tokenizes each of them, using a model such as BERT.

[1961] Input: Data for the entire sentence

[1962] Output: An array of tokens

[1963] The server tags each token with a part of speech and extracts key keywords and phrases, using TF-IDF and Word2Vec.

[1964] Input: Tokenized data

[1965] Output: Part-of-speech tagged tokens and key keywords and phrases

[1966] Step 3:

[1967] Extracting important keywords and phrases

[1968] The server compiles the extracted keywords and phrases and ranks them by importance.

[1969] Input: Part-of-speech tagged tokens and extracted keywords and phrases

[1970] Output: A list of keywords or phrases ranked by importance

[1971] Step 4:

[1972] Analysis for image and video generation

[1973] The server searches a database for relevant visual elements based on the extracted keywords or phrases, using an image search API (e.g., a general image search API).

[1974] Input: Ranking keyword or phrase

[1975] Output: A list of related visual elements

[1976] The server may also generate new images and videos using generative AI models (e.g., Generative Adversarial Networks, GANs).

[1977] Input: Keyword or phrase

[1978] Output: The new image or video generated

[1979] The server applies algorithms to optimize the placement and order of the generated visual elements to match the content of the original text.

[1980] Input: A list of visual elements

[1981] Output: properly positioned and ordered visual elements

[1982] Step 5:

[1983] emotion recognition

[1984] The device analyzes the user's facial expressions and reactions when viewing visual elements through the camera and recognizes the user's emotions using an emotion engine (e.g., a general emotion recognition API).

[1985] Input: Camera video data

[1986] Output: Recognized user emotion data

[1987] The server adjusts the content and style of the generated visual elements based on the recognized user emotion.

[1988] Input: User emotion data and visual elements

[1989] Output: Adjusted visual elements

[1990] The server changes the format of the summary results to be presented depending on the user's emotions.

[1991] Input: User emotion data and summary results

[1992] Output: Summarization results according to emotions

[1993] Step 6:

[1994] Content Integration

[1995] The server integrates the generated visual elements in the appropriate order to create a summary while preserving the story and logical structure of the entire text.

[1996] Input: Optimized visual elements

[1997] Output: Integrated summary image or video

[1998] The server reviews the generated summary and makes any necessary corrections or optimizations.

[1999] Input: Synthesized summary image or video

[2000] Output: revised and optimized summary

[2001] Step 7:

[2002] Output of conversion results

[2003] The server sends the generated image or video summary to the user's device, typically using an HTTP POST request.

[2004] Input: Corrected and optimized summary

[2005] Output: Summary sent to the user's terminal

[2006] The terminal displays the received summary to the user, allowing the user to understand the information without stress.

[2007] Input: Submitted Abstract

[2008] Output: The summary displayed to the user

[2009] This allows users to efficiently understand long or complex textual data through visual summaries.In addition, the way information is presented is adapted according to the user's emotions, deepening understanding and enabling information acquisition with less stress.

[2010] (Application example 2)

[2011] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2012] In today's world, people are bombarded with a vast amount of information, and in the field of advertising in particular, highly expressive content and content that takes into account the recipient's emotions are required to attract attention. However, conventional ad generation methods have difficulty dynamically changing content in response to user emotions, resulting in the inability to provide effective ads. To solve this problem, technology is needed that can recognize user emotions in real time and generate ads by combining appropriate visual elements and text content.

[2013] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2014] In this invention, the server includes means for receiving text data, means for analyzing the text data and extracting important keywords and phrases, means for generating or searching for related visual elements based on the important keywords and phrases, means for recognizing a user's emotions, means for adjusting the content and style of the generated visual elements based on the user's emotions, and means for outputting the generated images or videos, thereby enabling effective advertisement generation according to the user's emotions in real time.

[2015] "Text data" refers to information in text format provided by users, including content such as advertising copy and product descriptions.

[2016] "Analysis" is the process of breaking down text data into sentences and paragraphs and extracting important keywords and phrases.

[2017] "Keywords and phrases" refer to words and expressions that are particularly important in text data and express the gist of the information.

[2018] "Visual elements" refer to elements such as images and videos used to visually convey information to users.

[2019] "Integration" refers to the process of combining visual elements generated or retrieved based on extracted keywords or phrases and displaying them in a cohesive form.

[2020] A "generative AI model" refers to a model that uses artificial intelligence technology to generate new images and videos.

[2021] "Emotion recognition" is the process of detecting and classifying emotions from a user's facial expressions and reactions.

[2022] "Adjustment" is the procedure of dynamically changing the content and style of visual elements based on the user's emotions.

[2023] "Output" refers to the processing of the generated visual elements and integrated advertising content for display to the user.

[2024] The present invention is applied to a system that recognizes a user's emotions and visually summarizes long texts. The system for implementing the present invention is realized by linking a server and a user's terminal.

[2025] System configuration

[2026] The system includes the following main components:

[2027] 1. Server:

[2028] Includes software that implements natural language processing (NLP) techniques to analyze text and extract important keywords and phrases.

[2029] Includes software that implements generative AI models (e.g., Generative Adversarial Networks, GAN) to generate or search for relevant visual elements.

[2030] It includes an emotion engine that recognizes user emotions and adjusts the content and style of visual elements.

[2031] It includes software for generating and outputting a summary result format according to the user's emotions.

[2032] 2. On the user's device:

[2033] Includes software that provides an interface for users to input textual data. Example: smartphone applications.

[2034] It includes a camera for detecting the user's facial expressions and reactions in real time, and software for analyzing facial expressions, e.g., OpenCV library.

[2035] Includes software for displaying the visual summary received from the server to the user. Example: smartphone display.

[2036] Processing flow

[2037] 1. Receiving input data:

[2038] A user inputs text data such as advertising copy and product descriptions using a smartphone application.

[2039] 2. Text Analysis:

[2040] The server analyzes the received text data and extracts important keywords and phrases using natural language processing techniques (e.g., the TextBlob library).

[2041] 3. Visual element generation:

[2042] Based on the extracted keywords and phrases, relevant visual elements (images and videos) are generated or searched for, using a generative AI model (e.g., GAN).

[2043] 4. Emotion recognition:

[2044] While the user is viewing the advertisement on the device, the camera detects facial expressions in real time and analyzes the user's emotions using an emotion engine, which uses the OpenCV library.

[2045] 5. Content Adjustment:

[2046] Dynamically adjust the content and style of visual elements based on recognized user emotions.

[2047] 6. Output:

[2048] The final visual summary is sent to the user's terminal and displayed.

[2049] Specific examples

[2050] For example, suppose a user enters the following ad copy for a newly released cosmetic product: "New cosmetics! The secret to making your skin glow is here." The server analyzes this text and extracts key keywords and phrases (e.g., "new release," "cosmetics," "the secret to glowing"). It then uses a generative AI model to generate visual elements that match these keywords. When a user views an ad, the device's camera detects the user's facial expression of interest and selects visual elements and short video clips that best match that emotion to display the ad. This series of processes enables effective ad generation.

[2051] Prompt Sentence Examples

[2052] "If the user has an interested expression, the system will generate the most resonant visuals and short video clips to pair with this ad copy to create an ad."

[2053] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2054] Step 1:

[2055] A user uses a smartphone application to input text data such as advertising copy or product descriptions. As input, the user enters text into a text box and presses the "Send" button. As output, the device sends this text data to a server.

[2056] Step 2:

[2057] The server analyzes the received text data. As input, the server receives text data. To process the data, it uses natural language processing techniques (e.g., the TextBlob library) to split the data into sentences and paragraphs, tokenize them, and extract important keywords and phrases. As output, it generates a list of these keywords and phrases.

[2058] Step 3:

[2059] The server generates or searches for relevant visual elements based on the extracted keywords or phrases. As input, it receives a list of keywords and uses a generative AI model (e.g., GAN) to generate or search for appropriate images or videos from a database. As output, it obtains a list of generated or searched visual elements.

[2060] Step 4:

[2061] The device detects the user's facial expressions and reactions in real time and analyzes the user's emotions using an emotion engine. As input, the device's camera captures real-time video and applies facial recognition and emotion recognition algorithms (e.g., OpenCV). As data processing, the user's facial expression data is analyzed to identify the type of emotion (e.g., joy, surprise, interest). As output, emotion data is obtained as the analysis result.

[2062] Step 5:

[2063] The server adjusts the content and style of the generated visual elements based on the recognized user emotion. As input, the server receives emotion data and a list of visual elements. As data processing, it selects the visual elements that best fit the emotion data and dynamically adjusts their content and style. As output, it obtains the adjusted visual elements.

[2064] Step 6:

[2065] The server sends the final generated visual summary to the user's device, which displays it to the user. As input, it receives the adjusted visual elements and integrates them with the advertising copy. As data processing, it generates the final advertising content. As output, it displays the advertisement on the device display and allows the user to view the visual summary.

[2066] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2067] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2068] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2069] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2070] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2071] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2072] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2073] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2074] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2075] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2076] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2077] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2078] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2079] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2080] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2081] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2082] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2083] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2084] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2085] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2086] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2087] The following is further disclosed regarding the above embodiment.

[2088] (Claim 1)

[2089] means for receiving text data;

[2090] means for analyzing the text data and extracting important keywords and phrases;

[2091] means for generating or retrieving relevant visual elements based on said key keywords or phrases;

[2092] means for integrating the related visual elements to generate an image or video summarizing the content of the text;

[2093] means for outputting the generated image or video;

[2094] A system including:

[2095] (Claim 2)

[2096] 2. The system according to claim 1, further comprising means for dividing the text data into sentences or paragraphs and tokenizing each sentence or paragraph.

[2097] (Claim 3)

[2098] 10. The system of claim 1, further comprising means for generating new images or videos using a generative AI model based on the extracted keywords or phrases.

[2099] "Example 1"

[2100] (Claim 1)

[2101] means for receiving text data;

[2102] means for analyzing the text data and extracting important keywords and phrases;

[2103] means for generating or retrieving relevant visual elements based on said key keywords or phrases;

[2104] means for integrating the related visual elements to generate an image or video summarizing the content of the text;

[2105] means for outputting the generated image or video;

[2106] A system including:

[2107] (Claim 2)

[2108] 2. The system according to claim 1, further comprising means for dividing the text data into sentences or paragraphs and tokenizing each sentence or paragraph.

[2109] (Claim 3)

[2110] 10. The system of claim 1, further comprising means for generating new images or videos using a generative AI model based on the extracted keywords or phrases.

[2111] (Claim 4)

[2112] 10. The system of claim 1, further comprising means for analyzing the text data using natural language processing techniques.

[2113] (Claim 5)

[2114] 10. The system of claim 1, further comprising means for applying an algorithm to optimize the placement and ordering of the generated visual elements.

[2115] (Claim 6)

[2116] 10. The system of claim 1, further comprising means for reviewing, correcting, or optimizing the generated summary.

[2117] (Claim 7)

[2118] 10. The system of claim 1, further comprising means for displaying a summary of the image or video received through the user's terminal.

[2119] "Application Example 1"

[2120] (Claim 1)

[2121] means for receiving text data;

[2122] means for analyzing the text data and extracting important keywords and phrases;

[2123] means for generating or retrieving relevant visual elements based on said key keywords or phrases;

[2124] means for integrating the related visual elements to generate an image or video summarizing the content of the text;

[2125] means for interactively displaying the generated image or video;

[2126] means for sharing the generated image or video with other users or on a social network;

[2127] A system including:

[2128] (Claim 2)

[2129] 2. The system according to claim 1, further comprising means for dividing the text data into sentences or paragraphs and tokenizing each sentence or paragraph.

[2130] (Claim 3)

[2131] 10. The system of claim 1, further comprising means for generating new images or videos using a generative AI model based on the extracted keywords or phrases.

[2132] "Example 2: Combining Emotion Engines"

[2133] (Claim 1)

[2134] means for receiving text data;

[2135] means for analyzing the text data and extracting important keywords and phrases;

[2136] means for generating or retrieving relevant visual elements based on said key keywords or phrases;

[2137] means for analyzing a user's emotion and adjusting the content and style of said associated visual elements;

[2138] means for integrating the related visual elements to generate an image or video summarizing the content of the text;

[2139] means for outputting the generated image or video;

[2140] A system including:

[2141] (Claim 2)

[2142] 2. The system according to claim 1, further comprising means for dividing the text data into sentences or paragraphs and tokenizing each sentence or paragraph.

[2143] (Claim 3)

[2144] 10. The system of claim 1, further comprising means for generating new images or videos using a generative AI model based on the extracted keywords or phrases.

[2145] "Application example 2 when combining emotion engines"

[2146] (Claim 1)

[2147] means for receiving text data;

[2148] means for analyzing the text data and extracting important keywords and phrases;

[2149] means for generating or retrieving relevant visual elements based on said key keywords or phrases;

[2150] means for integrating the related visual elements to generate an image or video summarizing the content of the text;

[2151] means for recognizing a user's emotion;

[2152] means for adjusting the content and style of the generated visual elements based on the user's emotions;

[2153] means for outputting the generated image or video;

[2154] A system including:

[2155] (Claim 2)

[2156] 2. The system according to claim 1, further comprising means for dividing the text data into sentences or paragraphs and tokenizing each sentence or paragraph.

[2157] (Claim 3)

[2158] 10. The system of claim 1, further comprising means for generating new images or videos using a generative AI model based on the extracted keywords or phrases. [Explanation of symbols]

[2159] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving text data; means for analyzing the text data and extracting important keywords and phrases; means for generating or retrieving relevant visual elements based on said key keywords or phrases; means for integrating the related visual elements to generate an image or video summarizing the content of the text; means for outputting the generated image or video; A system including:

2. 2. The system according to claim 1, further comprising means for dividing the text data into sentences or paragraphs and tokenizing each sentence or paragraph.

3. The system of claim 1, further comprising means for generating new images or videos using a generative AI model based on the extracted keywords or phrases.

Images