system

Abstract

We provide the system. [Solution] A means for receiving user input, A means of analyzing the received input query and extracting relevant keywords and phrases, A means of retrieving information by calling an external search engine API based on extracted keywords, A means of filtering the acquired information based on the user's past search history and preferences, A means of displaying filtered information to the user, A system that includes this.

Description

Technical Field

[0004]

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of the chatbot's character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In conventional search systems, there was a problem that users had to manually search for information, resulting in time and labor consumption. In addition, the mechanism for providing information appropriately filtered according to users' search needs and preferences was insufficient, and it often took a long time for users to reach the accurate information they needed. Furthermore, since past search histories and personal preferences were not considered, the user experience was limited and inefficient.

Means for Solving the Problems

[0005] The present invention solves the above-mentioned problems by providing a system that includes means for receiving user input, means for analyzing the received input query and extracting relevant keywords and phrases, means for calling an external search engine API based on the extracted keywords and obtaining information, means for filtering the obtained information based on the user's past search history and preferences, and means for displaying the filtered information to the user. With the present invention, users can efficiently and accurately access the information they want simply by entering a search query, significantly reducing the effort required for searching. Furthermore, by considering past search history and personal preferences, the user experience can be improved.

[0006] 1. "User input" refers to search queries and commands that users enter into the system.

[0007] 2. "Means of receiving" refers to the functions and methods for obtaining user input from a terminal and processing it within the system.

[0008] 3. "Analysis" refers to the process of breaking down an input query and extracting important keywords and phrases contained within it.

[0009] 4. "Keywords" are important words or phrases extracted from the user's input query, and serve as criteria for performing searches.

[0010] 5. A "phrase" refers to a combination of related words or phrases in an input query, and is used to improve search accuracy.

[0011] 6. "External search engine API" refers to an application programming interface (API) that exists outside the system and provides search functionality.

[0012] 7. "Means of obtaining information" refers to functions and methods for calling external search engine APIs to obtain relevant search results.

[0013] 8. "Filtering" refers to the process of selecting the most relevant information for the user from the retrieved search results.

[0014] 9. "User's past search history" refers to the history of search queries and content viewed by the user up to now.

[0015] 10. "Preferences" refer to tendencies and preferences based on the interests and concerns that a user has shown in the past.

[0016] 11. "Filtering methods" refer to functions and methods for optimizing retrieved search results based on the user's past search history and preferences.

[0017] 12. "Display" refers to the visual presentation of filtered search results on the user's device.

[0018] 13. "System" refers to a set of functions and components that process user input, retrieve and filter search results, and provide them to the user. [Brief explanation of the drawing]

[0019] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Mode for Carrying Out the Invention

[0020] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described according to the accompanying drawings.

[0021] First, the language used in the following description will be explained.

[0022] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), and APU (Accelerated Processing Unit).

[0023] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0024] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0025] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0027] [First Embodiment]

[0028] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0029] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0032] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0035] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0039] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0040] The present invention provides a system for users to efficiently retrieve information. This system has the functionality to automate a series of processes including user input, query analysis, retrieval of search results, filtering, and display of results. Embodiments of the present invention are described below.

[0041] System Configuration

[0042] When a user searches for information, they first enter a search query using a device. This device can be a smartphone, tablet, or personal computer, and uses a keyboard or touchscreen as its interface. The device receives this input and sends it to the server for processing.

[0043] The server uses natural language processing (NLP) algorithms to analyze the received queries. Specifically, it tokenizes the input queries and extracts important keywords and phrases. This analysis generates appropriate requests for search engines.

[0044] The server calls external search engine APIs based on the analyzed keywords. For example, it uses Google's search API to retrieve relevant information from the internet. The server then filters the retrieved search results based on the user's past search history and preferences. This extracts the information that is most relevant to the user.

[0045] The filtered information is sent to the device. The device displays the received information to the user. This includes the title, links, and summary. The user can access more detailed information by selecting and clicking the appropriate link from the displayed information.

[0046] Specific example

[0047] Example 1: Recipe Search

[0048] 1. The user enters "How to make a simple chocolate cake" into their device.

[0049] The terminal sends this query to the server.

[0050] 2. The server analyzes the query and extracts keywords such as "easy," "chocolate cake," and "how to make."

[0051] 3. The server uses the Google Search API to perform a search based on the analyzed keywords.

[0052] 4. The server retrieves search results and filters them based on the user's past search history and preferences.

[0053] Prioritize highly-rated recipes and those with short preparation times.

[0054] 5. The server sends the filtered search results to the terminal.

[0055] 6. The device displays the search results to the user.

[0056] Titles, links, and summaries related to "Easy Chocolate Cake Recipe".

[0057] Example 2: Searching for news

[0058] 1. The user enters the "latest technology news" into their device.

[0059] The terminal sends this query to the server.

[0060] 2. The server analyzes the query and extracts the keywords "latest," "technology," and "news."

[0061] 3. The server uses the news site's API to search for relevant news.

[0062] 4. The server filters the news results it retrieves, taking into account the user's past browsing history and preferences.

[0063] We prioritize articles from reliable news sources and those deemed to be of interest to users.

[0064] 5. The server sends filtered news articles to the terminal.

[0065] 6. The device displays news articles to the user.

[0066] Titles, links, summaries, etc., related to "the latest technology news."

[0067] These examples clearly demonstrate how the system of the present invention improves user search efficiency and how it is effective. This system allows users to efficiently obtain necessary information and significantly reduce the effort required for searching.

[0068] The following describes the processing flow.

[0069] Step 1:

[0070] The user enters a search query into their device. For example, they might type "latest technology news."

[0071] Step 2:

[0072] The terminal receives the entered search query and sends that data to the server as an HTTP POST request.

[0073] Step 3:

[0074] The server analyzes the received search queries. Specifically, it uses a natural language processing (NLP) library to tokenize the queries and extract important keywords and phrases.

[0075] Step 4:

[0076] The server constructs a search engine API request based on the extracted keywords. For example, it generates a URL for a Google Search API request.

[0077] Step 5:

[0078] The server sends the request it has created to an external search engine API to retrieve the relevant search results.

[0079] Step 6:

[0080] The server receives the search results returned from the search engine API and parses them in JSON format.

[0081] Step 7:

[0082] The server filters the search results it receives. Here, an algorithm is used to select information based on the user's past search history and configured preferences.

[0083] Step 8:

[0084] The server formats the filtered search results into HTML or JSON format and sends that data to the terminal.

[0085] Step 9:

[0086] The device parses the data received from the server and displays it in a user-friendly format. For example, it provides search result titles, links, and snippets.

[0087] Step 10:

[0088] Users can access detailed information by selecting and clicking the necessary links from the search results displayed on their device.

[0089] (Example 1)

[0090] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0091] Traditional information retrieval systems have presented challenges in efficiently obtaining the information users seek, often requiring significant time and effort. In particular, the lack of features to tailor results to users' past search history and preferences frequently resulted in reduced search accuracy and a poor user experience.

[0092] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0093] In this invention, the server includes means for receiving user input, means for analyzing the input query and extracting relevant keywords and phrases, means for calling external information acquisition means based on the extracted keywords and collecting information, means for filtering the collected information based on the user's past usage history and preferences, and means for displaying the filtered information to the user. This enables the user to efficiently obtain the necessary information and significantly reduce the effort required for searching.

[0094] "User input" refers to operations performed by a user through an interface using a terminal to provide search queries or commands to the system.

[0095] An "input query" is the text or voice input that a user enters into the system to search for information.

[0096] "Analysis" is the process of breaking down input data to understand its structure and meaning, and extracting important information.

[0097] "Keywords and phrases" are the main words or phrases that are extracted from the input query and used as the search target.

[0098] "External information acquisition methods" refer to protocols and APIs for collecting information from the internet or other databases based on input queries.

[0099] "Means of collecting information" refers to the process of incorporating relevant data into a system using external information acquisition methods.

[0100] "Filtering" refers to the process of selecting collected information based on the user's past usage history and preferences, and extracting the most relevant information.

[0101] "Means of display" refers to an interface for presenting filtered information on the user's device in an easily understandable format.

[0102] "Natural language processing algorithms" are theories and technologies that enable machines to understand and process human language, such as tokenizing text and performing grammatical analysis.

[0103] "Past usage history" refers to a record of the searches and information accessed by the user in the past.

[0104] "Preference" refers to the degree of a user's favorability towards specific information or content, inferred from their previous behavior and choices.

[0105] An "API for collecting information on a network" is an application programming interface for obtaining data via the internet or other networks.

[0106] This invention is a system for users to efficiently retrieve information. The system automates a series of processes including user input, query analysis, information collection, filtering, and result display, enabling users to quickly obtain the information they need. Specific embodiments are described in detail below.

[0107] System Configuration

[0108] User input

[0109] Users enter search queries using devices such as smartphones and personal computers, employing keyboards or touchscreens. These search queries are then sent to the server, for example, as HTTP requests.

[0110] Query analysis

[0111] The server uses Natural Language Processing (NLP) algorithms to parse the queries it receives. Specifically, it uses Python's NLTK library or spaCy to tokenize the input queries and extract key keywords and phrases. For example, the query "How to make a simple chocolate cake" is split into the tokens "simple," "chocolate cake," and "how to make."

[0112] Information gathering

[0113] The server calls external information retrieval methods, such as the Google Search API, based on the analyzed keywords. It collects relevant information from the internet using HTTP requests. Specifically, for example, to search for "how to make chocolate cake," it sends this query to an external API.

[0114] filtering

[0115] The server filters the retrieved information based on the user's past usage history and preferences. Using libraries such as Python's pandas library, it references past search history stored in a data frame and prioritizes extracting information that the user highly rated or that is suited to the user's preferences.

[0116] Results display

[0117] The filtered information is sent to the device. The device displays the received information in its user interface. Specifically, it uses HTML and CSS to neatly display the retrieved information about "easy chocolate cake recipes," including the title, links, and summary.

[0118] Specific example

[0119] Example 1: Recipe Search

[0120] 1. The user enters "How to make a simple chocolate cake" into their device.

[0121] The terminal sends this query to the server.

[0122] 2. The server parses the query and extracts the keywords "easy," "chocolate cake," and "how to make."

[0123] 3. The server uses information gathering tools and performs a search using the analyzed keywords.

[0124] 4. The server filters the search results based on the user's past search history and preferences. High-rated recipes and those with short preparation times are prioritized.

[0125] 5. The server sends the filtered results to the terminal.

[0126] 6. The device displays the results to the user.

[0127] Example 2: Searching for news

[0128] 1. The user enters the "latest technology news" into their device.

[0129] The terminal sends this query to the server.

[0130] 2. The server analyzes the query and extracts the keywords "latest," "technology," and "news."

[0131] 3. The server uses the news API to search for relevant news.

[0132] 4. The server filters the news results it retrieves, taking into account the user's past browsing history and preferences. It prioritizes reliable news sources and articles that the user is deemed to be of interest to.

[0133] 5. The server sends filtered news articles to the terminal.

[0134] 6. The device displays news articles to the user.

[0135] Example prompts for generative AI models

[0136] 1. "Could you please provide information on an introductory course to machine learning using Python?"

[0137] 2. "Please tell me the latest entertainment news."

[0138] This invention allows users to efficiently obtain the information they need and significantly reduce the effort required for searching.

[0139] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0140] Step 1:

[0141] The user enters a search query into the terminal. The user uses the keyboard or touchscreen to input specific information into the terminal, such as "How to make a simple chocolate cake." The input data here is in string format. The terminal receives this input and packets it as query data.

[0142] Step 2:

[0143] The terminal sends query data to the server. Here, an HTTP request is used, and the input query is URL-encoded before being sent to the server. The input is the user's search query, and the output is the HTTP request sent to the server.

[0144] Step 3:

[0145] The system analyzes the query data received by the server. Specifically, it uses Natural Language Processing (NLP) algorithms to tokenize the query and extract important keywords and phrases. For example, it can use Python's NLTK or spaCy library to analyze the query. In this case, the input is the query data of the HTTP request, and the output is a set of tokenized keywords.

[0146] Step 4:

[0147] The server collects information by calling external information retrieval methods (e.g., search engine APIs) based on the analyzed keywords. Here, HTTP requests are used to send keywords to the external API and retrieve relevant information. The input is a set of tokenized keywords, and the output is the retrieved search result data.

[0148] Step 5:

[0149] The server filters the retrieved data based on the user's past usage history and preferences. For filtering, it uses, for example, the Python pandas library to reference past history stored in a dataframe. The input consists of the retrieved search results data and the user's past history data, and the output is the filtered information.

[0150] Step 6:

[0151] The server sends filtered information to the terminal. Here too, data is sent as an HTTP response, providing the filtered results in JSON format or similar. The input is the filtered information, and the output is the response data sent to the terminal.

[0152] Step 7:

[0153] The device analyzes the information it receives and displays it on the user interface. Here, HTML and CSS are used to neatly display filtered information (e.g., information on how to make a simple chocolate cake). The input is the response data from the server, and the output is the content displayed on the user interface.

[0154] (Application Example 1)

[0155] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0156] Conventional information retrieval systems only perform searches based on user-entered queries and display the results. Therefore, there is a need for a system that automatically searches for information that users visually collect and provides relevant information in real time. The present invention aims to provide a system that analyzes objects and landscapes viewed by a user using smart glasses, efficiently acquires relevant information, filters it, and displays it.

[0157] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0158] In this invention, the server includes means for receiving user input, means for analyzing the received input query and extracting relevant keywords and phrases, means for calling an external search engine API based on the extracted keywords and obtaining information, means for filtering the obtained information based on the user's past search history and preferences, means for displaying the filtered information to the user, means for capturing video through smart glasses and performing object recognition, means for automatically generating search queries based on the recognized objects, and means for displaying the information on the smart glasses in real time. This makes it possible for users to efficiently search for information they have visually collected and to be provided with relevant information in real time.

[0159] "Means of receiving user input" refers to an interface for users to input information into a system, and is a mechanism for receiving input through devices such as smartphones and smart glasses.

[0160] "Means for analyzing received input queries and extracting relevant keywords and phrases" refers to a process that analyzes information entered by the user and identifies important elements.

[0161] "A means of calling an external search engine API based on extracted keywords to obtain information" refers to a mechanism that uses the analyzed keywords to collect relevant information via an external search engine.

[0162] "Methods for filtering acquired information based on the user's past search history and preferences" refers to a method of evaluating the relevance of collected information based on the user's past behavioral data and preferences, and selecting the most relevant information.

[0163] "Means of displaying filtered information to the user" refers to an interface that clearly presents selected information to the user on the device screen.

[0164] "A means of capturing images and performing object recognition through smart glasses" refers to image analysis technology that uses the built-in camera of smart glasses to capture images and identify objects within those images.

[0165] "Means for automatically generating search queries based on recognized objects" refers to an algorithm that automatically creates appropriate search queries using information about identified objects.

[0166] "A means of displaying information in real time on smart glasses" refers to a function that uses the display of smart glasses to instantly display the information the user needs.

[0167] This invention provides a system that allows users to efficiently collect visual information and search for and display relevant information in real time based on that information. This system consists of smart glasses, a server, and related software. Specific embodiments of the system are shown below.

[0168] Hardware configuration

[0169] Smart glasses: Equipped with a built-in camera, display, and network communication capabilities.

[0170] Server: Equipped with high-performance CPUs and GPUs, it processes large amounts of data. It also has internet connectivity.

[0171] Software Configuration

[0172] Image processing library (OpenCV): Analyzes video transmitted from smart glasses and performs object recognition.

[0173] API Request Library (Requests): Sends search queries to external search engine APIs to retrieve information.

[0174] Natural Language Processing Module (NLPProcessor): Generates appropriate search queries based on the results of object recognition.

[0175] Data processing and data calculation

[0176] Server Processing

[0177] Camera Capture: The smart glasses use their built-in camera to capture visual information and send the video to a server.

[0178] Object Recognition: The server performs object recognition on the received video using an image processing library (e.g., OpenCV). YOLO or MobileNet are used as object recognition models.

[0179] Query generation: Based on the recognized objects, the natural language processing module (NLPProcessor) automatically generates appropriate search queries.

[0180] Information Retrieval: Using the generated query, relevant information is retrieved from an external search engine API using the API Requests library.

[0181] Result filtering: Filters retrieved information based on the user's past search history and preferences.

[0182] Processing of smart glasses

[0183] Display: Filtered information is displayed in real time on the smart glasses' screen. This allows the user to instantly see relevant information.

[0184] Specific example

[0185] Imagine a user is in a museum wearing smart glasses and viewing a painting. The smart glasses' camera captures the painting and sends the image to a server. The server analyzes the image and identifies the painting's name and artist. Next, it generates a search query based on the identified information and retrieves relevant information (such as the artist's biography, other works, and explanations of the painting) through an external search engine API. The retrieved information is filtered based on the user's past search history and preferences and displayed in real time on the smart glasses' display.

[0186] Example of a prompt

[0187] "It recognizes objects from captured video footage and provides real-time information about those objects."

[0188] This invention allows users to efficiently and in real time acquire relevant information based on visually collected information, significantly reducing the effort required for searching and enabling them to obtain necessary information instantly.

[0189] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0190] Step 1:

[0191] Smart glasses use a built-in camera to capture visual information.

[0192] Input: The object or scenery the user is looking at.

[0193] Data processing: The camera acquires high-resolution video and generates video data from it.

[0194] Output: Video data.

[0195] Step 2:

[0196] The smart glasses send the captured video data to the server.

[0197] Input: Video data.

[0198] Data processing: Send data to the server using a network communication protocol.

[0199] Output: Video data sent to the server.

[0200] Step 3:

[0201] The server performs image processing on the received video data and then performs object recognition.

[0202] Input: Video data.

[0203] Data processing: Using an image processing library (OpenCV), the video is analyzed, and specific objects are identified using an object recognition model (e.g., YOLO or MobileNet).

[0204] Output: Recognition results such as object names.

[0205] Step 4:

[0206] The server automatically generates search queries using a natural language processing module (NLPProcessor) based on the objects it recognizes.

[0207] Input: Recognition results such as object names.

[0208] Data processing: Tokenize the recognition results and generate appropriate keywords and phrases.

[0209] Output: Search query.

[0210] Step 5:

[0211] The server uses the generated search query to send a request to an external search engine API and retrieve relevant information.

[0212] Input: Search query.

[0213] Data processing: Use the API Requests library to send requests to APIs that search for information on the internet and retrieve the information.

[0214] Output: Information from the search results.

[0215] Step 6:

[0216] The server filters the search results it retrieves based on the user's past search history and preferences.

[0217] Input: Information from the search results.

[0218] Data processing: Select highly relevant information by comparing it with past search history and user preference patterns.

[0219] Output: Filtered search results.

[0220] Step 7:

[0221] Display filtered search results on smart glasses.

[0222] Input: Filtered search results.

[0223] Data processing: Convert the data to a format suitable for smart glasses displays and display the information in real time.

[0224] Output: Relevant information displayed to the user.

[0225] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0226] The present invention provides a system that optimizes search results by combining an emotion engine that recognizes user emotions. This system has the functionality to automate a series of processes including user input, query analysis, emotion recognition, search result retrieval, filtering, and display of results. Embodiments of the present invention are described below.

[0227] System Configuration

[0228] When a user searches for information, they first enter a search query using a device. This device can be a smartphone, tablet, or personal computer, and uses a keyboard or touchscreen as its interface. The device receives this input and sends it to a server for processing. The device also incorporates an emotion engine to recognize the user's emotions, analyzing their facial expressions and tone of voice when they enter the search query to obtain emotional information.

[0229] The server uses natural language processing (NLP) and sentiment analysis algorithms to analyze the received queries and sentiment information. Specifically, it tokenizes the input queries and extracts important keywords and phrases. This analysis generates appropriate requests for the search engine.

[0230] The server calls external search engine APIs based on the analyzed keywords and sentiment information. For example, it uses Google's search API to retrieve relevant information from the internet. The server then filters the retrieved search results based on the user's past search history, preferences, and sentiment information. This extracts information that is most relevant to the user and appropriate to their emotional state.

[0231] The filtered information is sent to the device. The device displays the received information to the user. This includes the title, links, and summary. The display method and content are adjusted according to the user's emotional state. For example, users in a positive emotional state are shown in a bright and easy-to-read layout, while users in a negative emotional state are shown in a calming color scheme.

[0232] Specific example

[0233] Example 1: Recipe search and sentiment recognition

[0234] 1. The user enters "How to make a simple chocolate cake" into their device.

[0235] The device recognizes the user's emotions using its camera and microphone along with the search query, and the emotion engine detects if the user appears happy.

[0236] The device sends this query and sentiment information to the server.

[0237] 2. The server analyzes the query and sentiment information, extracting keywords such as "easy," "chocolate cake," and "how to make," along with positive sentiment information.

[0238] 3. The server uses the Google Search API to perform a search based on the analyzed keywords.

[0239] 4. The server retrieves search results and filters them based on the user's sentiment information.

[0240] Based on positive emotional information, we prioritize highly-rated recipes and easy, fun recipes that users will enjoy.

[0241] 5. The server sends the filtered search results to the terminal.

[0242] 6. The device displays search results to the user in a bright layout that emphasizes a positive atmosphere.

[0243] Titles, links, and summaries related to "Easy Chocolate Cake Recipe".

[0244] Example 2: News search and sentiment recognition

[0245] 1. The user enters the "latest technology news" into their device.

[0246] The device, along with the query, uses an emotion engine to detect that the user has a tired expression.

[0247] The device sends this query and sentiment information to the server.

[0248] 2. The server analyzes the query and sentiment information, extracting keywords such as "latest," "technology," and "news," as well as negative sentiment information.

[0249] 3. The server uses the news site's API to search for relevant news.

[0250] 4. The server filters the news results it retrieves, taking into account the user's sentiment.

[0251] Based on negative emotional information, the system prioritizes displaying news with less stressful content or positive news.

[0252] 5. The server sends filtered news articles to the terminal.

[0253] 6. The device displays news articles to the user using a calm color scheme.

[0254] Titles, links, summaries, etc., related to "the latest technology news."

[0255] This allows users to efficiently access the information they need and receive an optimal information experience tailored to their emotional state. This system makes the user's search experience more personalized and improves their satisfaction.

[0256] The following describes the processing flow.

[0257] Step 1:

[0258] The user enters a search query into their device. For example, they might type "latest technology news."

[0259] Step 2:

[0260] The device receives the entered search query, and simultaneously uses the camera and microphone to analyze the user's facial expressions and tone of voice, allowing the emotion engine to acquire the user's emotional information.

[0261] Step 3:

[0262] The device sends the search query and sentiment information to the server as an HTTP POST request.

[0263] Step 4:

[0264] The server analyzes the received search queries. Specifically, it uses a natural language processing (NLP) library to tokenize the queries and extract important keywords and phrases.

[0265] Step 5:

[0266] The server analyzes the received emotional information to identify the user's emotional state (positive, negative, neutral, etc.).

[0267] Step 6:

[0268] The server constructs a search engine API request based on the analyzed keywords and sentiment information. For example, it generates a URL for a Google Search API request.

[0269] Step 7:

[0270] The server sends the request it has created to an external search engine API to retrieve the relevant search results.

[0271] Step 8:

[0272] The server receives the search results returned from the search engine API and parses them in JSON format.

[0273] Step 9:

[0274] The server filters the search results it receives. Here, an algorithm is used to select information that takes into account the user's past search history, preferences, and sentiment.

[0275] Step 10:

[0276] The server formats the filtered search results into HTML or JSON format and sends that data to the terminal.

[0277] Step 11:

[0278] The device parses the data received from the server and displays it in a user-friendly format. The display format and layout are adjusted according to the user's emotional state.

[0279] Step 12:

[0280] Users can access detailed information by selecting and clicking the necessary links from the search results displayed on their device.

[0281] (Example 2)

[0282] Next, Example 2 will be described. In the following description, the data processing device 12 is referred to as a "server", and the smart device 14 is referred to as a "terminal".

[0283] Since the current search system does not consider the user's emotional state, the search results may not match the user's psychological state. As a result, there is a problem that the user cannot quickly access the information they need, leading to a decrease in satisfaction. Also, since the optimization of search results based on the user's emotions is not performed, the information provided may not necessarily match the user's current needs. Furthermore, the methods of filtering and display are fixed, and the personalization of the user experience is not sufficiently carried out.

[0284] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving user input, means for analyzing the received input query to extract relevant keywords and phrases, means for recognizing the user's emotions and obtaining emotion information, means for calling a search engine API based on the extracted keywords and the obtained emotion information to obtain information, means for filtering the obtained information based on the user's past search history and preferences and further optimizing it according to the emotion information, and means for displaying the filtered information to the user and adjusting the display method according to the emotion information. This enables the provision of search results suitable for the user's emotional state, improves the personalized user experience, and enhances user satisfaction.

[0285] "User input" refers to a search query or instruction input by the user via a keyboard or touch screen in a search system.

[0286] "Emotion information" refers to data related to the emotional state obtained from the user's facial expression and voice tone.

[0287] "Search engine API" refers to an application programming interface for accessing an external search engine service to obtain information.

[0288] "Search result filtering" is the process of selecting retrieved search results based on the user's past search history and preferences.

[0289] "User emotion recognition" is the process of identifying a user's emotional state by analyzing their facial expressions and voice tone using cameras and microphones.

[0290] A "natural language processing (NLP) algorithm" is a technical method for analyzing text data and extracting important keywords and phrases.

[0291] "User's past search history" refers to data that records the search queries a user has previously made and the information they have used.

[0292] An "emotion analysis algorithm" is a technical method that analyzes audio and video data to identify a user's emotions.

[0293] "Adjusting the display method" is the process of changing the layout and color scheme of search results according to the user's emotional state.

[0294] "Information on the Internet" refers to publicly available data that can be obtained from websites and online databases.

[0295] This invention relates to a system that recognizes a user's emotions and optimizes search results based on those emotions. The system performs user input, emotion recognition, communication with a search engine, filtering of results, and display of results as a series of automated processes.

[0296] System hardware and software configuration

[0297] 1. User input:

[0298] The user enters a search query using a terminal such as a smartphone, tablet, or personal computer. These terminals include a keyboard or touch screen as an interface.

[0299] 2. Emotion Recognition:

[0300] The terminal is equipped with a camera and a microphone, and these are used to analyze the user's expression and voice tone. The emotion engine obtains emotion information from this data and generates data in a tag format such as "positive" or "negative".

[0301] 3. Transmission of Search Query and Emotion Information:

[0302] The terminal transmits the acquired search query and emotion information to the server. In this process, the data is encrypted and transmitted securely.

[0303] 4. Query Analysis:

[0304] The server uses natural language processing (NLP) algorithms to tokenize the received query and extract important keywords and phrases. This analysis generates an appropriate request for the search engine.

[0305] 5. Emotion Analysis:

[0306] The server analyzes the received emotion information using emotion analysis algorithms. This enables search optimization according to the user's emotional state.

[0307] 6. Communication with Search Engine API:

[0308] The server calls an external search engine API such as the Google Search API and obtains relevant information based on the analyzed keywords. [[ID=4l]]

[0309] 7. Filtering and Optimization of Search Results:

[0310] The server filters the retrieved search results based on the user's past search history and preferences, and further optimizes them according to sentiment information. For example, users with positive sentiment information will be shown highly-rated recipes.

[0311] 8. Displaying search results:

[0312] The device displays filtered search results to the user. The display method is adjusted according to the user's emotional state, with options such as bright or calming color schemes being selected.

[0313] Specific example

[0314] Example 1: Recipe search and sentiment recognition

[0315] 1. The user enters "How to make a simple chocolate cake" into their device.

[0316] Along with the search query, the device uses its camera and microphone to detect if the user is making a happy expression, which is then detected by its emotion engine.

[0317] The device sends this query and sentiment information to the server.

[0318] 2. The server analyzes the query and sentiment information, extracting keywords such as "easy," "chocolate cake," and "how to make," along with positive sentiment information.

[0319] 3. The server uses the Google Search API to perform a search based on the analyzed keywords.

[0320] 4. The server retrieves search results and filters them based on the user's sentiment information.

[0321] Based on positive emotional information, we prioritize highly-rated recipes and easy, fun recipes that users will enjoy.

[0322] 5. The server sends the filtered search results to the terminal.

[0323] 6. The device displays search results to the user in a bright layout that emphasizes a positive atmosphere.

[0324] Titles, links, and summaries related to "Easy Chocolate Cake Recipe".

[0325] Example 2: News search and sentiment recognition

[0326] 1. The user enters the "latest technology news" into their device.

[0327] The device, along with the query, uses an emotion engine to detect that the user has a tired expression.

[0328] The device sends this query and sentiment information to the server.

[0329] 2. The server analyzes the query and sentiment information, extracting keywords such as "latest," "technology," and "news," as well as negative sentiment information.

[0330] 3. The server uses the news site's API to search for relevant news.

[0331] 4. The server filters the news results it retrieves, taking into account the user's sentiment.

[0332] Based on negative emotional information, the system prioritizes displaying news with less stressful content or positive news.

[0333] 5. The server sends filtered news articles to the terminal.

[0334] 6. The device displays news articles to the user using a calm color scheme.

[0335] Titles, links, summaries, etc., related to "the latest technology news."

[0336] Example of a prompt

[0337] "How to make an easy chocolate cake"

[0338] "Latest Technology News"

[0339] This system allows users to efficiently access the information they need and receive an optimal information experience tailored to their emotional state. The user's search experience is personalized, leading to increased satisfaction.

[0340] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0341] Step 1:

[0342] The user enters a search query.

[0343] Input: The user enters a search query such as "easy chocolate cake recipe" into the terminal.

[0344] Specific operation: The user enters queries using a keyboard or touchscreen on a smartphone or computer.

[0345] Output: The search query is entered into the terminal.

[0346] Step 2:

[0347] The device collects emotional information.

[0348] Input: User's facial expressions and voice data obtained from the camera and microphone built into the device.

[0349] Specific operation: The device's camera captures the user's face, and the microphone records the tone of their voice. The emotion engine analyzes this data in real time.

[0350] Output: Emotional information such as "positive" or "negative" is obtained.

[0351] Step 3:

[0352] The device sends search queries and sentiment information to the server.

[0353] Input: Search queries entered into the terminal and collected sentiment information.

[0354] Specific operation: The terminal uses a protocol that packages search queries and sentiment information, encrypts the data, and sends it to the server.

[0355] Output: The server receives the search query and sentiment information.

[0356] Step 4:

[0357] The server analyzes queries and sentiment information.

[0358] Input: Received search queries and sentiment information.

[0359] Specific operation: The server uses a natural language processing (NLP) algorithm to tokenize the query and extract important keywords and phrases. In parallel, a sentiment analysis algorithm analyzes sentiment information.

[0360] Output: Extracted keywords and analyzed sentiment information.

[0361] Step 5:

[0362] The server retrieves information using a search engine API.

[0363] Input: Extracted keywords and analyzed sentiment information.

[0364] Specific operation: The server generates a query to a search engine API (e.g., Google Search API), sends the appropriate request, and retrieves the information.

[0365] Output: Information obtained from the search engine API.

[0366] Step 6:

[0367] The server filters the search results.

[0368] Input: Information obtained from search engine APIs, user's past search history, and analyzed sentiment information.

[0369] Specific operation: The server filters information based on the user's past search history and preferences, and further optimizes it according to sentiment information. For example, users with positive sentiment information will be given priority in displaying highly-rated recipes.

[0370] Output: Filtered search results.

[0371] Step 7:

[0372] The server sends the filtered results to the terminal.

[0373] Input: Filtered search results.

[0374] Specific operation: The server uses a protocol to package filtered search results and send them to the terminal.

[0375] Output: The device receives filtered search results.

[0376] Step 8:

[0377] The device displays the results to the user.

[0378] Input: Filtered search results received from the server.

[0379] Specific operation: When displaying search results, the device adjusts the display method according to the user's emotional state. Users with a positive emotional state will be shown information in a bright layout, while users with a negative emotional state will be shown information in a calm color scheme.

[0380] Output: Optimized search results displayed to the user.

[0381] (Application Example 2)

[0382] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0383] Traditional search systems often display results without considering the user's emotional state, resulting in results that are unsuitable for the user's current mental state. This makes it difficult for users to quickly and accurately obtain the information they truly want, leading to a lower level of satisfaction with the search experience.

[0384] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for receiving user input, means for analyzing the received input query and extracting relevant keywords and phrases, means for calling an external search engine API based on the extracted keywords and obtaining information, means for filtering the obtained information based on the user's past search history and preferences, means for analyzing the user's facial expressions and tone of voice to obtain emotional information, means for further adjusting the filtering of search results based on the emotional information, and means for displaying the filtered information to the user. As a result, search results are provided that correspond to the user's emotional state, making it possible for the user to efficiently obtain the information they are looking for.

[0385] "Means for receiving user input" refers to devices or software that provide an interface that allows users to enter search queries.

[0386] "Means for analyzing received input queries and extracting relevant keywords and phrases" refers to algorithms and software that analyze input search queries and identify important words and phrases within them.

[0387] "A means of calling an external search engine API based on extracted keywords to obtain information" refers to algorithms or software that execute programs to search external resources such as the internet using keywords obtained from the analysis.

[0388] "Means for filtering acquired information based on the user's past search history and preferences" refers to systems and algorithms that reconstruct acquired search results into information that is highly relevant according to the user's past search behavior and preferences.

[0389] "Methods for acquiring emotional information by analyzing the user's facial expressions and tone of voice" refers to an emotion recognition engine that uses devices such as cameras and microphones to analyze the user's emotional state in real time and identify their emotions.

[0390] "Means for further adjusting the filtering of search results based on that sentiment information" refers to algorithms or software that customize search results to suit the user's current emotional state based on the acquired sentiment information.

[0391] "Means of displaying filtered information to the user" refers to displays or interfaces that visually present the filtered search results to the user.

[0392] The system for carrying out this invention is configured as follows.

[0393] System Configuration

[0394] 1. Means of receiving user input:

[0395] Users enter search queries using devices such as smartphones and tablets.

[0396] Keyboards and touchscreens are used as interfaces.

[0397] 2. Means for analyzing the received input query and extracting relevant keywords and phrases:

[0398] The server tokenizes the incoming search queries and uses natural language processing (NLP) algorithms to extract important keywords and phrases.

[0399] 3. Means of calling an external search engine API based on extracted keywords to retrieve information:

[0400] The server uses the analyzed keywords to call Google and other search engine APIs to retrieve relevant information.

[0401] 4. Means for filtering acquired information based on the user's past search history and preferences:

[0402] The server filters the search results by referring to the user's past search history and preferred databases.

[0403] 5. Means for obtaining emotional information by analyzing the user's facial expressions and tone of voice:

[0404] The system uses the smartphone's camera and microphone to analyze the user's facial expressions and voice tone in real time, acquiring emotional information. An emotion recognition engine (such as EmotionRecognizer) is used for emotion analysis.

[0405] 6. Means for further refining the filtering of search results based on that sentiment information:

[0406] The server adjusts the display order and content of search results based on the acquired sentiment information. For example, if a user is in a positive sentiment state, colorful and highly-rated products will be prioritized.

[0407] 7. Means for displaying filtered information to the user:

[0408] Filtered search results are displayed with a visual design that suits the user's emotional state. The screen's color scheme and layout are adjusted according to the user's emotions.

[0409] Specific example

[0410] For example, if a user searching for "smartphone cases" has a happy expression, follow these steps:

[0411] 1. User input: The user enters "smartphone case" into the smartphone app.

[0412] 2. Emotion Recognition: Using a camera and microphone, the system analyzes the user's joyful facial expressions and voice tone to detect positive emotions.

[0413] 3. Query Analysis and Search: The server uses a natural language processing algorithm to analyze the query for "smartphone case" and retrieves relevant information using the Google Search API.

[0414] 4. Filtering: The retrieved search results are filtered based on the user's past search history and positive sentiment information, prioritizing the display of colorful and highly-rated smartphone cases.

[0415] 5. Display Results: Filtered results are displayed with bright colors and large images.

[0416] An example of a prompt would be, "If the user has a cheerful expression, implement a filtering algorithm that prioritizes displaying only products with a bright, cheerful design and high ratings." On the other hand, if the user has a tired expression, it would be, "If the user has a tired expression, implement a filtering algorithm that prioritizes displaying only products with a calm design and low stress levels."

[0417] This system is expected to improve user experience and satisfaction by providing search results in real time that are tailored to the user's emotional state.

[0418] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0419] Step 1:

[0420] The user enters a search query using a device such as a smartphone or tablet. An interface is provided that allows the user to enter the query using a keyboard or touchscreen. The entered search query (e.g., "smartphone cases") is received by the device.

[0421] Step 2:

[0422] The device uses its camera and microphone to capture the user's facial expressions and voice tone in real time. The captured data is passed to an emotion recognition engine (e.g., EmotionRecognizer) and analyzed as user emotion information (e.g., positive emotions).

[0423] Step 3:

[0424] The device sends the entered search query and sentiment information to the server. The server tokenizes the received search query using a natural language processing algorithm (NLP) and extracts important keywords and phrases (e.g., "phone case").

[0425] Step 4:

[0426] The server calls an external search engine API (e.g., Google Search API) based on the extracted keywords to retrieve relevant information. The request sent to the search API includes the analyzed keywords. The information retrieved from the API is returned to the server.

[0427] Step 5:

[0428] The server filters the retrieved information based on the user's past search history and preferences. By referencing a database of the user's past search behavior and preferences, highly relevant information is identified.

[0429] Step 6:

[0430] The server further refines the filtering of search results based on the acquired sentiment information. Users with positive sentiment states will be filtered to prioritize displaying colorful and highly-rated products.

[0431] Step 7:

[0432] The server sends filtered information to the terminal. The terminal displays the received information to the user. The displayed search results are visually presented with a color scheme and layout appropriate to the user's emotional state. For example, a user in a positive emotional state will see search results displayed with bright colors and large images.

[0433] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0434] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0435] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0436] [Second Embodiment]

[0437] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0438] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0439] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0440] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0441] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0442] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0443] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0444] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0445] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0446] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0447] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0448] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0449] The present invention provides a system for users to efficiently retrieve information. This system has the functionality to automate a series of processes including user input, query analysis, retrieval of search results, filtering, and display of results. Embodiments of the present invention are described below.

[0450] System Configuration

[0451] When a user searches for information, they first enter a search query using a device. This device can be a smartphone, tablet, or personal computer, and uses a keyboard or touchscreen as its interface. The device receives this input and sends it to the server for processing.

[0452] The server uses natural language processing (NLP) algorithms to analyze the received queries. Specifically, it tokenizes the input queries and extracts important keywords and phrases. This analysis generates appropriate requests for search engines.

[0453] The server calls external search engine APIs based on the analyzed keywords. For example, it uses Google's search API to retrieve relevant information from the internet. The server then filters the retrieved search results based on the user's past search history and preferences. This extracts the information that is most relevant to the user.

[0454] The filtered information is sent to the device. The device displays the received information to the user. This includes the title, links, and summary. The user can access more detailed information by selecting and clicking the appropriate link from the displayed information.

[0455] Specific example

[0456] Example 1: Recipe Search

[0457] 1. The user enters "How to make a simple chocolate cake" into their device.

[0458] The terminal sends this query to the server.

[0459] 2. The server analyzes the query and extracts keywords such as "easy," "chocolate cake," and "how to make."

[0460] 3. The server uses the Google Search API to perform a search based on the analyzed keywords.

[0461] 4. The server retrieves the search results and filters them based on the user's past search history and preferences.

[0462] Prioritize highly-rated recipes and those with short preparation times.

[0463] 5. The server sends the filtered search results to the terminal.

[0464] 6. The device displays the search results to the user.

[0465] Titles, links, and summaries related to "Easy Chocolate Cake Recipe".

[0466] Example 2: Searching for news

[0467] 1. The user enters the "latest technology news" into their device.

[0468] The terminal sends this query to the server.

[0469] 2. The server analyzes the query and extracts the keywords "latest," "technology," and "news."

[0470] 3. The server uses the news site's API to search for relevant news.

[0471] 4. The server filters the news results it retrieves, taking into account the user's past browsing history and preferences.

[0472] We prioritize articles from reliable news sources and those deemed to be of interest to users.

[0473] 5. The server sends filtered news articles to the terminal.

[0474] 6. The device displays news articles to the user.

[0475] Titles, links, summaries, etc., related to "the latest technology news."

[0476] These examples clearly demonstrate how the system of the present invention improves user search efficiency and how it is effective. This system allows users to efficiently obtain necessary information and significantly reduce the effort required for searching.

[0477] The following describes the processing flow.

[0478] Step 1:

[0479] The user enters a search query into their device. For example, they might type "latest technology news."

[0480] Step 2:

[0481] The terminal receives the entered search query and sends that data to the server as an HTTP POST request.

[0482] Step 3:

[0483] The server analyzes the received search queries. Specifically, it uses a natural language processing (NLP) library to tokenize the queries and extract important keywords and phrases.

[0484] Step 4:

[0485] The server constructs a search engine API request based on the extracted keywords. For example, it generates a URL for a Google Search API request.

[0486] Step 5:

[0487] The server sends the request it has created to an external search engine API to retrieve the relevant search results.

[0488] Step 6:

[0489] The server receives the search results returned from the search engine API and parses them in JSON format.

[0490] Step 7:

[0491] The server filters the search results it receives. Here, an algorithm is used to select information based on the user's past search history and configured preferences.

[0492] Step 8:

[0493] The server formats the filtered search results into HTML or JSON format and sends that data to the terminal.

[0494] Step 9:

[0495] The device parses the data received from the server and displays it in a user-friendly format. For example, it provides search result titles, links, and snippets.

[0496] Step 10:

[0497] Users can access detailed information by selecting and clicking the necessary links from the search results displayed on their device.

[0498] (Example 1)

[0499] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0500] Traditional information retrieval systems have presented challenges in efficiently obtaining the information users seek, often requiring significant time and effort. In particular, the lack of features to tailor results to users' past search history and preferences frequently resulted in reduced search accuracy and a poor user experience.

[0501] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0502] In this invention, the server includes means for receiving user input, means for analyzing the input query and extracting relevant keywords and phrases, means for calling external information acquisition means based on the extracted keywords and collecting information, means for filtering the collected information based on the user's past usage history and preferences, and means for displaying the filtered information to the user. This enables the user to efficiently obtain the necessary information and significantly reduce the effort required for searching.

[0503] "User input" refers to operations performed by a user through an interface using a terminal to provide search queries or commands to the system.

[0504] An "input query" is the text or voice input that a user enters into the system to search for information.

[0505] "Analysis" is the process of breaking down input data to understand its structure and meaning, and extracting important information.

[0506] "Keywords and phrases" are the main words or phrases that are extracted from the input query and used as the search target.

[0507] "External information acquisition methods" refer to protocols and APIs for collecting information from the internet or other databases based on input queries.

[0508] "Means of collecting information" refers to the process of incorporating relevant data into a system using external information acquisition methods.

[0509] "Filtering" refers to the process of selecting collected information based on the user's past usage history and preferences, and extracting the most relevant information.

[0510] "Means of display" refers to an interface for presenting filtered information on the user's device in an easily understandable format.

[0511] "Natural language processing algorithms" are theories and technologies that enable machines to understand and process human language, such as tokenizing text and performing grammatical analysis.

[0512] "Past usage history" refers to a record of the searches and information accessed by the user in the past.

[0513] "Preference" refers to the degree of a user's favorability towards specific information or content, inferred from their previous behavior and choices.

[0514] An "API for collecting information on a network" is an application programming interface for obtaining data via the internet or other networks.

[0515] This invention is a system for users to efficiently retrieve information. The system automates a series of processes including user input, query analysis, information collection, filtering, and result display, enabling users to quickly obtain the information they need. Specific embodiments are described in detail below.

[0516] System Configuration

[0517] User input

[0518] Users enter search queries using devices such as smartphones and personal computers, employing keyboards or touchscreens. These search queries are then sent to the server, for example, as HTTP requests.

[0519] Query analysis

[0520] The server uses Natural Language Processing (NLP) algorithms to parse the queries it receives. Specifically, it uses Python's NLTK library or spaCy to tokenize the input queries and extract key keywords and phrases. For example, the query "How to make a simple chocolate cake" is split into the tokens "simple," "chocolate cake," and "how to make."

[0521] Information gathering

[0522] The server calls external information retrieval methods, such as the Google Search API, based on the analyzed keywords. It collects relevant information from the internet using HTTP requests. Specifically, for example, to search for "how to make chocolate cake," it sends this query to an external API.

[0523] filtering

[0524] The server filters the retrieved information based on the user's past usage history and preferences. Using libraries such as Python's pandas library, it references past search history stored in a data frame and prioritizes extracting information that the user highly rated or that is suited to the user's preferences.

[0525] Results display

[0526] The filtered information is sent to the device. The device displays the received information in its user interface. Specifically, it uses HTML and CSS to neatly display the retrieved information about "easy chocolate cake recipes," including the title, links, and summary.

[0527] Specific example

[0528] Example 1: Recipe Search

[0529] 1. The user enters "How to make a simple chocolate cake" into their device.

[0530] The terminal sends this query to the server.

[0531] 2. The server parses the query and extracts the keywords "easy," "chocolate cake," and "how to make."

[0532] 3. The server uses information gathering tools and performs a search using the analyzed keywords.

[0533] 4. The server filters the search results based on the user's past search history and preferences. High-rated recipes and those with short preparation times are prioritized.

[0534] 5. The server sends the filtered results to the terminal.

[0535] 6. The device displays the results to the user.

[0536] Example 2: Searching for news

[0537] 1. The user enters the "latest technology news" into their device.

[0538] The terminal sends this query to the server.

[0539] 2. The server analyzes the query and extracts the keywords "latest," "technology," and "news."

[0540] 3. The server uses the news API to search for relevant news.

[0541] 4. The server filters the news results it retrieves, taking into account the user's past browsing history and preferences. It prioritizes reliable news sources and articles that the user is deemed to be of interest to.

[0542] 5. The server sends filtered news articles to the terminal.

[0543] 6. The device displays news articles to the user.

[0544] Example prompts for generative AI models

[0545] 1. "Could you please provide information on an introductory course to machine learning using Python?"

[0546] 2. "Please tell me the latest entertainment news."

[0547] This invention allows users to efficiently obtain the information they need and significantly reduce the effort required for searching.

[0548] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0549] Step 1:

[0550] The user enters a search query into the terminal. The user uses the keyboard or touchscreen to input specific information into the terminal, such as "How to make a simple chocolate cake." The input data here is in string format. The terminal receives this input and packets it as query data.

[0551] Step 2:

[0552] The terminal sends query data to the server. Here, an HTTP request is used, and the input query is URL-encoded before being sent to the server. The input is the user's search query, and the output is the HTTP request sent to the server.

[0553] Step 3:

[0554] The system analyzes the query data received by the server. Specifically, it uses Natural Language Processing (NLP) algorithms to tokenize the query and extract important keywords and phrases. For example, it can use Python's NLTK or spaCy library to analyze the query. In this case, the input is the query data of the HTTP request, and the output is a set of tokenized keywords.

[0555] Step 4:

[0556] The server collects information by calling external information retrieval methods (e.g., search engine APIs) based on the analyzed keywords. Here, HTTP requests are used to send keywords to the external API and retrieve relevant information. The input is a set of tokenized keywords, and the output is the retrieved search result data.

[0557] Step 5:

[0558] The server filters the retrieved data based on the user's past usage history and preferences. For filtering, it uses, for example, the Python pandas library to reference past history stored in a dataframe. The input consists of the retrieved search results data and the user's past history data, and the output is the filtered information.

[0559] Step 6:

[0560] The server sends filtered information to the terminal. Here too, data is sent as an HTTP response, providing the filtered results in JSON format or similar. The input is the filtered information, and the output is the response data sent to the terminal.

[0561] Step 7:

[0562] The device analyzes the information it receives and displays it on the user interface. Here, HTML and CSS are used to neatly display filtered information (e.g., information on how to make a simple chocolate cake). The input is the response data from the server, and the output is the content displayed on the user interface.

[0563] (Application Example 1)

[0564] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0565] Conventional information retrieval systems only perform searches based on user-entered queries and display the results. Therefore, there is a need for a system that automatically searches for information that users visually collect and provides relevant information in real time. The present invention aims to provide a system that analyzes objects and landscapes viewed by a user using smart glasses, efficiently acquires relevant information, filters it, and displays it.

[0566] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0567] In this invention, the server includes means for receiving user input, means for analyzing the received input query and extracting relevant keywords and phrases, means for calling an external search engine API based on the extracted keywords and obtaining information, means for filtering the obtained information based on the user's past search history and preferences, means for displaying the filtered information to the user, means for capturing video through smart glasses and performing object recognition, means for automatically generating search queries based on the recognized objects, and means for displaying the information on the smart glasses in real time. This makes it possible for users to efficiently search for information they have visually collected and to be provided with relevant information in real time.

[0568] "Means of receiving user input" refers to an interface for users to input information into a system, and is a mechanism for receiving input through devices such as smartphones and smart glasses.

[0569] "Means for analyzing received input queries and extracting relevant keywords and phrases" refers to a process that analyzes information entered by the user and identifies important elements.

[0570] "A means of calling an external search engine API based on extracted keywords to obtain information" refers to a mechanism that uses the analyzed keywords to collect relevant information via an external search engine.

[0571] "Methods for filtering acquired information based on the user's past search history and preferences" refers to a method of evaluating the relevance of collected information based on the user's past behavioral data and preferences, and selecting the most relevant information.

[0572] "Means of displaying filtered information to the user" refers to an interface that clearly presents selected information to the user on the device screen.

[0573] "A means of capturing images and performing object recognition through smart glasses" refers to image analysis technology that uses the built-in camera of smart glasses to capture images and identify objects within those images.

[0574] "Means for automatically generating search queries based on recognized objects" refers to an algorithm that automatically creates appropriate search queries using information about identified objects.

[0575] "A means of displaying information in real time on smart glasses" refers to a function that uses the display of smart glasses to instantly display the information the user needs.

[0576] This invention provides a system that allows users to efficiently collect visual information and search for and display relevant information in real time based on that information. This system consists of smart glasses, a server, and related software. Specific embodiments of the system are shown below.

[0577] Hardware configuration

[0578] Smart glasses: Equipped with a built-in camera, display, and network communication capabilities.

[0579] Server: Equipped with high-performance CPUs and GPUs, it processes large amounts of data. It also has internet connectivity.

[0580] Software Configuration

[0581] Image processing library (OpenCV): Analyzes video transmitted from smart glasses and performs object recognition.

[0582] API Request Library (Requests): Sends search queries to external search engine APIs to retrieve information.

[0583] Natural Language Processing Module (NLPProcessor): Generates appropriate search queries based on the results of object recognition.

[0584] Data processing and data calculation

[0585] Server Processing

[0586] Camera Capture: The smart glasses use their built-in camera to capture visual information and send the video to a server.

[0587] Object Recognition: The server performs object recognition on the received video using an image processing library (e.g., OpenCV). YOLO or MobileNet are used as object recognition models.

[0588] Query generation: Based on the recognized objects, the natural language processing module (NLPProcessor) automatically generates appropriate search queries.

[0589] Information Retrieval: Using the generated query, relevant information is retrieved from an external search engine API using the API Requests library.

[0590] Result filtering: Filters retrieved information based on the user's past search history and preferences.

[0591] Processing of smart glasses

[0592] Display: Filtered information is displayed in real time on the smart glasses' screen. This allows the user to instantly see relevant information.

[0593] Specific example

[0594] Imagine a user is in a museum wearing smart glasses and viewing a painting. The smart glasses' camera captures the painting and sends the image to a server. The server analyzes the image and identifies the painting's name and artist. Next, it generates a search query based on the identified information and retrieves relevant information (such as the artist's biography, other works, and explanations of the painting) through an external search engine API. The retrieved information is filtered based on the user's past search history and preferences and displayed in real time on the smart glasses' display.

[0595] Example of a prompt

[0596] "It recognizes objects from captured video footage and provides real-time information about those objects."

[0597] This invention allows users to efficiently and in real time acquire relevant information based on visually collected information, significantly reducing the effort required for searching and enabling them to obtain necessary information instantly.

[0598] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0599] Step 1:

[0600] Smart glasses use a built-in camera to capture visual information.

[0601] Input: The object or scenery the user is looking at.

[0602] Data processing: The camera acquires high-resolution video and generates video data from it.

[0603] Output: Video data.

[0604] Step 2:

[0605] The smart glasses send the captured video data to the server.

[0606] Input: Video data.

[0607] Data processing: Send data to the server using a network communication protocol.

[0608] Output: Video data sent to the server.

[0609] Step 3:

[0610] The server performs image processing on the received video data and then performs object recognition.

[0611] Input: Video data.

[0612] Data processing: Using an image processing library (OpenCV), the video is analyzed, and specific objects are identified using an object recognition model (e.g., YOLO or MobileNet).

[0613] Output: Recognition results such as object names.

[0614] Step 4:

[0615] The server automatically generates search queries using a natural language processing module (NLPProcessor) based on the objects it recognizes.

[0616] Input: Recognition results such as object names.

[0617] Data processing: Tokenize the recognition results and generate appropriate keywords and phrases.

[0618] Output: Search query.

[0619] Step 5:

[0620] The server uses the generated search query to send a request to an external search engine API and retrieve relevant information.

[0621] Input: Search query.

[0622] Data processing: Use the API Requests library to send requests to APIs that search for information on the internet and retrieve the information.

[0623] Output: Information from the search results.

[0624] Step 6:

[0625] The server filters the search results it retrieves based on the user's past search history and preferences.

[0626] Input: Information from the search results.

[0627] Data processing: Select highly relevant information by comparing it with past search history and user preference patterns.

[0628] Output: Filtered search results.

[0629] Step 7:

[0630] Display filtered search results on smart glasses.

[0631] Input: Filtered search results.

[0632] Data processing: Convert the data to a format suitable for smart glasses displays and display the information in real time.

[0633] Output: Relevant information displayed to the user.

[0634] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0635] The present invention provides a system that optimizes search results by combining an emotion engine that recognizes user emotions. This system has the functionality to automate a series of processes including user input, query analysis, emotion recognition, search result retrieval, filtering, and display of results. Embodiments of the present invention are described below.

[0636] System Configuration

[0637] When a user searches for information, they first enter a search query using a device. This device can be a smartphone, tablet, or personal computer, and uses a keyboard or touchscreen as its interface. The device receives this input and sends it to a server for processing. The device also incorporates an emotion engine to recognize the user's emotions, analyzing their facial expressions and tone of voice when they enter the search query to obtain emotional information.

[0638] The server uses natural language processing (NLP) and sentiment analysis algorithms to analyze the received queries and sentiment information. Specifically, it tokenizes the input queries and extracts important keywords and phrases. This analysis generates appropriate requests for the search engine.

[0639] The server calls external search engine APIs based on the analyzed keywords and sentiment information. For example, it uses Google's search API to retrieve relevant information from the internet. The server then filters the retrieved search results based on the user's past search history, preferences, and sentiment information. This extracts information that is most relevant to the user and appropriate to their emotional state.

[0640] The filtered information is sent to the device. The device displays the received information to the user. This includes the title, links, and summary. The display method and content are adjusted according to the user's emotional state. For example, users in a positive emotional state are shown in a bright and easy-to-read layout, while users in a negative emotional state are shown in a calming color scheme.

[0641] Specific example

[0642] Example 1: Recipe search and sentiment recognition

[0643] 1. The user enters "How to make a simple chocolate cake" into their device.

[0644] The device recognizes the user's emotions using its camera and microphone along with the search query, and the emotion engine detects if the user appears happy.

[0645] The device sends this query and sentiment information to the server.

[0646] 2. The server analyzes the query and sentiment information, extracting keywords such as "easy," "chocolate cake," and "how to make," along with positive sentiment information.

[0647] 3. The server uses the Google Search API to perform a search based on the analyzed keywords.

[0648] 4. The server retrieves search results and filters them based on the user's sentiment information.

[0649] Based on positive emotional information, we prioritize highly-rated recipes and easy, fun recipes that users will enjoy.

[0650] 5. The server sends the filtered search results to the terminal.

[0651] 6. The device displays search results to the user in a bright layout that emphasizes a positive atmosphere.

[0652] Titles, links, and summaries related to "Easy Chocolate Cake Recipe".

[0653] Example 2: News search and sentiment recognition

[0654] 1. The user enters the "latest technology news" into their device.

[0655] The device, along with the query, uses an emotion engine to detect that the user has a tired expression.

[0656] The device sends this query and sentiment information to the server.

[0657] 2. The server analyzes the query and sentiment information, extracting keywords such as "latest," "technology," and "news," as well as negative sentiment information.

[0658] 3. The server uses the news site's API to search for relevant news.

[0659] 4. The server filters the news results it retrieves, taking into account the user's sentiment.

[0660] Based on negative emotional information, the system prioritizes displaying news with less stressful content or positive news.

[0661] 5. The server sends filtered news articles to the terminal.

[0662] 6. The device displays news articles to the user using a calm color scheme.

[0663] Titles, links, summaries, etc., related to "the latest technology news."

[0664] This allows users to efficiently access the information they need and receive an optimal information experience tailored to their emotional state. This system makes the user's search experience more personalized and improves their satisfaction.

[0665] The following describes the processing flow.

[0666] Step 1:

[0667] The user enters a search query into their device. For example, they might type "latest technology news."

[0668] Step 2:

[0669] The device receives the entered search query, and simultaneously uses the camera and microphone to analyze the user's facial expressions and tone of voice, allowing the emotion engine to acquire the user's emotional information.

[0670] Step 3:

[0671] The device sends the search query and sentiment information to the server as an HTTP POST request.

[0672] Step 4:

[0673] The server analyzes the received search queries. Specifically, it uses a natural language processing (NLP) library to tokenize the queries and extract important keywords and phrases.

[0674] Step 5:

[0675] The server analyzes the received emotional information to identify the user's emotional state (positive, negative, neutral, etc.).

[0676] Step 6:

[0677] The server constructs a search engine API request based on the analyzed keywords and sentiment information. For example, it generates a URL for a Google Search API request.

[0678] Step 7:

[0679] The server sends the request it has created to an external search engine API to retrieve the relevant search results.

[0680] Step 8:

[0681] The server receives the search results returned from the search engine API and parses them in JSON format.

[0682] Step 9:

[0683] The server filters the search results it receives. Here, an algorithm is used to select information based on the user's past search history, preferences, and sentiment.

[0684] Step 10:

[0685] The server formats the filtered search results into HTML or JSON format and sends that data to the terminal.

[0686] Step 11:

[0687] The device parses the data received from the server and displays it in a user-friendly format. The display format and layout are adjusted according to the user's emotional state.

[0688] Step 12:

[0689] Users can access detailed information by selecting and clicking the necessary links from the search results displayed on their device.

[0690] (Example 2)

[0691] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0692] Current search systems do not take into account the user's emotional state, meaning search results may not be relevant to the user's psychological state. This can lead to problems such as users not being able to quickly access the information they need, resulting in decreased satisfaction. Furthermore, because search results are not optimized based on the user's emotions, the information provided does not always match the user's current needs. In addition, filtering and display methods are fixed, and the user experience is not sufficiently personalized.

[0693] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving user input, means for analyzing the received input query and extracting relevant keywords and phrases, means for recognizing the user's emotions and acquiring emotion information, means for calling a search engine API based on the extracted keywords and acquired emotion information and acquiring information, means for filtering the acquired information based on the user's past search history and preferences and further optimizing it according to the emotion information, and means for displaying the filtered information to the user and adjusting the display method according to the emotion information. This makes it possible to provide search results that are suitable for the user's emotional state, improve the personalized user experience, and increase user satisfaction.

[0694] "User input" refers to search queries and instructions entered by the user via a keyboard or touchscreen in a search system.

[0695] "Emotional information" refers to data about a user's emotional state, obtained from their facial expressions and tone of voice.

[0696] A "search engine API" is an application programming interface for accessing external search engine services and retrieving information.

[0697] "Search result filtering" is the process of selecting retrieved search results based on the user's past search history and preferences.

[0698] "User emotion recognition" is the process of identifying a user's emotional state by analyzing their facial expressions and voice tone using cameras and microphones.

[0699] A "natural language processing (NLP) algorithm" is a technical method for analyzing text data and extracting important keywords and phrases.

[0700] "User's past search history" refers to data that records the search queries a user has previously made and the information they have used.

[0701] An "emotion analysis algorithm" is a technical method that analyzes audio and video data to identify a user's emotions.

[0702] "Adjusting the display method" is the process of changing the layout and color scheme of search results according to the user's emotional state.

[0703] "Information on the Internet" refers to publicly available data that can be obtained from websites and online databases.

[0704] This invention relates to a system that recognizes a user's emotions and optimizes search results based on those emotions. The system performs user input, emotion recognition, communication with a search engine, filtering of results, and display of results as a series of automated processes.

[0705] System hardware and software configuration

[0706] 1. User input:

[0707] Users enter search queries using devices such as smartphones, tablets, and personal computers. These devices include keyboards and touchscreens as interfaces.

[0708] 2. Emotion recognition:

[0709] The device has a built-in camera and microphone, which are used to analyze the user's facial expressions and voice tone. The emotion engine retrieves emotional information from this data and generates data in the form of tags such as "positive" and "negative."

[0710] 3. Submitting search queries and sentiment information:

[0711] The device sends the retrieved search queries and sentiment information to the server. During this process, the data is encrypted and transmitted securely.

[0712] 4. Query analysis:

[0713] The server uses natural language processing (NLP) algorithms to tokenize incoming queries and extract key keywords and phrases. This analysis generates appropriate requests for search engines.

[0714] 5. Emotion analysis:

[0715] The server uses an emotion analysis algorithm to analyze the received emotional information. This enables search optimization based on the user's emotional state.

[0716] 6. Communication with search engine APIs:

[0717] The server calls external search engine APIs, such as the Google Search API, to retrieve relevant information based on the analyzed keywords.

[0718] 7. Filtering and optimizing search results:

[0719] The server filters the retrieved search results based on the user's past search history and preferences, and further optimizes them according to sentiment information. For example, users with positive sentiment information will be shown highly-rated recipes.

[0720] 8. Displaying search results:

[0721] The device displays filtered search results to the user. The display method is adjusted according to the user's emotional state, with options such as bright or calming color schemes being selected.

[0722] Specific example

[0723] Example 1: Recipe search and sentiment recognition

[0724] 1. The user enters "How to make a simple chocolate cake" into their device.

[0725] The device uses its camera and microphone to detect if the user is making happy facial expressions, along with the search query, through its emotion engine.

[0726] The device sends this query and sentiment information to the server.

[0727] 2. The server analyzes the query and sentiment information, extracting keywords such as "easy," "chocolate cake," and "how to make," along with positive sentiment information.

[0728] 3. The server uses the Google Search API to perform a search based on the analyzed keywords.

[0729] 4. The server retrieves search results and filters them based on the user's sentiment information.

[0730] Based on positive emotional information, we prioritize highly-rated recipes and easy, fun recipes that users will enjoy.

[0731] 5. The server sends the filtered search results to the terminal.

[0732] 6. The device displays search results to the user in a bright layout that emphasizes a positive atmosphere.

[0733] Titles, links, and summaries related to "Easy Chocolate Cake Recipe".

[0734] Example 2: News search and sentiment recognition

[0735] 1. The user enters the "latest technology news" into their device.

[0736] The device, along with the query, uses an emotion engine to detect that the user has a tired expression.

[0737] The device sends this query and sentiment information to the server.

[0738] 2. The server analyzes the query and sentiment information, extracting keywords such as "latest," "technology," and "news," as well as negative sentiment information.

[0739] 3. The server uses the news site's API to search for relevant news.

[0740] 4. The server filters the news results it retrieves, taking into account the user's sentiment.

[0741] Based on negative emotional information, the system prioritizes displaying news with less stressful content or positive news.

[0742] 5. The server sends filtered news articles to the terminal.

[0743] 6. The device displays news articles to the user using a calm color scheme.

[0744] Titles, links, summaries, etc., related to "the latest technology news."

[0745] Example of a prompt

[0746] "How to make an easy chocolate cake"

[0747] "Latest Technology News"

[0748] This system allows users to efficiently access the information they need and receive an optimal information experience tailored to their emotional state. The user's search experience is personalized, leading to increased satisfaction.

[0749] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0750] Step 1:

[0751] The user enters a search query.

[0752] Input: The user enters a search query such as "easy chocolate cake recipe" into the terminal.

[0753] Specific operation: The user enters queries using a keyboard or touchscreen on a smartphone or computer.

[0754] Output: The search query is entered into the terminal.

[0755] Step 2:

[0756] The device collects emotional information.

[0757] Input: User's facial expressions and voice data obtained from the camera and microphone built into the device.

[0758] Specific operation: The device's camera captures the user's face, and the microphone records the tone of their voice. The emotion engine analyzes this data in real time.

[0759] Output: Emotional information such as "positive" or "negative" is obtained.

[0760] Step 3:

[0761] The device sends search queries and sentiment information to the server.

[0762] Input: Search queries entered into the terminal and collected sentiment information.

[0763] Specific operation: The terminal uses a protocol that packages search queries and sentiment information, encrypts the data, and sends it to the server.

[0764] Output: The server receives the search query and sentiment information.

[0765] Step 4:

[0766] The server analyzes queries and sentiment information.

[0767] Input: Received search queries and sentiment information.

[0768] Specific operation: The server uses a natural language processing (NLP) algorithm to tokenize the query and extract important keywords and phrases. In parallel, a sentiment analysis algorithm analyzes sentiment information.

[0769] Output: Extracted keywords and analyzed sentiment information.

[0770] Step 5:

[0771] The server retrieves information using a search engine API.

[0772] Input: Extracted keywords and analyzed sentiment information.

[0773] Specific operation: The server generates a query to a search engine API (e.g., Google Search API), sends the appropriate request, and retrieves the information.

[0774] Output: Information obtained from the search engine API.

[0775] Step 6:

[0776] The server filters the search results.

[0777] Input: Information obtained from search engine APIs, user's past search history, and analyzed sentiment information.

[0778] Specific operation: The server filters information based on the user's past search history and preferences, and further optimizes it according to sentiment information. For example, users with positive sentiment information will be given priority in displaying highly-rated recipes.

[0779] Output: Filtered search results.

[0780] Step 7:

[0781] The server sends the filtered results to the terminal.

[0782] Input: Filtered search results.

[0783] Specific operation: The server uses a protocol to package filtered search results and send them to the terminal.

[0784] Output: The device receives filtered search results.

[0785] Step 8:

[0786] The device displays the results to the user.

[0787] Input: Filtered search results received from the server.

[0788] Specific operation: When displaying search results, the device adjusts the display method according to the user's emotional state. Users with a positive emotional state will be shown information in a bright layout, while users with a negative emotional state will be shown information in a calm color scheme.

[0789] Output: Optimized search results displayed to the user.

[0790] (Application Example 2)

[0791] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0792] Traditional search systems often display results without considering the user's emotional state, resulting in results that are unsuitable for the user's current mental state. This makes it difficult for users to quickly and accurately obtain the information they truly want, leading to a lower level of satisfaction with the search experience.

[0793] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for receiving user input, means for analyzing the received input query and extracting relevant keywords and phrases, means for calling an external search engine API based on the extracted keywords and obtaining information, means for filtering the obtained information based on the user's past search history and preferences, means for analyzing the user's facial expressions and tone of voice to obtain emotional information, means for further adjusting the filtering of search results based on the emotional information, and means for displaying the filtered information to the user. As a result, search results are provided that correspond to the user's emotional state, making it possible for the user to efficiently obtain the information they are looking for.

[0794] "Means for receiving user input" refers to devices or software that provide an interface that allows users to enter search queries.

[0795] "Means for analyzing received input queries and extracting relevant keywords and phrases" refers to algorithms and software that analyze input search queries and identify important words and phrases within them.

[0796] "A means of calling an external search engine API based on extracted keywords to obtain information" refers to algorithms or software that execute programs to search external resources such as the internet using keywords obtained from the analysis.

[0797] "Means for filtering acquired information based on the user's past search history and preferences" refers to systems and algorithms that reconstruct acquired search results into information that is highly relevant according to the user's past search behavior and preferences.

[0798] "Methods for acquiring emotional information by analyzing the user's facial expressions and tone of voice" refers to an emotion recognition engine that uses devices such as cameras and microphones to analyze the user's emotional state in real time and identify their emotions.

[0799] "Means for further adjusting the filtering of search results based on that sentiment information" refers to algorithms or software that customize search results to suit the user's current emotional state based on the acquired sentiment information.

[0800] "Means of displaying filtered information to the user" refers to displays or interfaces that visually present the filtered search results to the user.

[0801] The system for carrying out this invention is configured as follows.

[0802] System Configuration

[0803] 1. Means of receiving user input:

[0804] Users enter search queries using devices such as smartphones and tablets.

[0805] Keyboards and touchscreens are used as interfaces.

[0806] 2. Means for analyzing the received input query and extracting relevant keywords and phrases:

[0807] The server tokenizes the incoming search queries and uses natural language processing (NLP) algorithms to extract important keywords and phrases.

[0808] 3. Means of calling an external search engine API based on extracted keywords to retrieve information:

[0809] The server uses the analyzed keywords to call Google and other search engine APIs to retrieve relevant information.

[0810] 4. Means for filtering acquired information based on the user's past search history and preferences:

[0811] The server filters the search results by referring to the user's past search history and preferred databases.

[0812] 5. Means for obtaining emotional information by analyzing the user's facial expressions and voice tone:

[0813] The system uses the smartphone's camera and microphone to analyze the user's facial expressions and voice tone in real time and acquire emotional information. An emotion recognition engine (e.g., EmotionRecognizer) is used for emotion analysis.

[0814] 6. Means for further refining the filtering of search results based on that sentiment information:

[0815] The server adjusts the display order and content of search results based on the acquired sentiment information. For example, if a user is in a positive sentiment state, colorful and highly-rated products will be prioritized.

[0816] 7. Means for displaying filtered information to the user:

[0817] Filtered search results are displayed with a visual design that suits the user's emotional state. The screen's color scheme and layout are adjusted according to the user's emotions.

[0818] Specific example

[0819] For example, if a user searching for "smartphone cases" has a happy expression, follow these steps:

[0820] 1. User input: The user enters "smartphone case" into the smartphone app.

[0821] 2. Emotion Recognition: Using a camera and microphone, the system analyzes the user's joyful facial expressions and voice tone to detect positive emotions.

[0822] 3. Query Analysis and Search: The server uses a natural language processing algorithm to analyze the query "smartphone case" and retrieves relevant information using the Google Search API.

[0823] 4. Filtering: The retrieved search results are filtered based on the user's past search history and positive sentiment information, prioritizing the display of colorful and highly-rated smartphone cases.

[0824] 5. Display Results: Filtered results are displayed with bright colors and large images.

[0825] An example of a prompt would be, "If the user has a cheerful expression, implement a filtering algorithm that prioritizes displaying only products with a bright, cheerful design and high ratings." On the other hand, if the user has a tired expression, it would be, "If the user has a tired expression, implement a filtering algorithm that prioritizes displaying only products with a calm design and low stress levels."

[0826] This system is expected to improve user experience and satisfaction by providing search results in real time that are tailored to the user's emotional state.

[0827] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0828] Step 1:

[0829] The user enters a search query using a device such as a smartphone or tablet. An interface is provided that allows the user to enter the query using a keyboard or touchscreen. The entered search query (e.g., "smartphone cases") is received by the device.

[0830] Step 2:

[0831] The device uses its camera and microphone to capture the user's facial expressions and voice tone in real time. The captured data is passed to an emotion recognition engine (e.g., EmotionRecognizer) and analyzed as user emotion information (e.g., positive emotions).

[0832] Step 3:

[0833] The device sends the entered search query and sentiment information to the server. The server tokenizes the received search query using a natural language processing (NLP) algorithm and extracts important keywords and phrases (e.g., "phone case").

[0834] Step 4:

[0835] The server calls an external search engine API (e.g., Google Search API) based on the extracted keywords to retrieve relevant information. The request sent to the search API includes the analyzed keywords. The information retrieved from the API is returned to the server.

[0836] Step 5:

[0837] The server filters the retrieved information based on the user's past search history and preferences. By referencing a database of the user's past search behavior and preferences, highly relevant information is identified.

[0838] Step 6:

[0839] The server further refines the filtering of search results based on the acquired sentiment information. Users with positive sentiment states will be filtered to prioritize displaying colorful and highly-rated products.

[0840] Step 7:

[0841] The server sends filtered information to the terminal. The terminal displays the received information to the user. The displayed search results are visually presented with a color scheme and layout appropriate to the user's emotional state. For example, a user in a positive emotional state will see search results displayed with bright colors and large images.

[0842] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0843] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0844] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0845] [Third Embodiment]

[0846] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0847] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0848] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0849] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0850] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0851] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0852] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0853] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0854] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0855] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0856] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0857] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0858] The present invention provides a system for users to efficiently retrieve information. This system has the functionality to automate a series of processes including user input, query analysis, retrieval of search results, filtering, and display of results. Embodiments of the present invention are described below.

[0859] System Configuration

[0860] When a user searches for information, they first enter a search query using a device. This device can be a smartphone, tablet, or personal computer, and uses a keyboard or touchscreen as its interface. The device receives this input and sends it to the server for processing.

[0861] The server uses natural language processing (NLP) algorithms to analyze the received queries. Specifically, it tokenizes the input queries and extracts important keywords and phrases. This analysis generates appropriate requests for search engines.

[0862] The server calls external search engine APIs based on the analyzed keywords. For example, it uses Google's search API to retrieve relevant information from the internet. The server then filters the retrieved search results based on the user's past search history and preferences. This extracts the information that is most relevant to the user.

[0863] The filtered information is sent to the device. The device displays the received information to the user. This includes the title, links, and summary. The user can access more detailed information by selecting and clicking the appropriate link from the displayed information.

[0864] Specific example

[0865] Example 1: Recipe Search

[0866] 1. The user enters "How to make a simple chocolate cake" into their device.

[0867] The terminal sends this query to the server.

[0868] 2. The server analyzes the query and extracts keywords such as "easy," "chocolate cake," and "how to make."

[0869] 3. The server uses the Google Search API to perform a search based on the analyzed keywords.

[0870] 4. The server retrieves the search results and filters them based on the user's past search history and preferences.

[0871] Prioritize highly-rated recipes and those with short preparation times.

[0872] 5. The server sends the filtered search results to the terminal.

[0873] 6. The device displays the search results to the user.

[0874] Titles, links, and summaries related to "Easy Chocolate Cake Recipe".

[0875] Example 2: Searching for news

[0876] 1. The user enters the "latest technology news" into their device.

[0877] The terminal sends this query to the server.

[0878] 2. The server analyzes the query and extracts the keywords "latest," "technology," and "news."

[0879] 3. The server uses the news site's API to search for relevant news.

[0880] 4. The server filters the news results it retrieves, taking into account the user's past browsing history and preferences.

[0881] We prioritize articles from reliable news sources and those deemed to be of interest to users.

[0882] 5. The server sends filtered news articles to the terminal.

[0883] 6. The device displays news articles to the user.

[0884] Titles, links, summaries, etc., related to "the latest technology news."

[0885] These examples clearly demonstrate how the system of the present invention improves user search efficiency and how it is effective. This system allows users to efficiently obtain necessary information and significantly reduce the effort required for searching.

[0886] The following describes the processing flow.

[0887] Step 1:

[0888] The user enters a search query into their device. For example, they might type "latest technology news."

[0889] Step 2:

[0890] The terminal receives the entered search query and sends that data to the server as an HTTP POST request.

[0891] Step 3:

[0892] The server analyzes the received search queries. Specifically, it uses a natural language processing (NLP) library to tokenize the queries and extract important keywords and phrases.

[0893] Step 4:

[0894] The server constructs a search engine API request based on the extracted keywords. For example, it generates a URL for a Google Search API request.

[0895] Step 5:

[0896] The server sends the request it has created to an external search engine API to retrieve the relevant search results.

[0897] Step 6:

[0898] The server receives the search results returned from the search engine API and parses them in JSON format.

[0899] Step 7:

[0900] The server filters the search results it receives. Here, an algorithm is used to select information based on the user's past search history and configured preferences.

[0901] Step 8:

[0902] The server formats the filtered search results into HTML or JSON format and sends that data to the terminal.

[0903] Step 9:

[0904] The device parses the data received from the server and displays it in a user-friendly format. For example, it provides search result titles, links, and snippets.

[0905] Step 10:

[0906] Users can access detailed information by selecting and clicking the necessary links from the search results displayed on their device.

[0907] (Example 1)

[0908] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0909] Traditional information retrieval systems have presented challenges in efficiently obtaining the information users seek, often requiring significant time and effort. In particular, the lack of features to tailor results to users' past search history and preferences frequently resulted in reduced search accuracy and a poor user experience.

[0910] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0911] In this invention, the server includes means for receiving user input, means for analyzing the input query and extracting relevant keywords and phrases, means for calling external information acquisition means based on the extracted keywords and collecting information, means for filtering the collected information based on the user's past usage history and preferences, and means for displaying the filtered information to the user. This enables the user to efficiently obtain the necessary information and significantly reduce the effort required for searching.

[0912] "User input" refers to operations performed by a user through an interface using a terminal to provide search queries or commands to the system.

[0913] An "input query" is the text or voice input that a user enters into the system to search for information.

[0914] "Analysis" is the process of breaking down input data to understand its structure and meaning, and extracting important information.

[0915] "Keywords and phrases" are the main words or phrases that are extracted from the input query and used as the search target.

[0916] "External information acquisition methods" refer to protocols and APIs for collecting information from the internet or other databases based on input queries.

[0917] "Means of collecting information" refers to the process of incorporating relevant data into a system using external information acquisition methods.

[0918] "Filtering" refers to the process of selecting collected information based on the user's past usage history and preferences, and extracting the most relevant information.

[0919] "Means of display" refers to an interface for presenting filtered information on the user's device in an easily understandable format.

[0920] "Natural language processing algorithms" are theories and technologies that enable machines to understand and process human language, such as tokenizing text and performing grammatical analysis.

[0921] "Past usage history" refers to a record of the searches and information accessed by the user in the past.

[0922] "Preference" refers to the degree of a user's favorability towards specific information or content, inferred from their previous behavior and choices.

[0923] An "API for collecting information on a network" is an application programming interface for obtaining data via the internet or other networks.

[0924] This invention is a system for users to efficiently retrieve information. The system automates a series of processes including user input, query analysis, information collection, filtering, and result display, enabling users to quickly obtain the information they need. Specific embodiments are described in detail below.

[0925] System Configuration

[0926] User input

[0927] Users enter search queries using devices such as smartphones and personal computers, employing keyboards or touchscreens. These search queries are then sent to the server, for example, as HTTP requests.

[0928] Query analysis

[0929] The server uses Natural Language Processing (NLP) algorithms to parse the queries it receives. Specifically, it uses Python's NLTK library or spaCy to tokenize the input queries and extract key keywords and phrases. For example, the query "How to make a simple chocolate cake" is split into the tokens "simple," "chocolate cake," and "how to make."

[0930] Information gathering

[0931] The server calls external information retrieval methods, such as the Google Search API, based on the analyzed keywords. It collects relevant information from the internet using HTTP requests. Specifically, for example, to search for "how to make chocolate cake," it sends this query to an external API.

[0932] filtering

[0933] The server filters the retrieved information based on the user's past usage history and preferences. Using libraries such as Python's pandas library, it references past search history stored in a data frame and prioritizes extracting information that the user highly rated or that is suited to the user's preferences.

[0934] Results display

[0935] The filtered information is sent to the device. The device displays the received information in its user interface. Specifically, it uses HTML and CSS to neatly display the retrieved information about "easy chocolate cake recipes," including the title, links, and summary.

[0936] Specific example

[0937] Example 1: Recipe Search

[0938] 1. The user enters "How to make a simple chocolate cake" into their device.

[0939] The terminal sends this query to the server.

[0940] 2. The server parses the query and extracts the keywords "easy," "chocolate cake," and "how to make."

[0941] 3. The server uses information gathering tools and performs a search using the analyzed keywords.

[0942] 4. The server filters the search results based on the user's past search history and preferences. High-rated recipes and those with short preparation times are prioritized.

[0943] 5. The server sends the filtered results to the terminal.

[0944] 6. The device displays the results to the user.

[0945] Example 2: Searching for news

[0946] 1. The user enters the "latest technology news" into their device.

[0947] The terminal sends this query to the server.

[0948] 2. The server analyzes the query and extracts the keywords "latest," "technology," and "news."

[0949] 3. The server uses the news API to search for relevant news.

[0950] 4. The server filters the news results it retrieves, taking into account the user's past browsing history and preferences. It prioritizes reliable news sources and articles that the user is deemed to be of interest to.

[0951] 5. The server sends filtered news articles to the terminal.

[0952] 6. The device displays news articles to the user.

[0953] Example prompts for generative AI models

[0954] 1. "Could you please provide information on an introductory course to machine learning using Python?"

[0955] 2. "Please tell me the latest entertainment news."

[0956] This invention allows users to efficiently obtain the information they need and significantly reduce the effort required for searching.

[0957] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0958] Step 1:

[0959] The user enters a search query into the terminal. The user uses the keyboard or touchscreen to input specific information into the terminal, such as "How to make a simple chocolate cake." The input data here is in string format. The terminal receives this input and packets it as query data.

[0960] Step 2:

[0961] The terminal sends query data to the server. Here, an HTTP request is used, and the input query is URL-encoded before being sent to the server. The input is the user's search query, and the output is the HTTP request sent to the server.

[0962] Step 3:

[0963] The system analyzes the query data received by the server. Specifically, it uses Natural Language Processing (NLP) algorithms to tokenize the query and extract important keywords and phrases. For example, it can use Python's NLTK or spaCy library to analyze the query. In this case, the input is the query data of the HTTP request, and the output is a set of tokenized keywords.

[0964] Step 4:

[0965] The server collects information by calling external information retrieval methods (e.g., search engine APIs) based on the analyzed keywords. Here, HTTP requests are used to send keywords to the external API and retrieve relevant information. The input is a set of tokenized keywords, and the output is the retrieved search result data.

[0966] Step 5:

[0967] The server filters the retrieved data based on the user's past usage history and preferences. For filtering, it uses, for example, the Python pandas library to reference past history stored in a dataframe. The input consists of the retrieved search results data and the user's past history data, and the output is the filtered information.

[0968] Step 6:

[0969] The server sends filtered information to the terminal. Here too, data is sent as an HTTP response, providing the filtered results in JSON format or similar. The input is the filtered information, and the output is the response data sent to the terminal.

[0970] Step 7:

[0971] The device analyzes the information it receives and displays it on the user interface. Here, HTML and CSS are used to neatly display filtered information (e.g., information on how to make a simple chocolate cake). The input is the response data from the server, and the output is the content displayed on the user interface.

[0972] (Application Example 1)

[0973] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0974] Conventional information retrieval systems only perform searches based on user-entered queries and display the results. Therefore, there is a need for a system that automatically searches for information that users visually collect and provides relevant information in real time. The present invention aims to provide a system that analyzes objects and landscapes viewed by a user using smart glasses, efficiently acquires relevant information, filters it, and displays it.

[0975] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0976] In this invention, the server includes means for receiving user input, means for analyzing the received input query and extracting relevant keywords and phrases, means for calling an external search engine API based on the extracted keywords and obtaining information, means for filtering the obtained information based on the user's past search history and preferences, means for displaying the filtered information to the user, means for capturing video through smart glasses and performing object recognition, means for automatically generating search queries based on the recognized objects, and means for displaying the information on the smart glasses in real time. This makes it possible for users to efficiently search for information they have visually collected and to be provided with relevant information in real time.

[0977] "Means of receiving user input" refers to an interface for users to input information into a system, and is a mechanism for receiving input through devices such as smartphones and smart glasses.

[0978] "Means for analyzing received input queries and extracting relevant keywords and phrases" refers to a process that analyzes information entered by the user and identifies important elements.

[0979] "A means of calling an external search engine API based on extracted keywords to obtain information" refers to a mechanism that uses the analyzed keywords to collect relevant information via an external search engine.

[0980] "Methods for filtering acquired information based on the user's past search history and preferences" refers to a method of evaluating the relevance of collected information based on the user's past behavioral data and preferences, and selecting the most relevant information.

[0981] "Means of displaying filtered information to the user" refers to an interface that clearly presents selected information to the user on the device screen.

[0982] "A means of capturing images and performing object recognition through smart glasses" refers to image analysis technology that uses the built-in camera of smart glasses to capture images and identify objects within those images.

[0983] "Means for automatically generating search queries based on recognized objects" refers to an algorithm that automatically creates appropriate search queries using information about identified objects.

[0984] "A means of displaying information in real time on smart glasses" refers to a function that uses the display of smart glasses to instantly display the information the user needs.

[0985] This invention provides a system that allows users to efficiently collect visual information and search for and display relevant information in real time based on that information. This system consists of smart glasses, a server, and related software. Specific embodiments of the system are shown below.

[0986] Hardware configuration

[0987] Smart glasses: Equipped with a built-in camera, display, and network communication capabilities.

[0988] Server: Equipped with high-performance CPUs and GPUs, it processes large amounts of data. It also has internet connectivity.

[0989] Software Configuration

[0990] Image processing library (OpenCV): Analyzes video transmitted from smart glasses and performs object recognition.

[0991] API Request Library (Requests): Sends search queries to external search engine APIs to retrieve information.

[0992] Natural Language Processing Module (NLPProcessor): Generates appropriate search queries based on the results of object recognition.

[0993] Data processing and data calculation

[0994] Server Processing

[0995] Camera Capture: The smart glasses use their built-in camera to capture visual information and send the video to a server.

[0996] Object Recognition: The server performs object recognition on the received video using an image processing library (e.g., OpenCV). YOLO or MobileNet are used as object recognition models.

[0997] Query generation: Based on the recognized objects, the natural language processing module (NLPProcessor) automatically generates appropriate search queries.

[0998] Information Retrieval: Using the generated query, relevant information is retrieved from an external search engine API using the API Requests library.

[0999] Result filtering: Filters retrieved information based on the user's past search history and preferences.

[1000] Processing of smart glasses

[1001] Display: Filtered information is displayed in real time on the smart glasses' screen. This allows the user to instantly see relevant information.

[1002] Specific example

[1003] Imagine a user is in a museum wearing smart glasses and viewing a painting. The smart glasses' camera captures the painting and sends the image to a server. The server analyzes the image and identifies the painting's name and artist. Next, it generates a search query based on the identified information and retrieves relevant information (such as the artist's biography, other works, and explanations of the painting) through an external search engine API. The retrieved information is filtered based on the user's past search history and preferences and displayed in real time on the smart glasses' display.

[1004] Example of a prompt

[1005] "It recognizes objects from captured video footage and provides real-time information about those objects."

[1006] This invention allows users to efficiently and in real time acquire relevant information based on visually collected information, significantly reducing the effort required for searching and enabling them to obtain necessary information instantly.

[1007] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1008] Step 1:

[1009] Smart glasses use a built-in camera to capture visual information.

[1010] Input: The object or scenery the user is looking at.

[1011] Data processing: The camera acquires high-resolution video and generates video data from it.

[1012] Output: Video data.

[1013] Step 2:

[1014] The smart glasses send the captured video data to the server.

[1015] Input: Video data.

[1016] Data processing: Send data to the server using a network communication protocol.

[1017] Output: Video data sent to the server.

[1018] Step 3:

[1019] The server performs image processing on the received video data and then performs object recognition.

[1020] Input: Video data.

[1021] Data processing: Using an image processing library (OpenCV), the video is analyzed, and specific objects are identified using an object recognition model (e.g., YOLO or MobileNet).

[1022] Output: Recognition results such as object names.

[1023] Step 4:

[1024] The server automatically generates search queries using a natural language processing module (NLPProcessor) based on the objects it recognizes.

[1025] Input: Recognition results such as object names.

[1026] Data processing: Tokenize the recognition results and generate appropriate keywords and phrases.

[1027] Output: Search query.

[1028] Step 5:

[1029] The server uses the generated search query to send a request to an external search engine API and retrieve relevant information.

[1030] Input: Search query.

[1031] Data processing: Use the API Requests library to send requests to APIs that search for information on the internet and retrieve the information.

[1032] Output: Information from the search results.

[1033] Step 6:

[1034] The server filters the search results it retrieves based on the user's past search history and preferences.

[1035] Input: Information from the search results.

[1036] Data processing: Select highly relevant information by comparing it with past search history and user preference patterns.

[1037] Output: Filtered search results.

[1038] Step 7:

[1039] Display filtered search results on smart glasses.

[1040] Input: Filtered search results.

[1041] Data processing: Convert the data to a format suitable for smart glasses displays and display the information in real time.

[1042] Output: Relevant information displayed to the user.

[1043] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1044] The present invention provides a system that optimizes search results by combining an emotion engine that recognizes user emotions. This system has the functionality to automate a series of processes including user input, query analysis, emotion recognition, search result retrieval, filtering, and display of results. Embodiments of the present invention are described below.

[1045] System Configuration

[1046] When a user searches for information, they first enter a search query using a device. This device can be a smartphone, tablet, or personal computer, and uses a keyboard or touchscreen as its interface. The device receives this input and sends it to a server for processing. The device also incorporates an emotion engine to recognize the user's emotions, analyzing their facial expressions and tone of voice when they enter the search query to obtain emotional information.

[1047] The server uses natural language processing (NLP) and sentiment analysis algorithms to analyze the received queries and sentiment information. Specifically, it tokenizes the input queries and extracts important keywords and phrases. This analysis generates appropriate requests for the search engine.

[1048] The server calls external search engine APIs based on the analyzed keywords and sentiment information. For example, it uses Google's search API to retrieve relevant information from the internet. The server then filters the retrieved search results based on the user's past search history, preferences, and sentiment information. This extracts information that is most relevant to the user and appropriate to their emotional state.

[1049] The filtered information is sent to the device. The device displays the received information to the user. This includes the title, links, and summary. The display method and content are adjusted according to the user's emotional state. For example, users in a positive emotional state are shown in a bright and easy-to-read layout, while users in a negative emotional state are shown in a calming color scheme.

[1050] Specific example

[1051] Example 1: Recipe search and sentiment recognition

[1052] 1. The user enters "How to make a simple chocolate cake" into their device.

[1053] The device recognizes the user's emotions using its camera and microphone along with the search query, and the emotion engine detects if the user appears happy.

[1054] The device sends this query and sentiment information to the server.

[1055] 2. The server analyzes the query and sentiment information, extracting keywords such as "easy," "chocolate cake," and "how to make," along with positive sentiment information.

[1056] 3. The server uses the Google Search API to perform a search based on the analyzed keywords.

[1057] 4. The server retrieves search results and filters them based on the user's sentiment information.

[1058] Based on positive emotional information, we prioritize highly-rated recipes and easy, fun recipes that users will enjoy.

[1059] 5. The server sends the filtered search results to the terminal.

[1060] 6. The device displays search results to the user in a bright layout that emphasizes a positive atmosphere.

[1061] Titles, links, and summaries related to "Easy Chocolate Cake Recipe".

[1062] Example 2: News search and sentiment recognition

[1063] 1. The user enters the "latest technology news" into their device.

[1064] The device, along with the query, uses an emotion engine to detect that the user has a tired expression.

[1065] The device sends this query and sentiment information to the server.

[1066] 2. The server analyzes the query and sentiment information, extracting keywords such as "latest," "technology," and "news," as well as negative sentiment information.

[1067] 3. The server uses the news site's API to search for relevant news.

[1068] 4. The server filters the news results it retrieves, taking into account the user's sentiment.

[1069] Based on negative emotional information, the system prioritizes displaying news with less stressful content or positive news.

[1070] 5. The server sends filtered news articles to the terminal.

[1071] 6. The device displays news articles to the user using a calm color scheme.

[1072] Titles, links, summaries, etc., related to "the latest technology news."

[1073] This allows users to efficiently access the information they need and receive an optimal information experience tailored to their emotional state. This system makes the user's search experience more personalized and improves their satisfaction.

[1074] The following describes the processing flow.

[1075] Step 1:

[1076] The user enters a search query into their device. For example, they might type "latest technology news."

[1077] Step 2:

[1078] The device receives the entered search query, and simultaneously uses the camera and microphone to analyze the user's facial expressions and tone of voice, allowing the emotion engine to acquire the user's emotional information.

[1079] Step 3:

[1080] The device sends the search query and sentiment information to the server as an HTTP POST request.

[1081] Step 4:

[1082] The server analyzes the received search queries. Specifically, it uses a natural language processing (NLP) library to tokenize the queries and extract important keywords and phrases.

[1083] Step 5:

[1084] The server analyzes the received emotional information to identify the user's emotional state (positive, negative, neutral, etc.).

[1085] Step 6:

[1086] The server constructs a search engine API request based on the analyzed keywords and sentiment information. For example, it generates a URL for a Google Search API request.

[1087] Step 7:

[1088] The server sends the request it has created to an external search engine API to retrieve the relevant search results.

[1089] Step 8:

[1090] The server receives the search results returned from the search engine API and parses them in JSON format.

[1091] Step 9:

[1092] The server filters the search results it receives. Here, an algorithm is used to select information based on the user's past search history, preferences, and sentiment.

[1093] Step 10:

[1094] The server formats the filtered search results into HTML or JSON format and sends that data to the terminal.

[1095] Step 11:

[1096] The device parses the data received from the server and displays it in a user-friendly format. The display format and layout are adjusted according to the user's emotional state.

[1097] Step 12:

[1098] Users can access detailed information by selecting and clicking the necessary links from the search results displayed on their device.

[1099] (Example 2)

[1100] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1101] Current search systems do not take into account the user's emotional state, meaning search results may not be relevant to the user's psychological state. This can lead to problems such as users not being able to quickly access the information they need, resulting in decreased satisfaction. Furthermore, because search results are not optimized based on the user's emotions, the information provided does not always match the user's current needs. In addition, filtering and display methods are fixed, and the user experience is not sufficiently personalized.

[1102] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving user input, means for analyzing the received input query and extracting relevant keywords and phrases, means for recognizing the user's emotions and acquiring emotion information, means for calling a search engine API based on the extracted keywords and acquired emotion information and acquiring information, means for filtering the acquired information based on the user's past search history and preferences and further optimizing it according to the emotion information, and means for displaying the filtered information to the user and adjusting the display method according to the emotion information. This makes it possible to provide search results that are suitable for the user's emotional state, improve the personalized user experience, and increase user satisfaction.

[1103] "User input" refers to search queries and instructions entered by the user via a keyboard or touchscreen in a search system.

[1104] "Emotional information" refers to data about a user's emotional state, obtained from their facial expressions and tone of voice.

[1105] A "search engine API" is an application programming interface for accessing external search engine services and retrieving information.

[1106] "Search result filtering" is the process of selecting retrieved search results based on the user's past search history and preferences.

[1107] "User emotion recognition" is the process of identifying a user's emotional state by analyzing their facial expressions and voice tone using cameras and microphones.

[1108] A "natural language processing (NLP) algorithm" is a technical method for analyzing text data and extracting important keywords and phrases.

[1109] "User's past search history" refers to data that records the search queries a user has previously made and the information they have used.

[1110] An "emotion analysis algorithm" is a technical method that analyzes audio and video data to identify a user's emotions.

[1111] "Adjusting the display method" is the process of changing the layout and color scheme of search results according to the user's emotional state.

[1112] "Information on the Internet" refers to publicly available data that can be obtained from websites and online databases.

[1113] This invention relates to a system that recognizes a user's emotions and optimizes search results based on those emotions. The system performs user input, emotion recognition, communication with a search engine, filtering of results, and display of results as a series of automated processes.

[1114] System hardware and software configuration

[1115] 1. User input:

[1116] Users enter search queries using devices such as smartphones, tablets, and personal computers. These devices include keyboards and touchscreens as interfaces.

[1117] 2. Emotion recognition:

[1118] The device has a built-in camera and microphone, which are used to analyze the user's facial expressions and voice tone. The emotion engine retrieves emotional information from this data and generates data in the form of tags such as "positive" and "negative."

[1119] 3. Submitting search queries and sentiment information:

[1120] The device sends the retrieved search queries and sentiment information to the server. During this process, the data is encrypted and transmitted securely.

[1121] 4. Query analysis:

[1122] The server uses natural language processing (NLP) algorithms to tokenize incoming queries and extract key keywords and phrases. This analysis generates appropriate requests for search engines.

[1123] 5. Emotion analysis:

[1124] The server uses an emotion analysis algorithm to analyze the received emotional information. This enables search optimization based on the user's emotional state.

[1125] 6. Communication with search engine APIs:

[1126] The server calls external search engine APIs, such as the Google Search API, to retrieve relevant information based on the analyzed keywords.

[1127] 7. Filtering and optimizing search results:

[1128] The server filters the retrieved search results based on the user's past search history and preferences, and further optimizes them according to sentiment information. For example, users with positive sentiment information will be shown highly-rated recipes.

[1129] 8. Displaying search results:

[1130] The device displays filtered search results to the user. The display method is adjusted according to the user's emotional state, with options such as bright or calming color schemes being selected.

[1131] Specific example

[1132] Example 1: Recipe search and sentiment recognition

[1133] 1. The user enters "How to make a simple chocolate cake" into their device.

[1134] The device uses its camera and microphone to detect if the user is making happy facial expressions, along with the search query, through its emotion engine.

[1135] The device sends this query and sentiment information to the server.

[1136] 2. The server analyzes the query and sentiment information, extracting keywords such as "easy," "chocolate cake," and "how to make," along with positive sentiment information.

[1137] 3. The server uses the Google Search API to perform a search based on the analyzed keywords.

[1138] 4. The server retrieves search results and filters them based on the user's sentiment information.

[1139] Based on positive emotional information, we prioritize highly-rated recipes and easy, fun recipes that users will enjoy.

[1140] 5. The server sends the filtered search results to the terminal.

[1141] 6. The device displays search results to the user in a bright layout that emphasizes a positive atmosphere.

[1142] Titles, links, and summaries related to "Easy Chocolate Cake Recipe".

[1143] Example 2: News search and sentiment recognition

[1144] 1. The user enters the "latest technology news" into their device.

[1145] The device, along with the query, uses an emotion engine to detect that the user has a tired expression.

[1146] The device sends this query and sentiment information to the server.

[1147] 2. The server analyzes the query and sentiment information, extracting keywords such as "latest," "technology," and "news," as well as negative sentiment information.

[1148] 3. The server uses the news site's API to search for relevant news.

[1149] 4. The server filters the news results it retrieves, taking into account the user's sentiment.

[1150] Based on negative emotional information, the system prioritizes displaying news with less stressful content or positive news.

[1151] 5. The server sends filtered news articles to the terminal.

[1152] 6. The device displays news articles to the user using a calm color scheme.

[1153] Titles, links, summaries, etc., related to "the latest technology news."

[1154] Example of a prompt

[1155] "How to make an easy chocolate cake"

[1156] "Latest Technology News"

[1157] This system allows users to efficiently access the information they need and receive an optimal information experience tailored to their emotional state. The user's search experience is personalized, leading to increased satisfaction.

[1158] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1159] Step 1:

[1160] The user enters a search query.

[1161] Input: The user enters a search query such as "easy chocolate cake recipe" into the terminal.

[1162] Specific operation: The user enters queries using a keyboard or touchscreen on a smartphone or computer.

[1163] Output: The search query is entered into the terminal.

[1164] Step 2:

[1165] The device collects emotional information.

[1166] Input: User's facial expressions and voice data obtained from the camera and microphone built into the device.

[1167] Specific operation: The device's camera captures the user's face, and the microphone records the tone of their voice. The emotion engine analyzes this data in real time.

[1168] Output: Emotional information such as "positive" or "negative" is obtained.

[1169] Step 3:

[1170] The device sends search queries and sentiment information to the server.

[1171] Input: Search queries entered into the terminal and collected sentiment information.

[1172] Specific operation: The terminal uses a protocol that packages search queries and sentiment information, encrypts the data, and sends it to the server.

[1173] Output: The server receives the search query and sentiment information.

[1174] Step 4:

[1175] The server analyzes queries and sentiment information.

[1176] Input: Received search queries and sentiment information.

[1177] Specific operation: The server uses a natural language processing (NLP) algorithm to tokenize the query and extract important keywords and phrases. In parallel, a sentiment analysis algorithm analyzes sentiment information.

[1178] Output: Extracted keywords and analyzed sentiment information.

[1179] Step 5:

[1180] The server retrieves information using a search engine API.

[1181] Input: Extracted keywords and analyzed sentiment information.

[1182] Specific operation: The server generates a query to a search engine API (e.g., Google Search API), sends the appropriate request, and retrieves the information.

[1183] Output: Information obtained from the search engine API.

[1184] Step 6:

[1185] The server filters the search results.

[1186] Input: Information obtained from search engine APIs, user's past search history, and analyzed sentiment information.

[1187] Specific operation: The server filters information based on the user's past search history and preferences, and further optimizes it according to sentiment information. For example, users with positive sentiment information will be given priority in displaying highly-rated recipes.

[1188] Output: Filtered search results.

[1189] Step 7:

[1190] The server sends the filtered results to the terminal.

[1191] Input: Filtered search results.

[1192] Specific operation: The server uses a protocol to package filtered search results and send them to the terminal.

[1193] Output: The device receives filtered search results.

[1194] Step 8:

[1195] The device displays the results to the user.

[1196] Input: Filtered search results received from the server.

[1197] Specific operation: When displaying search results, the device adjusts the display method according to the user's emotional state. Users with a positive emotional state will be shown information in a bright layout, while users with a negative emotional state will be shown information in a calm color scheme.

[1198] Output: Optimized search results displayed to the user.

[1199] (Application Example 2)

[1200] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1201] Traditional search systems often display results without considering the user's emotional state, resulting in results that are unsuitable for the user's current mental state. This makes it difficult for users to quickly and accurately obtain the information they truly want, leading to a lower level of satisfaction with the search experience.

[1202] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for receiving user input, means for analyzing the received input query and extracting relevant keywords and phrases, means for calling an external search engine API based on the extracted keywords and obtaining information, means for filtering the obtained information based on the user's past search history and preferences, means for analyzing the user's facial expressions and tone of voice to obtain emotional information, means for further adjusting the filtering of search results based on the emotional information, and means for displaying the filtered information to the user. As a result, search results are provided that correspond to the user's emotional state, making it possible for the user to efficiently obtain the information they are looking for.

[1203] "Means for receiving user input" refers to devices or software that provide an interface that allows users to enter search queries.

[1204] "Means for analyzing received input queries and extracting relevant keywords and phrases" refers to algorithms and software that analyze input search queries and identify important words and phrases within them.

[1205] "A means of calling an external search engine API based on extracted keywords to obtain information" refers to algorithms or software that execute programs to search external resources such as the internet using keywords obtained from the analysis.

[1206] "Means for filtering acquired information based on the user's past search history and preferences" refers to systems and algorithms that reconstruct acquired search results into information that is highly relevant according to the user's past search behavior and preferences.

[1207] "Methods for acquiring emotional information by analyzing the user's facial expressions and tone of voice" refers to an emotion recognition engine that uses devices such as cameras and microphones to analyze the user's emotional state in real time and identify their emotions.

[1208] "Means for further adjusting the filtering of search results based on that sentiment information" refers to algorithms or software that customize search results to suit the user's current emotional state based on the acquired sentiment information.

[1209] "Means of displaying filtered information to the user" refers to displays or interfaces that visually present the filtered search results to the user.

[1210] The system for carrying out this invention is configured as follows.

[1211] System Configuration

[1212] 1. Means of receiving user input:

[1213] Users enter search queries using devices such as smartphones and tablets.

[1214] Keyboards and touchscreens are used as interfaces.

[1215] 2. Means for analyzing the received input query and extracting relevant keywords and phrases:

[1216] The server tokenizes the incoming search queries and uses natural language processing (NLP) algorithms to extract important keywords and phrases.

[1217] 3. Means of calling an external search engine API based on extracted keywords to retrieve information:

[1218] The server uses the analyzed keywords to call Google and other search engine APIs to retrieve relevant information.

[1219] 4. Means for filtering acquired information based on the user's past search history and preferences:

[1220] The server filters the search results by referring to the user's past search history and preferred databases.

[1221] 5. Means for obtaining emotional information by analyzing the user's facial expressions and voice tone:

[1222] The system uses the smartphone's camera and microphone to analyze the user's facial expressions and voice tone in real time and acquire emotional information. An emotion recognition engine (e.g., EmotionRecognizer) is used for emotion analysis.

[1223] 6. Means for further refining the filtering of search results based on that sentiment information:

[1224] The server adjusts the display order and content of search results based on the acquired sentiment information. For example, if a user is in a positive sentiment state, colorful and highly-rated products will be prioritized.

[1225] 7. Means for displaying filtered information to the user:

[1226] Filtered search results are displayed with a visual design that suits the user's emotional state. The screen's color scheme and layout are adjusted according to the user's emotions.

[1227] Specific example

[1228] For example, if a user searching for "smartphone cases" has a happy expression, follow these steps:

[1229] 1. User input: The user enters "smartphone case" into the smartphone app.

[1230] 2. Emotion Recognition: Using a camera and microphone, the system analyzes the user's joyful facial expressions and voice tone to detect positive emotions.

[1231] 3. Query Analysis and Search: The server uses a natural language processing algorithm to analyze the query "smartphone case" and retrieves relevant information using the Google Search API.

[1232] 4. Filtering: The retrieved search results are filtered based on the user's past search history and positive sentiment information, prioritizing the display of colorful and highly-rated smartphone cases.

[1233] 5. Display Results: Filtered results are displayed with bright colors and large images.

[1234] An example of a prompt would be, "If the user has a cheerful expression, implement a filtering algorithm that prioritizes displaying only products with a bright, cheerful design and high ratings." On the other hand, if the user has a tired expression, it would be, "If the user has a tired expression, implement a filtering algorithm that prioritizes displaying only products with a calm design and low stress levels."

[1235] This system is expected to improve user experience and satisfaction by providing search results in real time that are tailored to the user's emotional state.

[1236] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1237] Step 1:

[1238] The user enters a search query using a device such as a smartphone or tablet. An interface is provided that allows the user to enter the query using a keyboard or touchscreen. The entered search query (e.g., "smartphone cases") is received by the device.

[1239] Step 2:

[1240] The device uses its camera and microphone to capture the user's facial expressions and voice tone in real time. The captured data is passed to an emotion recognition engine (e.g., EmotionRecognizer) and analyzed as user emotion information (e.g., positive emotions).

[1241] Step 3:

[1242] The device sends the entered search query and sentiment information to the server. The server tokenizes the received search query using a natural language processing (NLP) algorithm and extracts important keywords and phrases (e.g., "phone case").

[1243] Step 4:

[1244] The server calls an external search engine API (e.g., Google Search API) based on the extracted keywords to retrieve relevant information. The request sent to the search API includes the analyzed keywords. The information retrieved from the API is returned to the server.

[1245] Step 5:

[1246] The server filters the retrieved information based on the user's past search history and preferences. By referencing a database of the user's past search behavior and preferences, highly relevant information is identified.

[1247] Step 6:

[1248] The server further refines the filtering of search results based on the acquired sentiment information. Users with positive sentiment states will be filtered to prioritize displaying colorful and highly-rated products.

[1249] Step 7:

[1250] The server sends filtered information to the terminal. The terminal displays the received information to the user. The displayed search results are visually presented with a color scheme and layout appropriate to the user's emotional state. For example, a user in a positive emotional state will see search results displayed with bright colors and large images.

[1251] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1252] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1253] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1254] [Fourth Embodiment]

[1255] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1256] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1257] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1258] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1259] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1260] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1261] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1262] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1263] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1264] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1265] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1266] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1267] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1268] The present invention provides a system for users to efficiently retrieve information. This system has the functionality to automate a series of processes including user input, query analysis, retrieval of search results, filtering, and display of results. Embodiments of the present invention are described below.

[1269] System Configuration

[1270] When a user searches for information, they first enter a search query using a device. This device can be a smartphone, tablet, or personal computer, and uses a keyboard or touchscreen as its interface. The device receives this input and sends it to the server for processing.

[1271] The server uses natural language processing (NLP) algorithms to analyze the received queries. Specifically, it tokenizes the input queries and extracts important keywords and phrases. This analysis generates appropriate requests for search engines.

[1272] The server calls external search engine APIs based on the analyzed keywords. For example, it uses Google's search API to retrieve relevant information from the internet. The server then filters the retrieved search results based on the user's past search history and preferences. This extracts the information that is most relevant to the user.

[1273] The filtered information is sent to the device. The device displays the received information to the user. This includes the title, links, and summary. The user can access more detailed information by selecting and clicking the appropriate link from the displayed information.

[1274] Specific example

[1275] Example 1: Recipe Search

[1276] 1. The user enters "How to make a simple chocolate cake" into their device.

[1277] The terminal sends this query to the server.

[1278] 2. The server analyzes the query and extracts keywords such as "easy," "chocolate cake," and "how to make."

[1279] 3. The server uses the Google Search API to perform a search based on the analyzed keywords.

[1280] 4. The server retrieves the search results and filters them based on the user's past search history and preferences.

[1281] Prioritize highly-rated recipes and those with short preparation times.

[1282] 5. The server sends the filtered search results to the terminal.

[1283] 6. The device displays the search results to the user.

[1284] Titles, links, and summaries related to "Easy Chocolate Cake Recipe".

[1285] Example 2: Searching for news

[1286] 1. The user enters the "latest technology news" into their device.

[1287] The terminal sends this query to the server.

[1288] 2. The server analyzes the query and extracts the keywords "latest," "technology," and "news."

[1289] 3. The server uses the news site's API to search for relevant news.

[1290] 4. The server filters the news results it retrieves, taking into account the user's past browsing history and preferences.

[1291] We prioritize articles from reliable news sources and those deemed to be of interest to users.

[1292] 5. The server sends filtered news articles to the terminal.

[1293] 6. The device displays news articles to the user.

[1294] Titles, links, summaries, etc., related to "the latest technology news."

[1295] These examples clearly demonstrate how the system of the present invention improves user search efficiency and how it is effective. This system allows users to efficiently obtain necessary information and significantly reduce the effort required for searching.

[1296] The following describes the processing flow.

[1297] Step 1:

[1298] The user enters a search query into their device. For example, they might type "latest technology news."

[1299] Step 2:

[1300] The terminal receives the entered search query and sends that data to the server as an HTTP POST request.

[1301] Step 3:

[1302] The server analyzes the received search queries. Specifically, it uses a natural language processing (NLP) library to tokenize the queries and extract important keywords and phrases.

[1303] Step 4:

[1304] The server constructs a search engine API request based on the extracted keywords. For example, it generates a URL for a Google Search API request.

[1305] Step 5:

[1306] The server sends the request it has created to an external search engine API to retrieve the relevant search results.

[1307] Step 6:

[1308] The server receives the search results returned from the search engine API and parses them in JSON format.

[1309] Step 7:

[1310] The server filters the search results it receives. Here, an algorithm is used to select information based on the user's past search history and configured preferences.

[1311] Step 8:

[1312] The server formats the filtered search results into HTML or JSON format and sends that data to the terminal.

[1313] Step 9:

[1314] The device parses the data received from the server and displays it in a user-friendly format. For example, it provides search result titles, links, and snippets.

[1315] Step 10:

[1316] Users can access detailed information by selecting and clicking the necessary links from the search results displayed on their device.

[1317] (Example 1)

[1318] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1319] Traditional information retrieval systems have presented challenges in efficiently obtaining the information users seek, often requiring significant time and effort. In particular, the lack of features to tailor results to users' past search history and preferences frequently resulted in reduced search accuracy and a poor user experience.

[1320] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1321] In this invention, the server includes means for receiving user input, means for analyzing the input query and extracting relevant keywords and phrases, means for calling external information acquisition means based on the extracted keywords and collecting information, means for filtering the collected information based on the user's past usage history and preferences, and means for displaying the filtered information to the user. This enables the user to efficiently obtain the necessary information and significantly reduce the effort required for searching.

[1322] "User input" refers to operations performed by a user through an interface using a terminal to provide search queries or commands to the system.

[1323] An "input query" is the text or voice input that a user enters into the system to search for information.

[1324] "Analysis" is the process of breaking down input data to understand its structure and meaning, and extracting important information.

[1325] "Keywords and phrases" are the main words or phrases that are extracted from the input query and used as the search target.

[1326] "External information acquisition methods" refer to protocols and APIs for collecting information from the internet or other databases based on input queries.

[1327] "Means of collecting information" refers to the process of incorporating relevant data into a system using external information acquisition methods.

[1328] "Filtering" refers to the process of selecting collected information based on the user's past usage history and preferences, and extracting the most relevant information.

[1329] "Means of display" refers to an interface for presenting filtered information on the user's device in an easily understandable format.

[1330] "Natural language processing algorithms" are theories and technologies that enable machines to understand and process human language, such as tokenizing text and performing grammatical analysis.

[1331] "Past usage history" refers to a record of the searches and information accessed by the user in the past.

[1332] "Preference" refers to the degree of a user's favorability towards specific information or content, inferred from their previous behavior and choices.

[1333] An "API for collecting information on a network" is an application programming interface for obtaining data via the internet or other networks.

[1334] This invention is a system for users to efficiently retrieve information. The system automates a series of processes including user input, query analysis, information collection, filtering, and result display, enabling users to quickly obtain the information they need. Specific embodiments are described in detail below.

[1335] System Configuration

[1336] User input

[1337] Users enter search queries using devices such as smartphones and personal computers, employing keyboards or touchscreens. These search queries are then sent to the server, for example, as HTTP requests.

[1338] Query analysis

[1339] The server uses Natural Language Processing (NLP) algorithms to parse the queries it receives. Specifically, it uses Python's NLTK library or spaCy to tokenize the input queries and extract key keywords and phrases. For example, the query "How to make a simple chocolate cake" is split into the tokens "simple," "chocolate cake," and "how to make."

[1340] Information gathering

[1341] The server calls external information retrieval methods, such as the Google Search API, based on the analyzed keywords. It collects relevant information from the internet using HTTP requests. Specifically, for example, to search for "how to make chocolate cake," it sends this query to an external API.

[1342] filtering

[1343] The server filters the retrieved information based on the user's past usage history and preferences. Using libraries such as Python's pandas library, it references past search history stored in a data frame and prioritizes extracting information that the user highly rated or that is suited to the user's preferences.

[1344] Results display

[1345] The filtered information is sent to the device. The device displays the received information in its user interface. Specifically, it uses HTML and CSS to neatly display the retrieved information about "easy chocolate cake recipes," including the title, links, and summary.

[1346] Specific example

[1347] Example 1: Recipe Search

[1348] 1. The user enters "How to make a simple chocolate cake" into their device.

[1349] The terminal sends this query to the server.

[1350] 2. The server parses the query and extracts the keywords "easy," "chocolate cake," and "how to make."

[1351] 3. The server uses information gathering tools and performs a search using the analyzed keywords.

[1352] 4. The server filters the search results based on the user's past search history and preferences. High-rated recipes and those with short preparation times are prioritized.

[1353] 5. The server sends the filtered results to the terminal.

[1354] 6. The device displays the results to the user.

[1355] Example 2: Searching for news

[1356] 1. The user enters the "latest technology news" into their device.

[1357] The terminal sends this query to the server.

[1358] 2. The server analyzes the query and extracts the keywords "latest," "technology," and "news."

[1359] 3. The server uses the news API to search for relevant news.

[1360] 4. The server filters the news results it retrieves, taking into account the user's past browsing history and preferences. It prioritizes reliable news sources and articles that the user is deemed to be of interest to.

[1361] 5. The server sends filtered news articles to the terminal.

[1362] 6. The device displays news articles to the user.

[1363] Example prompts for generative AI models

[1364] 1. "Could you please provide information on an introductory course to machine learning using Python?"

[1365] 2. "Please tell me the latest entertainment news."

[1366] This invention allows users to efficiently obtain the information they need and significantly reduce the effort required for searching.

[1367] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1368] Step 1:

[1369] The user enters a search query into the terminal. The user uses the keyboard or touchscreen to input specific information into the terminal, such as "How to make a simple chocolate cake." The input data here is in string format. The terminal receives this input and packets it as query data.

[1370] Step 2:

[1371] The terminal sends query data to the server. Here, an HTTP request is used, and the input query is URL-encoded before being sent to the server. The input is the user's search query, and the output is the HTTP request sent to the server.

[1372] Step 3:

[1373] The system analyzes the query data received by the server. Specifically, it uses Natural Language Processing (NLP) algorithms to tokenize the query and extract important keywords and phrases. For example, it can use Python's NLTK or spaCy library to analyze the query. In this case, the input is the query data of the HTTP request, and the output is a set of tokenized keywords.

[1374] Step 4:

[1375] The server collects information by calling external information retrieval methods (e.g., search engine APIs) based on the analyzed keywords. Here, HTTP requests are used to send keywords to the external API and retrieve relevant information. The input is a set of tokenized keywords, and the output is the retrieved search result data.

[1376] Step 5:

[1377] The server filters the retrieved data based on the user's past usage history and preferences. For filtering, it uses, for example, the Python pandas library to reference past history stored in a dataframe. The input consists of the retrieved search results data and the user's past history data, and the output is the filtered information.

[1378] Step 6:

[1379] The server sends filtered information to the terminal. Here too, data is sent as an HTTP response, providing the filtered results in JSON format or similar. The input is the filtered information, and the output is the response data sent to the terminal.

[1380] Step 7:

[1381] The device analyzes the information it receives and displays it on the user interface. Here, HTML and CSS are used to neatly display filtered information (e.g., information on how to make a simple chocolate cake). The input is the response data from the server, and the output is the content displayed on the user interface.

[1382] (Application Example 1)

[1383] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1384] Conventional information retrieval systems only perform searches based on user-entered queries and display the results. Therefore, there is a need for a system that automatically searches for information that users visually collect and provides relevant information in real time. The present invention aims to provide a system that analyzes objects and landscapes viewed by a user using smart glasses, efficiently acquires relevant information, filters it, and displays it.

[1385] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1386] In this invention, the server includes means for receiving user input, means for analyzing the received input query and extracting relevant keywords and phrases, means for calling an external search engine API based on the extracted keywords and obtaining information, means for filtering the obtained information based on the user's past search history and preferences, means for displaying the filtered information to the user, means for capturing video through smart glasses and performing object recognition, means for automatically generating search queries based on the recognized objects, and means for displaying the information on the smart glasses in real time. This makes it possible for users to efficiently search for information they have visually collected and to be provided with relevant information in real time.

[1387] "Means of receiving user input" refers to an interface for users to input information into a system, and is a mechanism for receiving input through devices such as smartphones and smart glasses.

[1388] "Means for analyzing received input queries and extracting relevant keywords and phrases" refers to a process that analyzes information entered by the user and identifies important elements.

[1389] "A means of calling an external search engine API based on extracted keywords to obtain information" refers to a mechanism that uses the analyzed keywords to collect relevant information via an external search engine.

[1390] "Methods for filtering acquired information based on the user's past search history and preferences" refers to a method of evaluating the relevance of collected information based on the user's past behavioral data and preferences, and selecting the most relevant information.

[1391] "Means of displaying filtered information to the user" refers to an interface that clearly presents selected information to the user on the device screen.

[1392] "A means of capturing images and performing object recognition through smart glasses" refers to image analysis technology that uses the built-in camera of smart glasses to capture images and identify objects within those images.

[1393] "Means for automatically generating search queries based on recognized objects" refers to an algorithm that automatically creates appropriate search queries using information about identified objects.

[1394] "A means of displaying information in real time on smart glasses" refers to a function that uses the display of smart glasses to instantly display the information the user needs.

[1395] This invention provides a system that allows users to efficiently collect visual information and search for and display relevant information in real time based on that information. This system consists of smart glasses, a server, and related software. Specific embodiments of the system are shown below.

[1396] Hardware configuration

[1397] Smart glasses: Equipped with a built-in camera, display, and network communication capabilities.

[1398] Server: Equipped with high-performance CPUs and GPUs, it processes large amounts of data. It also has internet connectivity.

[1399] Software Configuration

[1400] Image processing library (OpenCV): Analyzes video transmitted from smart glasses and performs object recognition.

[1401] API Request Library (Requests): Sends search queries to external search engine APIs to retrieve information.

[1402] Natural Language Processing Module (NLPProcessor): Generates appropriate search queries based on the results of object recognition.

[1403] Data processing and data calculation

[1404] Server Processing

[1405] Camera Capture: The smart glasses use their built-in camera to capture visual information and send the video to a server.

[1406] Object Recognition: The server performs object recognition on the received video using an image processing library (e.g., OpenCV). YOLO or MobileNet are used as object recognition models.

[1407] Query generation: Based on the recognized objects, the natural language processing module (NLPProcessor) automatically generates appropriate search queries.

[1408] Information Retrieval: Using the generated query, relevant information is retrieved from an external search engine API using the API Requests library.

[1409] Result filtering: Filters retrieved information based on the user's past search history and preferences.

[1410] Processing of smart glasses

[1411] Display: Filtered information is displayed in real time on the smart glasses' screen. This allows the user to instantly see relevant information.

[1412] Specific example

[1413] Imagine a user is in a museum wearing smart glasses and viewing a painting. The smart glasses' camera captures the painting and sends the image to a server. The server analyzes the image and identifies the painting's name and artist. Next, it generates a search query based on the identified information and retrieves relevant information (such as the artist's biography, other works, and explanations of the painting) through an external search engine API. The retrieved information is filtered based on the user's past search history and preferences and displayed in real time on the smart glasses' display.

[1414] Example of a prompt

[1415] "It recognizes objects from captured video footage and provides real-time information about those objects."

[1416] This invention allows users to efficiently and in real time acquire relevant information based on visually collected information, significantly reducing the effort required for searching and enabling them to obtain necessary information instantly.

[1417] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1418] Step 1:

[1419] Smart glasses use a built-in camera to capture visual information.

[1420] Input: The object or scenery the user is looking at.

[1421] Data processing: The camera acquires high-resolution video and generates video data from it.

[1422] Output: Video data.

[1423] Step 2:

[1424] The smart glasses send the captured video data to the server.

[1425] Input: Video data.

[1426] Data processing: Send data to the server using a network communication protocol.

[1427] Output: Video data sent to the server.

[1428] Step 3:

[1429] The server performs image processing on the received video data and then performs object recognition.

[1430] Input: Video data.

[1431] Data processing: Using an image processing library (OpenCV), the video is analyzed, and specific objects are identified using an object recognition model (e.g., YOLO or MobileNet).

[1432] Output: Recognition results such as object names.

[1433] Step 4:

[1434] The server automatically generates search queries using a natural language processing module (NLPProcessor) based on the objects it recognizes.

[1435] Input: Recognition results such as object names.

[1436] Data processing: Tokenize the recognition results and generate appropriate keywords and phrases.

[1437] Output: Search query.

[1438] Step 5:

[1439] The server uses the generated search query to send a request to an external search engine API and retrieve relevant information.

[1440] Input: Search query.

[1441] Data processing: Use the API Requests library to send requests to APIs that search for information on the internet and retrieve the information.

[1442] Output: Information from the search results.

[1443] Step 6:

[1444] The server filters the search results it retrieves based on the user's past search history and preferences.

[1445] Input: Information from the search results.

[1446] Data processing: Select highly relevant information by comparing it with past search history and user preference patterns.

[1447] Output: Filtered search results.

[1448] Step 7:

[1449] Display filtered search results on smart glasses.

[1450] Input: Filtered search results.

[1451] Data processing: Convert the data to a format suitable for smart glasses displays and display the information in real time.

[1452] Output: Relevant information displayed to the user.

[1453] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1454] The present invention provides a system that optimizes search results by combining an emotion engine that recognizes user emotions. This system has the functionality to automate a series of processes including user input, query analysis, emotion recognition, search result retrieval, filtering, and display of results. Embodiments of the present invention are described below.

[1455] System Configuration

[1456] When a user searches for information, they first enter a search query using a device. This device can be a smartphone, tablet, or personal computer, and uses a keyboard or touchscreen as its interface. The device receives this input and sends it to a server for processing. The device also incorporates an emotion engine to recognize the user's emotions, analyzing their facial expressions and tone of voice when they enter the search query to obtain emotional information.

[1457] The server uses natural language processing (NLP) and sentiment analysis algorithms to analyze the received queries and sentiment information. Specifically, it tokenizes the input queries and extracts important keywords and phrases. This analysis generates appropriate requests for the search engine.

[1458] The server calls external search engine APIs based on the analyzed keywords and sentiment information. For example, it uses Google's search API to retrieve relevant information from the internet. The server then filters the retrieved search results based on the user's past search history, preferences, and sentiment information. This extracts information that is most relevant to the user and appropriate to their emotional state.

[1459] The filtered information is sent to the device. The device displays the received information to the user. This includes the title, links, and summary. The display method and content are adjusted according to the user's emotional state. For example, users in a positive emotional state are shown in a bright and easy-to-read layout, while users in a negative emotional state are shown in a calming color scheme.

[1460] Specific example

[1461] Example 1: Recipe search and sentiment recognition

[1462] 1. The user enters "How to make a simple chocolate cake" into their device.

[1463] The device recognizes the user's emotions using its camera and microphone along with the search query, and the emotion engine detects if the user appears happy.

[1464] The device sends this query and sentiment information to the server.

[1465] 2. The server analyzes the query and sentiment information, extracting keywords such as "easy," "chocolate cake," and "how to make," along with positive sentiment information.

[1466] 3. The server uses the Google Search API to perform a search based on the analyzed keywords.

[1467] 4. The server retrieves search results and filters them based on the user's sentiment information.

[1468] Based on positive emotional information, we prioritize highly-rated recipes and easy, fun recipes that users will enjoy.

[1469] 5. The server sends the filtered search results to the terminal.

[1470] 6. The device displays search results to the user in a bright layout that emphasizes a positive atmosphere.

[1471] Titles, links, and summaries related to "Easy Chocolate Cake Recipe".

[1472] Example 2: News search and sentiment recognition

[1473] 1. The user enters the "latest technology news" into their device.

[1474] The device, along with the query, uses an emotion engine to detect that the user has a tired expression.

[1475] The device sends this query and sentiment information to the server.

[1476] 2. The server analyzes the query and sentiment information, extracting keywords such as "latest," "technology," and "news," as well as negative sentiment information.

[1477] 3. The server uses the news site's API to search for relevant news.

[1478] 4. The server filters the news results it retrieves, taking into account the user's sentiment.

[1479] Based on negative emotional information, the system prioritizes displaying news with less stressful content or positive news.

[1480] 5. The server sends filtered news articles to the terminal.

[1481] 6. The device displays news articles to the user using a calm color scheme.

[1482] Titles, links, summaries, etc., related to "the latest technology news."

[1483] This allows users to efficiently access the information they need and receive an optimal information experience tailored to their emotional state. This system makes the user's search experience more personalized and improves their satisfaction.

[1484] The following describes the processing flow.

[1485] Step 1:

[1486] The user enters a search query into their device. For example, they might type "latest technology news."

[1487] Step 2:

[1488] The device receives the entered search query, and simultaneously uses the camera and microphone to analyze the user's facial expressions and tone of voice, allowing the emotion engine to acquire the user's emotional information.

[1489] Step 3:

[1490] The device sends the search query and sentiment information to the server as an HTTP POST request.

[1491] Step 4:

[1492] The server analyzes the received search queries. Specifically, it uses a natural language processing (NLP) library to tokenize the queries and extract important keywords and phrases.

[1493] Step 5:

[1494] The server analyzes the received emotional information to identify the user's emotional state (positive, negative, neutral, etc.).

[1495] Step 6:

[1496] The server constructs a search engine API request based on the analyzed keywords and sentiment information. For example, it generates a URL for a Google Search API request.

[1497] Step 7:

[1498] The server sends the request it has created to an external search engine API to retrieve the relevant search results.

[1499] Step 8:

[1500] The server receives the search results returned from the search engine API and parses them in JSON format.

[1501] Step 9:

[1502] The server filters the search results it receives. Here, an algorithm is used to select information based on the user's past search history, preferences, and sentiment.

[1503] Step 10:

[1504] The server formats the filtered search results into HTML or JSON format and sends that data to the terminal.

[1505] Step 11:

[1506] The device parses the data received from the server and displays it in a user-friendly format. The display format and layout are adjusted according to the user's emotional state.

[1507] Step 12:

[1508] Users can access detailed information by selecting and clicking the necessary links from the search results displayed on their device.

[1509] (Example 2)

[1510] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1511] Current search systems do not take into account the user's emotional state, meaning search results may not be relevant to the user's psychological state. This can lead to problems such as users not being able to quickly access the information they need, resulting in decreased satisfaction. Furthermore, because search results are not optimized based on the user's emotions, the information provided does not always match the user's current needs. In addition, filtering and display methods are fixed, and the user experience is not sufficiently personalized.

[1512] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving user input, means for analyzing the received input query and extracting relevant keywords and phrases, means for recognizing the user's emotions and acquiring emotion information, means for calling a search engine API based on the extracted keywords and acquired emotion information and acquiring information, means for filtering the acquired information based on the user's past search history and preferences and further optimizing it according to the emotion information, and means for displaying the filtered information to the user and adjusting the display method according to the emotion information. This makes it possible to provide search results that are suitable for the user's emotional state, improve the personalized user experience, and increase user satisfaction.

[1513] "User input" refers to search queries and instructions entered by the user via a keyboard or touchscreen in a search system.

[1514] "Emotional information" refers to data about a user's emotional state, obtained from their facial expressions and tone of voice.

[1515] A "search engine API" is an application programming interface for accessing external search engine services and retrieving information.

[1516] "Search result filtering" is the process of selecting retrieved search results based on the user's past search history and preferences.

[1517] "User emotion recognition" is the process of identifying a user's emotional state by analyzing their facial expressions and voice tone using cameras and microphones.

[1518] A "natural language processing (NLP) algorithm" is a technical method for analyzing text data and extracting important keywords and phrases.

[1519] "User's past search history" refers to data that records the search queries a user has previously made and the information they have used.

[1520] An "emotion analysis algorithm" is a technical method that analyzes audio and video data to identify a user's emotions.

[1521] "Adjusting the display method" is the process of changing the layout and color scheme of search results according to the user's emotional state.

[1522] "Information on the Internet" refers to publicly available data that can be obtained from websites and online databases.

[1523] This invention relates to a system that recognizes a user's emotions and optimizes search results based on those emotions. The system performs user input, emotion recognition, communication with a search engine, filtering of results, and display of results as a series of automated processes.

[1524] System hardware and software configuration

[1525] 1. User input:

[1526] Users enter search queries using devices such as smartphones, tablets, and personal computers. These devices include keyboards and touchscreens as interfaces.

[1527] 2. Emotion recognition:

[1528] The device has a built-in camera and microphone, which are used to analyze the user's facial expressions and voice tone. The emotion engine retrieves emotional information from this data and generates data in the form of tags such as "positive" and "negative."

[1529] 3. Submitting search queries and sentiment information:

[1530] The device sends the retrieved search queries and sentiment information to the server. During this process, the data is encrypted and transmitted securely.

[1531] 4. Query analysis:

[1532] The server uses natural language processing (NLP) algorithms to tokenize incoming queries and extract key keywords and phrases. This analysis generates appropriate requests for search engines.

[1533] 5. Emotion analysis:

[1534] The server uses an emotion analysis algorithm to analyze the received emotional information. This enables search optimization based on the user's emotional state.

[1535] 6. Communication with search engine APIs:

[1536] The server calls external search engine APIs, such as the Google Search API, to retrieve relevant information based on the analyzed keywords.

[1537] 7. Filtering and optimizing search results:

[1538] The server filters the retrieved search results based on the user's past search history and preferences, and further optimizes them according to sentiment information. For example, users with positive sentiment information will be shown highly-rated recipes.

[1539] 8. Displaying search results:

[1540] The device displays filtered search results to the user. The display method is adjusted according to the user's emotional state, with options such as bright or calming color schemes being selected.

[1541] Specific example

[1542] Example 1: Recipe search and sentiment recognition

[1543] 1. The user enters "How to make a simple chocolate cake" into their device.

[1544] The device uses its camera and microphone to detect if the user is making happy facial expressions, along with the search query, through its emotion engine.

[1545] The device sends this query and sentiment information to the server.

[1546] 2. The server analyzes the query and sentiment information, extracting keywords such as "easy," "chocolate cake," and "how to make," along with positive sentiment information.

[1547] 3. The server uses the Google Search API to perform a search based on the analyzed keywords.

[1548] 4. The server retrieves search results and filters them based on the user's sentiment information.

[1549] Based on positive emotional information, we prioritize highly-rated recipes and easy, fun recipes that users will enjoy.

[1550] 5. The server sends the filtered search results to the terminal.

[1551] 6. The device displays search results to the user in a bright layout that emphasizes a positive atmosphere.

[1552] Titles, links, and summaries related to "Easy Chocolate Cake Recipe".

[1553] Example 2: News search and sentiment recognition

[1554] 1. The user enters the "latest technology news" into their device.

[1555] The device, along with the query, uses an emotion engine to detect that the user has a tired expression.

[1556] The device sends this query and sentiment information to the server.

[1557] 2. The server analyzes the query and sentiment information, extracting keywords such as "latest," "technology," and "news," as well as negative sentiment information.

[1558] 3. The server uses the news site's API to search for relevant news.

[1559] 4. The server filters the news results it retrieves, taking into account the user's sentiment.

[1560] Based on negative emotional information, the system prioritizes displaying news with less stressful content or positive news.

[1561] 5. The server sends filtered news articles to the terminal.

[1562] 6. The device displays news articles to the user using a calm color scheme.

[1563] Titles, links, summaries, etc., related to "the latest technology news."

[1564] Example of a prompt

[1565] "How to make an easy chocolate cake"

[1566] "Latest Technology News"

[1567] This system allows users to efficiently access the information they need and receive an optimal information experience tailored to their emotional state. The user's search experience is personalized, leading to increased satisfaction.

[1568] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1569] Step 1:

[1570] The user enters a search query.

[1571] Input: The user enters a search query such as "easy chocolate cake recipe" into the terminal.

[1572] Specific operation: The user enters queries using a keyboard or touchscreen on a smartphone or computer.

[1573] Output: The search query is entered into the terminal.

[1574] Step 2:

[1575] The device collects emotional information.

[1576] Input: User's facial expressions and voice data obtained from the camera and microphone built into the device.

[1577] Specific operation: The device's camera captures the user's face, and the microphone records the tone of their voice. The emotion engine analyzes this data in real time.

[1578] Output: Emotional information such as "positive" or "negative" is obtained.

[1579] Step 3:

[1580] The device sends search queries and sentiment information to the server.

[1581] Input: Search queries entered into the terminal and collected sentiment information.

[1582] Specific operation: The terminal uses a protocol that packages search queries and sentiment information, encrypts the data, and sends it to the server.

[1583] Output: The server receives the search query and sentiment information.

[1584] Step 4:

[1585] The server analyzes queries and sentiment information.

[1586] Input: Received search queries and sentiment information.

[1587] Specific operation: The server uses a natural language processing (NLP) algorithm to tokenize the query and extract important keywords and phrases. In parallel, a sentiment analysis algorithm analyzes sentiment information.

[1588] Output: Extracted keywords and analyzed sentiment information.

[1589] Step 5:

[1590] The server retrieves information using a search engine API.

[1591] Input: Extracted keywords and analyzed sentiment information.

[1592] Specific operation: The server generates a query to a search engine API (e.g., Google Search API), sends the appropriate request, and retrieves the information.

[1593] Output: Information obtained from the search engine API.

[1594] Step 6:

[1595] The server filters the search results.

[1596] Input: Information obtained from search engine APIs, user's past search history, and analyzed sentiment information.

[1597] Specific operation: The server filters information based on the user's past search history and preferences, and further optimizes it according to sentiment information. For example, users with positive sentiment information will be given priority in displaying highly-rated recipes.

[1598] Output: Filtered search results.

[1599] Step 7:

[1600] The server sends the filtered results to the terminal.

[1601] Input: Filtered search results.

[1602] Specific operation: The server uses a protocol to package filtered search results and send them to the terminal.

[1603] Output: The device receives filtered search results.

[1604] Step 8:

[1605] The device displays the results to the user.

[1606] Input: Filtered search results received from the server.

[1607] Specific operation: When displaying search results, the device adjusts the display method according to the user's emotional state. Users with a positive emotional state will be shown information in a bright layout, while users with a negative emotional state will be shown information in a calm color scheme.

[1608] Output: Optimized search results displayed to the user.

[1609] (Application Example 2)

[1610] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1611] Traditional search systems often display results without considering the user's emotional state, resulting in results that are unsuitable for the user's current mental state. This makes it difficult for users to quickly and accurately obtain the information they truly want, leading to a lower level of satisfaction with the search experience.

[1612] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for receiving user input, means for analyzing the received input query and extracting relevant keywords and phrases, means for calling an external search engine API based on the extracted keywords and obtaining information, means for filtering the obtained information based on the user's past search history and preferences, means for analyzing the user's facial expressions and tone of voice to obtain emotional information, means for further adjusting the filtering of search results based on the emotional information, and means for displaying the filtered information to the user. As a result, search results are provided that correspond to the user's emotional state, making it possible for the user to efficiently obtain the information they are looking for.

[1613] "Means for receiving user input" refers to devices or software that provide an interface that allows users to enter search queries.

[1614] "Means for analyzing received input queries and extracting relevant keywords and phrases" refers to algorithms and software that analyze input search queries and identify important words and phrases within them.

[1615] "A means of calling an external search engine API based on extracted keywords to obtain information" refers to algorithms or software that execute programs to search external resources such as the internet using keywords obtained from the analysis.

[1616] "Means for filtering acquired information based on the user's past search history and preferences" refers to systems and algorithms that reconstruct acquired search results into information that is highly relevant according to the user's past search behavior and preferences.

[1617] "Methods for acquiring emotional information by analyzing the user's facial expressions and tone of voice" refers to an emotion recognition engine that uses devices such as cameras and microphones to analyze the user's emotional state in real time and identify their emotions.

[1618] "Means for further adjusting the filtering of search results based on that sentiment information" refers to algorithms or software that customize search results to suit the user's current emotional state based on the acquired sentiment information.

[1619] "Means of displaying filtered information to the user" refers to displays or interfaces that visually present the filtered search results to the user.

[1620] The system for carrying out this invention is configured as follows.

[1621] System Configuration

[1622] 1. Means of receiving user input:

[1623] Users enter search queries using devices such as smartphones and tablets.

[1624] Keyboards and touchscreens are used as interfaces.

[1625] 2. Means for analyzing the received input query and extracting relevant keywords and phrases:

[1626] The server tokenizes the incoming search queries and uses natural language processing (NLP) algorithms to extract important keywords and phrases.

[1627] 3. Means of calling an external search engine API based on extracted keywords to retrieve information:

[1628] The server uses the analyzed keywords to call Google and other search engine APIs to retrieve relevant information.

[1629] 4. Means for filtering acquired information based on the user's past search history and preferences:

[1630] The server filters the search results by referring to the user's past search history and preferred databases.

[1631] 5. Means for obtaining emotional information by analyzing the user's facial expressions and voice tone:

[1632] The system uses the smartphone's camera and microphone to analyze the user's facial expressions and voice tone in real time and acquire emotional information. An emotion recognition engine (e.g., EmotionRecognizer) is used for emotion analysis.

[1633] 6. Means for further refining the filtering of search results based on that sentiment information:

[1634] The server adjusts the display order and content of search results based on the acquired sentiment information. For example, if a user is in a positive sentiment state, colorful and highly-rated products will be prioritized.

[1635] 7. Means for displaying filtered information to the user:

[1636] Filtered search results are displayed with a visual design that suits the user's emotional state. The screen's color scheme and layout are adjusted according to the user's emotions.

[1637] Specific example

[1638] For example, if a user searching for "smartphone cases" has a happy expression, follow these steps:

[1639] 1. User input: The user enters "smartphone case" into the smartphone app.

[1640] 2. Emotion Recognition: Using a camera and microphone, the system analyzes the user's joyful facial expressions and voice tone to detect positive emotions.

[1641] 3. Query Analysis and Search: The server uses a natural language processing algorithm to analyze the query "smartphone case" and retrieves relevant information using the Google Search API.

[1642] 4. Filtering: The retrieved search results are filtered based on the user's past search history and positive sentiment information, prioritizing the display of colorful and highly-rated smartphone cases.

[1643] 5. Display Results: Filtered results are displayed with bright colors and large images.

[1644] An example of a prompt would be, "If the user has a cheerful expression, implement a filtering algorithm that prioritizes displaying only products with a bright, cheerful design and high ratings." On the other hand, if the user has a tired expression, it would be, "If the user has a tired expression, implement a filtering algorithm that prioritizes displaying only products with a calm design and low stress levels."

[1645] This system is expected to improve user experience and satisfaction by providing search results in real time that are tailored to the user's emotional state.

[1646] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1647] Step 1:

[1648] The user enters a search query using a device such as a smartphone or tablet. An interface is provided that allows the user to enter the query using a keyboard or touchscreen. The entered search query (e.g., "smartphone cases") is received by the device.

[1649] Step 2:

[1650] The device uses its camera and microphone to capture the user's facial expressions and voice tone in real time. The captured data is passed to an emotion recognition engine (e.g., EmotionRecognizer) and analyzed as user emotion information (e.g., positive emotions).

[1651] Step 3:

[1652] The device sends the entered search query and sentiment information to the server. The server tokenizes the received search query using a natural language processing (NLP) algorithm and extracts important keywords and phrases (e.g., "phone case").

[1653] Step 4:

[1654] The server calls an external search engine API (e.g., Google Search API) based on the extracted keywords to retrieve relevant information. The request sent to the search API includes the analyzed keywords. The information retrieved from the API is returned to the server.

[1655] Step 5:

[1656] The server filters the retrieved information based on the user's past search history and preferences. By referencing a database of the user's past search behavior and preferences, highly relevant information is identified.

[1657] Step 6:

[1658] The server further refines the filtering of search results based on the acquired sentiment information. Users with positive sentiment states will be filtered to prioritize displaying colorful and highly-rated products.

[1659] Step 7:

[1660] The server sends filtered information to the terminal. The terminal displays the received information to the user. The displayed search results are visually presented with a color scheme and layout appropriate to the user's emotional state. For example, a user in a positive emotional state will see search results displayed with bright colors and large images.

[1661] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1662] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1663] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1664] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1665] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1666] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1667] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1668] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1669] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1670] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1671] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1672] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1673] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1674] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1675] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1676] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1677] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1678] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1679] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1680] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1681] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[1682] The following is further disclosed regarding the embodiments described above.

[1683] (Claim 1)

[1684] Means for receiving user input,

[1685] A means of analyzing the received input query and extracting relevant keywords and phrases,

[1686] A means of retrieving information by calling an external search engine API based on extracted keywords,

[1687] A means of filtering the acquired information based on the user's past search history and preferences,

[1688] A means of displaying filtered information to the user,

[1689] A system that includes this.

[1690] (Claim 2)

[1691] The system according to claim 1, which uses an API to search for information on the internet as an external search engine API.

[1692] (Claim 3)

[1693] The system according to claim 1, which uses a natural language processing algorithm to analyze extracted keywords and phrases.

[1694] "Example 1"

[1695] (Claim 1)

[1696] Means for receiving user input,

[1697] A means of analyzing the received input query and extracting relevant keywords and phrases,

[1698] A means of calling an external information acquisition means based on extracted keywords and collecting information,

[1699] A means of filtering the collected information based on the user's past usage history and preferences,

[1700] A means of displaying filtered information to the user,

[1701] A system that includes this.

[1702] (Claim 2)

[1703] The system according to claim 1, which uses an API to collect information on a network as a means of acquiring external information.

[1704] (Claim 3)

[1705] The system according to claim 1, which uses a natural language processing algorithm to analyze extracted keywords and phrases.

[1706] "Application Example 1"

[1707] (Claim 1)

[1708] Means for receiving user input,

[1709] A means of analyzing the received input query and extracting relevant keywords and phrases,

[1710] A means of retrieving information by calling an external search engine API based on extracted keywords,

[1711] A means of filtering the acquired information based on the user's past search history and preferences,

[1712] A means of displaying filtered information to the user,

[1713] A means of capturing video through smart glasses and performing object recognition,

[1714] A means for automatically generating search queries based on recognized objects,

[1715] A means of displaying information in real time on smart glasses,

[1716] A system that includes this.

[1717] (Claim 2)

[1718] The system according to claim 1, which uses an API to search for information on the internet as an external search engine API.

[1719] (Claim 3)

[1720] The system according to claim 1, which uses a natural language processing algorithm to analyze extracted keywords and phrases.

[1721] "Example 2 of combining an emotion engine"

[1722] (Claim 1)

[1723] Means for receiving user input,

[1724] A means of analyzing the received input query and extracting relevant keywords and phrases,

[1725] A means of recognizing user emotions and acquiring emotional information,

[1726] A means of calling a search engine API based on extracted keywords and acquired sentiment information to retrieve information,

[1727] A means of filtering the acquired information based on the user's past search history and preferences, and further optimizing it according to sentiment information,

[1728] A means for displaying filtered information to the user and adjusting the display method according to emotional information,

[1729] A system that includes this.

[1730] (Claim 2)

[1731] The system according to claim 1, which uses an API to search for information on the internet as an external search engine API.

[1732] (Claim 3)

[1733] The system according to claim 1, which uses audio and video analysis techniques for analyzing emotional information.

[1734] "Application example 2 when combining with an emotional engine"

[1735] (Claim 1)

[1736] Means for receiving user input,

[1737] A means of analyzing the received input query and extracting relevant keywords and phrases,

[1738] A means of retrieving information by calling an external search engine API based on extracted keywords,

[1739] A means of filtering the acquired information based on the user's past search history and preferences,

[1740] A method for obtaining emotional information by analyzing the user's facial expressions and voice tone,

[1741] A means to further adjust the filtering of search results based on that sentiment information,

[1742] A means of displaying filtered information to the user,

[1743] A system that includes this.

[1744] (Claim 2)

[1745] The system according to claim 1, which uses an API to search for information on the internet as an external search engine API.

[1746] (Claim 3)

[1747] The system according to claim 1, which uses a natural language processing algorithm to analyze extracted keywords and phrases. [Explanation of Symbols]

[1748] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. Means for receiving user input, A means of analyzing the received input query and extracting relevant keywords and phrases, A means of retrieving information by calling an external search engine API based on extracted keywords, A means of filtering the acquired information based on the user's past search history and preferences, A means of displaying filtered information to the user, A system that includes this.

2. The system according to claim 1, which uses an API to search for information on the internet as an external search engine API.

3. The system according to claim 1, which uses a natural language processing algorithm to analyze extracted keywords and phrases.

Images