system

Abstract

We provide the system. [Solution] A means of locally recording information about web pages visited by user actions, A means for running a generative model that analyzes user interests using recorded information, A means of dynamically optimizing the display of content on a webpage based on analyzed interests, A means of personalizing and displaying ads on a webpage based on analyzed interests, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] With the spread of Internet use, the provision of personalized information and advertisements according to individual interests of users has become increasingly important. In conventional methods, it is often necessary to collect personal information of users and perform analysis on an external server, raising concerns about privacy issues and the risk of data leakage. In addition, in order to improve the accuracy of content provision according to users' interests and enhance the advertising effect, a complex system is required, which poses a problem of high cost for introduction and operation.

Means for Solving the Problems

[0005] This invention provides a method for integrating a lightweight generative model into a web browser to locally record and analyze a user's webpage visit history. This allows for the identification of user interests and the optimization of website content in real time. Furthermore, it can improve advertising effectiveness by personalizing advertisements based on the analysis results and displaying the most relevant ads to the user. This entire process is completed within the user's device, protecting privacy by eliminating the transmission of personal information to external parties, while simultaneously improving the user experience.

[0006] A "user" is an individual who accesses the internet via a computer network using a web browser and utilizes information and services.

[0007] A "web page" is a unit of document or resource containing information or data that is publicly available on the internet, and users typically view it through a web browser.

[0008] "Local" is a term that indicates that data and processing take place within the user's device and do not depend on external networks or servers.

[0009] "Means of recording" refers to methods and devices for collecting and storing information and history of web pages accessed by users.

[0010] A "generative model" is a mathematical model or algorithm used to analyze data and generate new information or results.

[0011] "Means of analysis" refers to methods and devices used to derive user interests and trends based on collected data.

[0012] "Optimization methods" refer to methods and devices that dynamically adjust the displayed content in order to improve the user experience.

[0013] Personalization refers to adjusting the content of information and services to best suit a specific user based on their individual preferences and interests. [Brief explanation of the drawing]

[0014] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14]It is a sequence diagram showing the processing flow of a data processing system in Application Example 2 when a sentiment engine is combined.

Embodiments for Carrying out the Invention

[0015] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] This invention is a system that uses a lightweight generative model operating within a browser to improve the user's web browsing experience. By completing all data processing on the device, it achieves personalized information delivery while protecting privacy.

[0036] The system has the functionality to record data on the user's device regarding the web pages they visit. This includes the URL of the visited web page, the links clicked, and the duration of the visit. It also collects user activity logs and stores this information locally in real time.

[0037] The generative model running on the device analyzes the recorded data to extract user interests. This analysis allows for the identification of characteristic content categories. For example, if a user frequently visits technology-related pages, this category is recognized as a primary area of ​​interest.

[0038] Based on the analysis results, the device personalizes the visited web pages in real time. Based on the user's interests, it adjusts the placement of news and articles on the visited web pages, prioritizing the display of highly relevant content. Furthermore, ad blocking is dynamically adjusted to display ads that best match the user's interests.

[0039] For example, when a user visits an online news platform, the system uses previously analyzed user interest information to prioritize relevant news articles and features. Simultaneously, the displayed advertisements are optimized based on the user's interests, improving advertising effectiveness.

[0040] Because this system performs all processing on the user's terminal, there is no need to transmit personal information externally, enabling efficient information provision while ensuring user privacy. This approach significantly improves the user experience and enables a more enriching browsing experience.

[0041] The following describes the processing flow.

[0042] Step 1:

[0043] Users access web pages via the internet.

[0044] Step 2:

[0045] The device records the URLs of accessed web pages, the links clicked, and the duration of visits. This data is stored in local storage.

[0046] Step 3:

[0047] The device analyzes data from local storage at regular intervals and runs a generative model. This model identifies the user's interests and extracts the main interest categories.

[0048] Step 4:

[0049] When a user visits a new webpage, the device optimizes the page content based on extracted interest categories. Specifically, it prioritizes displaying articles and information on topics of high interest to the user.

[0050] Step 5:

[0051] The device analyzes ad blockers on web pages, selects ads that match the user's interests, and adjusts its display to prioritize them.

[0052] Step 6:

[0053] When new data is generated through user actions, the device continues to collect and analyze the data to update the interest categories. This improves the accuracy of the analysis.

[0054] (Example 1)

[0055] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0056] In the modern information and communication field, providing personalized content and advertising while protecting user privacy is a critical challenge. However, traditional technologies commonly involved sending data to external servers for analysis at the expense of user privacy, which needed to be addressed. Furthermore, the lack of dynamic content tailoring based on user interests meant that the user experience was not sufficiently improved.

[0057] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0058] In this invention, the server includes means for internally recording data of information sources identified by user operations, means for operating a data processing device that analyzes user interests using the recorded data, means for dynamically optimizing information within the information sources based on the analyzed interests, and means for displaying personalized advertisements within the information sources based on the analyzed interests. This enables efficient personalized information provision and advertisement display while protecting user privacy.

[0059] An "information and communication device" is an electronic device that has the function of acquiring necessary data from information sources accessed by the user and recording it internally.

[0060] "User" refers to an individual who uses information and communication devices to access information sources and view content and advertisements.

[0061] An "information source" is a digital content or platform from which a user obtains information, such as a web page or application.

[0062] A "data processing device" is a computer program or hardware configuration that operates to analyze recorded data and extract user interests.

[0063] "Interest categories" refer to classifications identified by data processing devices that indicate specific categories or genres that a user is interested in.

[0064] "Personalization" is the process of tailoring the display of information and advertisements based on user interests, with the aim of providing a more relevant experience.

[0065] "Priority" refers to the criteria used to give more importance to information and advertisements related to specific interest groups than other factors, thereby ranking them higher in search results.

[0066] "Placement" refers to the design and adjustment process that determines how content and advertisements appear within an information source.

[0067] This invention is a technology that personalizes a user's web browsing experience using a system with information and communication devices. This system protects privacy by completing all data processing on the terminal rather than on a server. Specifically, it records web page data internally in response to user actions and uses that data to run a generated AI model for analysis.

[0068] The device utilizes browser functions such as IndexedDB and Web Storage API to record data such as the URLs of visited information sources, visit times, and clicked links using web browser extensions and web applications. This recorded data is analyzed by a generative AI model to extract the user's interests. The AI ​​model uses machine learning techniques to identify characteristic interest categories. For example, if high frequency of access to technology-related articles is observed, this is identified as a major area of ​​interest.

[0069] Using the analysis results, the device dynamically optimizes the internal content of the information source by manipulating the DOM with JavaScript®, adjusting the display of information and advertisements based on interests. For example, when a user visits a shopping site, gadget-related products are displayed at the top based on past activity logs, and advertisements are similarly personalized.

[0070] As a result, users will see more relevant content and advertisements while maintaining their privacy, leading to a more efficient and comfortable browsing experience. An example of a prompt might be: "Show how to analyze the user's browsing history, identify categories of interest, and prioritize displaying content related to those categories."

[0071] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0072] Step 1:

[0073] The device records data about the information sources visited each time a user visits a webpage. Inputs include the URLs accessed, links clicked, and visit duration. This data is stored locally using IndexedDB or the Web Storage API. Output is the visit record stored in the database on the device.

[0074] Step 2:

[0075] The device inputs recorded data into a generating AI model to analyze user interests. Specifically, log data such as URLs and click patterns are passed to the AI ​​model as input. The AI ​​model analyzes the data using machine learning algorithms and extracts characteristic interest categories. The output is categories indicating user interests and relevance scores.

[0076] Step 3:

[0077] The device dynamically optimizes the information on the visited webpage based on the analysis results. The input is the user's interest categories obtained from the AI ​​model. By manipulating the DOM using JavaScript, the layout of content on the page is adjusted. Specifically, it changes the order of news articles and product lists to prioritize the display of highly relevant information. The output is the optimized layout of the information sources.

[0078] Step 4:

[0079] The device personalizes and displays ads based on analyzed interests. The input is also user interest data. It provides appropriate ad selection information to the ad script and filters out irrelevant ads. The output is the display of ads that match the user's interests. Specifically, ads for products and services that the user is interested in are displayed preferentially.

[0080] (Application Example 1)

[0081] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0082] The challenge lies in effectively delivering personalized commercial messages based on user interests while maintaining the protection of personal information. Traditional methods raise privacy concerns because data is transmitted to external servers, and it has been difficult to display commercial messages that accurately reflect user interests.

[0083] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0084] In this invention, the server includes means for locally recording information of information resources visited by user operations, means for operating a generation algorithm that analyzes the user's interests using the recorded information, and means for personalizing and displaying commercial messages within the information resources based on the analyzed interests. This makes it possible to effectively provide highly relevant commercial messages based on the user's interests while maintaining the user's privacy.

[0085] "User actions" refer to actions performed by users on information resources, such as browsing, selecting, and clicking.

[0086] "Information resources" refer to digital media that provide data and content, such as websites and online platforms.

[0087] "Local recording methods" refer to mechanisms that store data directly on the user's device without sending it to an external server.

[0088] A "generative algorithm" is a computational method used to analyze collected data and infer user interests and preferences.

[0089] "Dynamic optimization" refers to the process of adjusting and optimizing the placement and display of digital content in real time according to user interests.

[0090] A "commercial message" is advertising information intended to promote products or provide information about services.

[0091] "Personalized display methods" refer to methods of selecting and adjusting the content of commercial messages based on the individual interests and preferences of each user.

[0092] To implement this invention, several software modules are required on the user's terminal. The user's terminal locally records information about the information resources visited by the user's operations. Specifically, a web browser extension collects the URLs visited by the user, the links clicked, and the time spent on each site, and stores this information on the terminal.

[0093] Next, a generation algorithm running within the device analyzes the user's interests using the collected information. This uses a lightweight machine learning library such as TENSORFLOW® Lite. This algorithm estimates characteristic areas of interest based on data from the information resources the user has visited. For example, if a user frequently visits technology-related pages, technology-related content will be recognized as their primary area of ​​interest.

[0094] Based on these results, the server dynamically optimizes and personalizes the digital content and commercial messages within the information resources. In this process, it retrieves highly relevant commercial messages from services such as the Google® Ads API and outputs the most relevant content based on the user's interests. For example, if the analysis reveals that the user is interested in smart home devices, the next time they browse, commercial messages related to smart homes will be displayed.

[0095] A concrete example of a prompt message would be, "Based on the themes of the web pages the user has recently visited, infer their interests and select the most relevant ad category." This allows for the delivery of personalized commercial messages in real time while protecting user privacy.

[0096] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0097] Step 1:

[0098] The user's device collects information about the information resources it visits. This input includes the URLs the user is viewing, the links they clicked, and the time spent on each resource. This data is recorded in the device's local storage and retained for later analysis.

[0099] Step 2:

[0100] The device runs the generative AI model using locally stored data. The input data used is the visit history and click data collected in Step 1. The generative AI model analyzes these data to identify the user's areas of interest as output. Specifically, it uses TensorFlow Lite to analyze data patterns and extract characteristic categories.

[0101] Step 3:

[0102] The server optimizes the content within the information resource based on the analyzed user interests. Using the output of the generative AI model, it identifies relevant digital content and dynamically adjusts its display placement. The input is the analysis results from step 2.

[0103] Step 4:

[0104] The server uses external service APIs to retrieve relevant commercial messages in order to personalize them according to the user's interests. The input is the user's area of ​​interest information. Based on this input information, the server queries the API, retrieves the commercial messages, and sends and displays them on the user's device.

[0105] Step 5:

[0106] The user's device integrates and displays commercial messages retrieved from the server with analyzed content. The final output is optimized content and commercial messages displayed on the device in a user-friendly format. The device dynamically updates the UI to improve the user experience.

[0107] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0108] This invention is a system designed to improve the user's web browsing experience, combining a sentiment engine with traditional personalization features. The system locally records information about the web pages the user visits and analyzes the user's interests by running a generative model.

[0109] Furthermore, this system incorporates an emotion engine that monitors the user's facial expressions and voice in real time. This emotion engine detects the user's emotional state through the camera and microphone and feeds that information back into the generative model. This makes it possible to display content and advertisements based on the user's emotions.

[0110] The device can dynamically adjust the content on a webpage based on this data. Specifically, if the user is excited, the webpage content will be optimized to display news and advertisements on more active themes. Conversely, if the user is relaxed, a calming design and colors will be applied to provide content that matches the user's mood.

[0111] For example, when a user visits an e-commerce site, if the emotion engine detects that the user is in a happy state, the device may prioritize displaying discount information on products the user likes or change the page to a color scheme that evokes happiness. This significantly improves the user experience and contributes to improved conversion rates.

[0112] This entire process is completed on the user's device, enabling personalization that takes into account the user's interests and emotions while protecting their privacy.

[0113] The following describes the processing flow.

[0114] Step 1:

[0115] A user accesses a web page through a web browser.

[0116] Step 2:

[0117] The device locally records data such as the URLs of the web pages the user visited, the links clicked, and the time spent on each page.

[0118] Step 3:

[0119] The device uses a camera and microphone to monitor the user's facial expressions and voice in real time, and an emotion engine analyzes the user's emotional state.

[0120] Step 4:

[0121] The generative model identifies user interests and emotions based on recorded web page data and sentiment analysis results.

[0122] Step 5:

[0123] The device dynamically adjusts the content and advertisements displayed on visited web pages based on the user's interests and emotions. For example, if the user is excited, it prioritizes displaying content on more active themes.

[0124] Step 6:

[0125] The device also adjusts the colors and themes of web pages based on the user's emotions, optimizing the overall user experience.

[0126] Step 7:

[0127] As new data is generated from user actions and emotional changes, the device continuously analyzes this data in real time and uses it to optimize content for the next visit.

[0128] (Example 2)

[0129] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0130] In today's digital environment, there is an urgent need to develop information delivery methods optimized for individual users, yet traditional methods have struggled to consider users' emotional states. This has resulted in limited user experiences and an inability to achieve a high degree of personalization.

[0131] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0132] In this invention, the server includes means for locally recording information of information sources visited by the user's operation, means for performing generative calculations to analyze the user's interests using the recorded information, and means for detecting the user's emotional state using a device that recognizes the user's facial expressions and voice. This enables advanced personalization based on the user's interests and emotions.

[0133] "Information source" is a term that refers to websites and digital content that users access.

[0134] "Local recording" refers to writing and saving information within the user's device.

[0135] "Generative computation" refers to the process of using AI technology to analyze data and identify user interests and trends.

[0136] "Personalization" refers to adjusting the content to suit the individual characteristics of each user.

[0137] A "device that recognizes facial expressions and voice" refers to a device that uses sensors such as cameras and microphones to detect the emotional state of a user.

[0138] "Emotional state" refers to information that indicates the user's psychological or emotional condition.

[0139] "Personalization" refers to providing information and services optimized for individual users.

[0140] This invention is a system for improving the user experience when a user visits an information source. The terminal locally records information about the information source visited by the user. This information includes the URL of the visited page, the time spent viewing it, and the page's metadata. Based on the collected data, the terminal uses a generative AI model to analyze the user's interests. The generative AI model processes this data to identify categories and keywords that the user is likely to be interested in.

[0141] Furthermore, the device recognizes the user's facial expressions and voice in real time through devices such as cameras and microphones, and detects their emotional state. The emotion engine feeds this information back into the generating AI model, providing information appropriate to the user's state.

[0142] For example, when a user visits an e-commerce site, if the emotion engine determines that the user is happy, the device will highlight discounts and promotions on preferred products. It can also change the page design to use colors that evoke happiness.

[0143] A concrete example of a prompt message would be, "Generate entertainment news recommendations for when the user is excited." In this way, a personalized experience based on the user's interests and emotions can be provided.

[0144] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0145] Step 1:

[0146] The device locally records information about the information sources the user visits. This includes the URL of the information source, the time spent viewing it, and page metadata. This data serves as input for subsequent analysis processes.

[0147] Step 2:

[0148] The device inputs recorded data into a generative AI model to analyze the user's interests. The generative AI model processes the content of visited information sources and past browsing history to identify categories and keywords that the user is likely to be interested in. As a result of this analysis, information related to the user's interests is output.

[0149] Step 3:

[0150] The device monitors the user's facial expressions and voice in real time through its camera and microphone. This allows it to detect the user's emotional state. The emotion engine analyzes this data to identify the user's current psychological state (e.g., joy, excitement, relaxation).

[0151] Step 4:

[0152] The device dynamically adjusts the content within the information source based on analyzed interests and emotional states. It comprehensively analyzes interest and emotional information to achieve a higher level of personalization. For example, when a user is excited, it displays information on topics relevant to their current interests.

[0153] Step 5:

[0154] The device delivers optimized, personalized content to the user. This includes personalized information and advertisements, achieving a high level of personalization based on the user's emotions and interests. This output improves the user experience and maximizes the effectiveness of the information source.

[0155] (Application Example 2)

[0156] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0157] In recent years, as personalization of information in users' web environments has become increasingly important, there is a need for systems that can quickly grasp users' emotional states and provide optimal content accordingly. However, conventional technologies are limited to personalization based solely on user interests, and there is a challenge in providing dynamic information that responds to real-time emotional states. Furthermore, there is a need for methods to optimize the user experience while protecting privacy.

[0158] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0159] In this invention, the server includes means for locally recording data of information pages visited by user operations, means for operating a generation algorithm that analyzes user interests using the recorded data, means for dynamically optimizing the display of content within the information pages based on the analyzed interests, means for using an emotion analysis engine that analyzes the user's emotional state in real time based on audio and image data, and means for adaptively selecting information and suggestions for products to be displayed based on the emotional state. This enables the provision of detailed content based on the user's emotions, thereby improving the user experience.

[0160] A "user" is an individual or group that uses the system.

[0161] "Operation" refers to the input or instructions that a user gives to a system.

[0162] "Information page" is a term that refers to websites and online content that users visit.

[0163] "Data" refers to a collection of information such as text, images, and audio obtained from an information page.

[0164] A "generative algorithm" refers to a computational method used to analyze user interests.

[0165] "Dynamic optimization" refers to the process of updating content display based on the user's current state.

[0166] "Content display" refers to visible elements such as text, images, and videos that are displayed on an information page.

[0167] An "emotion analysis engine" is a device or system that uses voice and image data to analyze a user's emotional state.

[0168] "Real-time" refers to a situation where information about the user's current state is processed instantly.

[0169] "Adaptive selection" refers to the act of dynamically choosing the optimal option based on specific criteria or conditions.

[0170] "Personalization" refers to the process of customizing content and suggestions according to the user's specific needs and interests.

[0171] This invention is a system that personalizes and optimizes the content and advertisements on an information page based on the user's browsing history and emotional state when they visit the page.

[0172] The server locally records data about the information pages that were visited. This data includes information such as which pages the user accessed and which content they viewed for a particularly long time.

[0173] The server uses OpenCV, an open-source computational library, to analyze the user's facial expressions in real time. It also uses the Google Speech-to-Text API to extract emotional information from the user's speech. These analysis results are used to understand the user's current emotional state.

[0174] Based on this data, the server runs a generation algorithm. This algorithm generates instructions to dynamically optimize content display and advertisements based on user interests. In doing so, the server utilizes a generation AI model running on a cloud platform to dynamically generate the most appropriate information and suggestions according to the user's emotional state. An example of such a prompt message might be: "Since the user is currently expressing feelings of joy, we will display special discount campaign products."

[0175] For example, if a user displays a joyful expression while using an online shopping app, a list of products related to happiness will be prioritized based on that emotional state information and their registered interests. This allows users to have a more fulfilling browsing experience through content that matches their emotions, while also maintaining their privacy.

[0176] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0177] Step 1:

[0178] The server collects and locally records data on the information pages visited by the user. Inputs include the URLs of visited pages and browsing history, and output is recorded in the user's browsing history database. Here, HTTP requests are parsed, and page titles and metadata are also saved.

[0179] Step 2:

[0180] The server analyzes the user's facial expressions in real time using a camera. The input is image data obtained from the camera, and the output is emotional information. Specifically, it uses OpenCV and image processing techniques to extract facial features and estimate emotions such as joy and surprise.

[0181] Step 3:

[0182] The server acquires user voice data through the microphone and extracts emotional information from the voice. The input is an audio signal, and the output is the result of the emotion analysis. The Google Speech-to-Text API is used to perform speech-to-text conversion, and the emotions are analyzed from that text data.

[0183] Step 4:

[0184] The server inputs the user's current interests and emotional state into a generative AI model based on collected browsing history data and sentiment analysis results. The generative AI model then generates prompt text and determines the next content to display based on this data. The output is a prompt text, which may include content such as "Relaxation products recommended."

[0185] Step 5:

[0186] The terminal dynamically optimizes the display of information page content based on the generated prompt text. The input is the prompt text, and the output is the updated page layout. The user interface is dynamically updated to display content that matches the user's emotional state.

[0187] Step 6:

[0188] Users view optimized information pages and enjoy an improved user experience. The input is the optimized page, and the output is expected to be increased user satisfaction. In this step, user responses are incorporated again into the next data collection cycle.

[0189] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0190] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0191] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0192] [Second Embodiment]

[0193] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0194] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0195] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0196] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0197] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0198] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0199] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0200] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0201] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0202] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0203] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0204] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0205] This invention is a system that uses a lightweight generative model operating within a browser to improve the user's web browsing experience. By completing all data processing on the device, it achieves personalized information delivery while protecting privacy.

[0206] The system has the functionality to record data on the user's device regarding the web pages they visit. This includes the URL of the visited web page, the links clicked, and the duration of the visit. It also collects user activity logs and stores this information locally in real time.

[0207] The generative model running on the device analyzes the recorded data to extract user interests. This analysis allows for the identification of characteristic content categories. For example, if a user frequently visits technology-related pages, this category is recognized as a primary area of ​​interest.

[0208] Based on the analysis results, the device personalizes the visited web pages in real time. Based on the user's interests, it adjusts the placement of news and articles on the visited web pages, prioritizing the display of highly relevant content. Furthermore, ad blocking is dynamically adjusted to display ads that best match the user's interests.

[0209] For example, when a user visits an online news platform, the system uses previously analyzed user interest information to prioritize relevant news articles and features. Simultaneously, the displayed advertisements are optimized based on the user's interests, improving advertising effectiveness.

[0210] Because this system performs all processing on the user's terminal, there is no need to transmit personal information externally, enabling efficient information provision while ensuring user privacy. This approach significantly improves the user experience and enables a more enriching browsing experience.

[0211] The following describes the processing flow.

[0212] Step 1:

[0213] Users access web pages via the internet.

[0214] Step 2:

[0215] The device records the URLs of accessed web pages, the links clicked, and the duration of visits. This data is stored in local storage.

[0216] Step 3:

[0217] The device analyzes data from local storage at regular intervals and runs a generative model. This model identifies the user's interests and extracts the main interest categories.

[0218] Step 4:

[0219] When a user visits a new webpage, the device optimizes the page content based on extracted interest categories. Specifically, it prioritizes displaying articles and information on topics of high interest to the user.

[0220] Step 5:

[0221] The device analyzes ad blockers on web pages, selects ads that match the user's interests, and adjusts its display to prioritize them.

[0222] Step 6:

[0223] When new data is generated through user actions, the device continues to collect and analyze the data to update the interest categories. This improves the accuracy of the analysis.

[0224] (Example 1)

[0225] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0226] In the modern information and communication field, providing personalized content and advertising while protecting user privacy is a critical challenge. However, traditional technologies commonly involved sending data to external servers for analysis at the expense of user privacy, which needed to be addressed. Furthermore, the lack of dynamic content tailoring based on user interests meant that the user experience was not sufficiently improved.

[0227] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0228] In this invention, the server includes means for internally recording data of information sources identified by user operations, means for operating a data processing device that analyzes user interests using the recorded data, means for dynamically optimizing information within the information sources based on the analyzed interests, and means for displaying personalized advertisements within the information sources based on the analyzed interests. This enables efficient personalized information provision and advertisement display while protecting user privacy.

[0229] An "information and communication device" is an electronic device that has the function of acquiring necessary data from information sources accessed by the user and recording it internally.

[0230] "User" refers to an individual who uses information and communication devices to access information sources and view content and advertisements.

[0231] An "information source" is a digital content or platform from which a user obtains information, such as a web page or application.

[0232] A "data processing device" is a computer program or hardware configuration that operates to analyze recorded data and extract user interests.

[0233] "Interest categories" refer to classifications identified by data processing devices that indicate specific categories or genres that a user is interested in.

[0234] "Personalization" is the process of tailoring the display of information and advertisements based on user interests, with the aim of providing a more relevant experience.

[0235] "Priority" refers to the criteria used to give more importance to information and advertisements related to specific interest groups than other factors, thereby ranking them higher in search results.

[0236] "Placement" refers to the design and adjustment process that determines how content and advertisements appear within an information source.

[0237] This invention is a technology that personalizes a user's web browsing experience using a system with information and communication devices. This system protects privacy by completing all data processing on the terminal rather than on a server. Specifically, it records web page data internally in response to user actions and uses that data to run a generated AI model for analysis.

[0238] The device utilizes browser functions such as IndexedDB and Web Storage API to record data such as the URLs of visited information sources, visit times, and clicked links using web browser extensions and web applications. This recorded data is analyzed by a generative AI model to extract the user's interests. The AI ​​model uses machine learning techniques to identify characteristic interest categories. For example, if high frequency of access to technology-related articles is observed, this is identified as a major area of ​​interest.

[0239] Using the analysis results, the device dynamically optimizes the internal content of the information source by manipulating the DOM with JavaScript, adjusting the display of information and advertisements based on interests. For example, when a user visits a shopping site, gadget-related products are displayed at the top based on past activity logs, and advertisements are similarly personalized.

[0240] As a result, users will see more relevant content and advertisements while maintaining their privacy, leading to a more efficient and comfortable browsing experience. An example of a prompt might be: "Show how to analyze the user's browsing history, identify categories of interest, and prioritize displaying content related to those categories."

[0241] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0242] Step 1:

[0243] The device records data about the information sources visited each time a user visits a webpage. Inputs include the URLs accessed, links clicked, and visit duration. This data is stored locally using IndexedDB or the Web Storage API. Output is the visit record stored in the database on the device.

[0244] Step 2:

[0245] The device inputs recorded data into a generating AI model to analyze user interests. Specifically, log data such as URLs and click patterns are passed to the AI ​​model as input. The AI ​​model analyzes the data using machine learning algorithms and extracts characteristic interest categories. The output is categories indicating user interests and relevance scores.

[0246] Step 3:

[0247] The device dynamically optimizes the information on the visited webpage based on the analysis results. The input is the user's interest categories obtained from the AI ​​model. By manipulating the DOM using JavaScript, the layout of content on the page is adjusted. Specifically, it changes the order of news articles and product lists to prioritize the display of highly relevant information. The output is the optimized layout of the information sources.

[0248] Step 4:

[0249] The device personalizes and displays ads based on analyzed interests. The input is also user interest data. It provides appropriate ad selection information to the ad script and filters out irrelevant ads. The output is the display of ads that match the user's interests. Specifically, ads for products and services that the user is interested in are displayed preferentially.

[0250] (Application Example 1)

[0251] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0252] The challenge lies in effectively delivering personalized commercial messages based on user interests while maintaining the protection of personal information. Traditional methods raise privacy concerns because data is transmitted to external servers, and it has been difficult to display commercial messages that accurately reflect user interests.

[0253] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0254] In this invention, the server includes means for locally recording information of information resources visited by user operations, means for operating a generation algorithm that analyzes the user's interests using the recorded information, and means for personalizing and displaying commercial messages within the information resources based on the analyzed interests. This makes it possible to effectively provide highly relevant commercial messages based on the user's interests while maintaining the user's privacy.

[0255] "User actions" refer to actions performed by users on information resources, such as browsing, selecting, and clicking.

[0256] "Information resources" refer to digital media that provide data and content, such as websites and online platforms.

[0257] "Local recording methods" refer to mechanisms that store data directly on the user's device without sending it to an external server.

[0258] A "generative algorithm" is a computational method used to analyze collected data and infer user interests and preferences.

[0259] "Dynamic optimization" refers to the process of adjusting and optimizing the placement and display of digital content in real time according to user interests.

[0260] A "commercial message" is advertising information intended to promote products or provide information about services.

[0261] "Personalized display methods" refer to methods of selecting and adjusting the content of commercial messages based on the individual interests and preferences of each user.

[0262] To implement this invention, several software modules are required on the user's terminal. The user's terminal locally records information about the information resources visited by the user's operations. Specifically, a web browser extension collects the URLs visited by the user, the links clicked, and the time spent on each site, and stores this information on the terminal.

[0263] Next, a generative algorithm running within the device analyzes the user's interests using the collected information. This uses a lightweight machine learning library such as TensorFlow Lite. Based on the data from the information resources the user has visited, this algorithm estimates characteristic areas of interest. For example, if a user frequently visits technology-related pages, technology-related content will be recognized as their primary area of ​​interest.

[0264] Based on these results, the server dynamically optimizes and personalizes the digital content and commercial messages within the information resources. In this process, it retrieves highly relevant commercial messages from services such as the Google Ads API and outputs the most relevant content based on the user's interests. For example, if the analysis reveals that the user is interested in smart home devices, the next time they browse, commercial messages related to smart homes will be displayed.

[0265] A concrete example of a prompt message would be, "Based on the themes of the web pages the user has recently visited, infer their interests and select the most relevant ad category." This allows for the delivery of personalized commercial messages in real time while protecting user privacy.

[0266] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0267] Step 1:

[0268] The user's device collects information about the information resources it visits. This input includes the URLs the user is viewing, the links they clicked, and the time spent on each resource. This data is recorded in the device's local storage and retained for later analysis.

[0269] Step 2:

[0270] The device runs the generative AI model using locally stored data. The input data used is the visit history and click data collected in Step 1. The generative AI model analyzes these data to identify the user's areas of interest as output. Specifically, it uses TensorFlow Lite to analyze data patterns and extract characteristic categories.

[0271] Step 3:

[0272] The server optimizes the content within the information resource based on the analyzed user interests. Using the output of the generative AI model, it identifies relevant digital content and dynamically adjusts its display placement. The input is the analysis results from step 2.

[0273] Step 4:

[0274] The server uses external service APIs to retrieve relevant commercial messages in order to personalize them according to the user's interests. The input is the user's area of ​​interest information. Based on this input information, the server queries the API, retrieves the commercial messages, and sends and displays them on the user's device.

[0275] Step 5:

[0276] The user's device integrates and displays commercial messages retrieved from the server with analyzed content. The final output is optimized content and commercial messages displayed on the device in a user-friendly format. The device dynamically updates the UI to improve the user experience.

[0277] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0278] This invention is a system designed to improve the user's web browsing experience, combining a sentiment engine with traditional personalization features. The system locally records information about the web pages the user visits and analyzes the user's interests by running a generative model.

[0279] Furthermore, this system incorporates an emotion engine that monitors the user's facial expressions and voice in real time. This emotion engine detects the user's emotional state through a camera and a microphone, and feeds the information back to the generation model. This enables the display of content and advertisements based on the user's emotions.

[0280] Based on this data, the terminal can dynamically adjust the content within the web page. Specifically, if the user is in an excited state, the content of the web page is optimized to display news and advertisements with more lively themes. Also, when the user is relaxed, a calm design and color are applied to provide content that suits the user's mood.

[0281] For example, when the user visits an e-commerce site, if the emotion engine can analyze that the user is in a happy state, the terminal can place discount information for the user's favorite products prominently or change the page to a color tone that evokes happiness. This significantly improves the user experience and also contributes to the improvement of the conversion rate.

[0282] All of this series of processes are completed on the user's terminal, enabling personalization that takes into account the user's interests and emotions while protecting privacy.

[0283] The following explains the process flow.

[0284] Step 1:

[0285] The user accesses a web page through a web browser.

[0286] Step 2:

[0287] The terminal locally records data such as the URL of the web page visited by the user, the links clicked, and the visit time.

[0288] Step 3:

[0289] The device uses a camera and microphone to monitor the user's facial expressions and voice in real time, and an emotion engine analyzes the user's emotional state.

[0290] Step 4:

[0291] The generative model identifies user interests and emotions based on recorded web page data and sentiment analysis results.

[0292] Step 5:

[0293] The device dynamically adjusts the content and advertisements displayed on visited web pages based on the user's interests and emotions. For example, if the user is excited, it prioritizes displaying content on more active themes.

[0294] Step 6:

[0295] The device also adjusts the colors and themes of web pages based on the user's emotions, optimizing the overall user experience.

[0296] Step 7:

[0297] As new data is generated from user actions and emotional changes, the device continuously analyzes this data in real time and uses it to optimize content for the next visit.

[0298] (Example 2)

[0299] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0300] In today's digital environment, there is an urgent need to develop information delivery methods optimized for individual users, yet traditional methods have struggled to consider users' emotional states. This has resulted in limited user experiences and an inability to achieve a high degree of personalization.

[0301] The specific processing by the specific processing unit 290 of the data processing device 12 according to Example 2 is realized by the following means.

[0302] In this invention, the server includes means for locally recording information of an information source accessed by a user's operation, means for operating a generation operation for analyzing the user's interests using the recorded information, and means for detecting the emotional state of the user using a device for recognizing the user's facial expressions and voice. As a result, advanced personalization based on the user's interests and emotions becomes possible.

[0303] The "information source" is a term referring to a website or digital content accessed by the user.

[0304] "Locally recording" means writing and storing information inside the user's terminal.

[0305] The "generation operation" refers to a process for analyzing data using AI technology to identify the user's interests and tendencies.

[0306] "Personalization" means adjusting the content according to the characteristics of each user.

[0307] The "device for recognizing facial expressions and voice" refers to a device for detecting the emotional state of the user using sensors such as a camera and a microphone.

[0308] The "emotional state" refers to information indicating the psychological or emotional situation of the user.

[0309] "Personalization" means providing information and services optimized for individual users.

[0310] This invention is a system for improving the user experience when a user visits an information source. The terminal locally records information about the information source visited by the user. This information includes the URL of the visited page, the time spent viewing it, and the page's metadata. Based on the collected data, the terminal uses a generative AI model to analyze the user's interests. The generative AI model processes this data to identify categories and keywords that the user is likely to be interested in.

[0311] Furthermore, the device recognizes the user's facial expressions and voice in real time through devices such as cameras and microphones, and detects their emotional state. The emotion engine feeds this information back into the generating AI model, providing information appropriate to the user's state.

[0312] For example, when a user visits an e-commerce site, if the emotion engine determines that the user is happy, the device will highlight discounts and promotions on preferred products. It can also change the page design to use colors that evoke happiness.

[0313] A concrete example of a prompt message would be, "Generate entertainment news recommendations for when the user is excited." In this way, a personalized experience based on the user's interests and emotions can be provided.

[0314] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0315] Step 1:

[0316] The device locally records information about the information sources the user visits. This includes the URL of the information source, the time spent viewing it, and page metadata. This data serves as input for subsequent analysis processes.

[0317] Step 2:

[0318] The device inputs recorded data into a generative AI model to analyze the user's interests. The generative AI model processes the content of visited information sources and past browsing history to identify categories and keywords that the user is likely to be interested in. As a result of this analysis, information related to the user's interests is output.

[0319] Step 3:

[0320] The device monitors the user's facial expressions and voice in real time through its camera and microphone. This allows it to detect the user's emotional state. The emotion engine analyzes this data to identify the user's current psychological state (e.g., joy, excitement, relaxation).

[0321] Step 4:

[0322] The device dynamically adjusts the content within the information source based on analyzed interests and emotional states. It comprehensively analyzes interest and emotional information to achieve a higher level of personalization. For example, when a user is excited, it displays information on topics relevant to their current interests.

[0323] Step 5:

[0324] The device delivers optimized, personalized content to the user. This includes personalized information and advertisements, achieving a high level of personalization based on the user's emotions and interests. This output improves the user experience and maximizes the effectiveness of the information source.

[0325] (Application Example 2)

[0326] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0327] In recent years, as personalization of information in users' web environments has become increasingly important, there is a need for systems that can quickly grasp users' emotional states and provide optimal content accordingly. However, conventional technologies are limited to personalization based solely on user interests, and there is a challenge in providing dynamic information that responds to real-time emotional states. Furthermore, there is a need for methods to optimize the user experience while protecting privacy.

[0328] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0329] In this invention, the server includes means for locally recording data of information pages visited by user operations, means for operating a generation algorithm that analyzes user interests using the recorded data, means for dynamically optimizing the display of content within the information pages based on the analyzed interests, means for using an emotion analysis engine that analyzes the user's emotional state in real time based on audio and image data, and means for adaptively selecting information and suggestions for products to be displayed based on the emotional state. This enables the provision of detailed content based on the user's emotions, thereby improving the user experience.

[0330] A "user" is an individual or group that uses the system.

[0331] "Operation" refers to the input or instructions that a user gives to a system.

[0332] "Information page" is a term that refers to websites and online content that users visit.

[0333] "Data" refers to a collection of information such as text, images, and audio obtained from an information page.

[0334] A "generative algorithm" refers to a computational method used to analyze user interests.

[0335] "Dynamic optimization" refers to the process of updating content display based on the user's current state.

[0336] "Content display" refers to visible elements such as text, images, and videos that are displayed on an information page.

[0337] An "emotion analysis engine" is a device or system that uses voice and image data to analyze a user's emotional state.

[0338] "Real-time" refers to a situation where information about the user's current state is processed instantly.

[0339] "Adaptive selection" refers to the act of dynamically choosing the optimal option based on specific criteria or conditions.

[0340] "Personalization" refers to the process of customizing content and suggestions according to the user's specific needs and interests.

[0341] This invention is a system that personalizes and optimizes the content and advertisements on an information page based on the user's browsing history and emotional state when they visit the page.

[0342] The server locally records data about the information pages that were visited. This data includes information such as which pages the user accessed and which content they viewed for a particularly long time.

[0343] The server uses OpenCV, an open-source computational library, to analyze the user's facial expressions in real time. It also uses the Google Speech-to-Text API to extract emotional information from the user's speech. These analysis results are used to understand the user's current emotional state.

[0344] Based on this data, the server runs a generation algorithm. This algorithm generates instructions to dynamically optimize content display and advertisements based on user interests. In doing so, the server utilizes a generation AI model running on a cloud platform to dynamically generate the most appropriate information and suggestions according to the user's emotional state. An example of such a prompt message might be: "Since the user is currently expressing feelings of joy, we will display special discount campaign products."

[0345] For example, if a user displays a joyful expression while using an online shopping app, a list of products related to happiness will be prioritized based on that emotional state information and their registered interests. This allows users to have a more fulfilling browsing experience through content that matches their emotions, while also maintaining their privacy.

[0346] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0347] Step 1:

[0348] The server collects and locally records data on the information pages visited by the user. Inputs include the URLs of visited pages and browsing history, and output is recorded in the user's browsing history database. Here, HTTP requests are parsed, and page titles and metadata are also saved.

[0349] Step 2:

[0350] The server analyzes the user's facial expressions in real time using a camera. The input is image data obtained from the camera, and the output is emotional information. Specifically, it uses OpenCV and image processing techniques to extract facial features and estimate emotions such as joy and surprise.

[0351] Step 3:

[0352] The server acquires user voice data through the microphone and extracts emotional information from the voice. The input is an audio signal, and the output is the result of the emotion analysis. The Google Speech-to-Text API is used to perform speech-to-text conversion, and the emotions are analyzed from that text data.

[0353] Step 4:

[0354] The server inputs the user's current interests and emotional state into a generative AI model based on collected browsing history data and sentiment analysis results. The generative AI model then generates prompt text and determines the next content to display based on this data. The output is a prompt text, which may include content such as "Relaxation products recommended."

[0355] Step 5:

[0356] The terminal dynamically optimizes the display of information page content based on the generated prompt text. The input is the prompt text, and the output is the updated page layout. The user interface is dynamically updated to display content that matches the user's emotional state.

[0357] Step 6:

[0358] Users view optimized information pages and enjoy an improved user experience. The input is the optimized page, and the output is expected to be increased user satisfaction. In this step, user responses are incorporated again into the next data collection cycle.

[0359] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0360] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0361] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0362] [Third Embodiment]

[0363] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0364] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0365] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0366] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0367] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0368] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0369] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0370] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0371] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0372] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0373] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0374] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0375] This invention is a system that uses a lightweight generative model operating within a browser to improve the user's web browsing experience. By completing all data processing on the device, it achieves personalized information delivery while protecting privacy.

[0376] The system has the functionality to record data on the user's device regarding the web pages they visit. This includes the URL of the visited web page, the links clicked, and the duration of the visit. It also collects user activity logs and stores this information locally in real time.

[0377] The generative model running on the device analyzes the recorded data to extract user interests. This analysis allows for the identification of characteristic content categories. For example, if a user frequently visits technology-related pages, this category is recognized as a primary area of ​​interest.

[0378] Based on the analysis results, the device personalizes the visited web pages in real time. Based on the user's interests, it adjusts the placement of news and articles on the visited web pages, prioritizing the display of highly relevant content. Furthermore, ad blocking is dynamically adjusted to display ads that best match the user's interests.

[0379] For example, when a user visits an online news platform, the system uses previously analyzed user interest information to prioritize relevant news articles and features. Simultaneously, the displayed advertisements are optimized based on the user's interests, improving advertising effectiveness.

[0380] Because this system performs all processing on the user's terminal, there is no need to transmit personal information externally, enabling efficient information provision while ensuring user privacy. This approach significantly improves the user experience and enables a more enriching browsing experience.

[0381] The following describes the processing flow.

[0382] Step 1:

[0383] Users access web pages via the internet.

[0384] Step 2:

[0385] The device records the URLs of accessed web pages, the links clicked, and the duration of visits. This data is stored in local storage.

[0386] Step 3:

[0387] The device analyzes data from local storage at regular intervals and runs a generative model. This model identifies the user's interests and extracts the main interest categories.

[0388] Step 4:

[0389] When a user visits a new webpage, the device optimizes the page content based on extracted interest categories. Specifically, it prioritizes displaying articles and information on topics of high interest to the user.

[0390] Step 5:

[0391] The device analyzes ad blockers on web pages, selects ads that match the user's interests, and adjusts its display to prioritize them.

[0392] Step 6:

[0393] When new data is generated through user actions, the device continues to collect and analyze the data to update the interest categories. This improves the accuracy of the analysis.

[0394] (Example 1)

[0395] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0396] In the modern information and communication field, providing personalized content and advertising while protecting user privacy is a critical challenge. However, traditional technologies commonly involved sending data to external servers for analysis at the expense of user privacy, which needed to be addressed. Furthermore, the lack of dynamic content tailoring based on user interests meant that the user experience was not sufficiently improved.

[0397] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0398] In this invention, the server includes means for internally recording data of information sources identified by user operations, means for operating a data processing device that analyzes user interests using the recorded data, means for dynamically optimizing information within the information sources based on the analyzed interests, and means for displaying personalized advertisements within the information sources based on the analyzed interests. This enables efficient personalized information provision and advertisement display while protecting user privacy.

[0399] An "information and communication device" is an electronic device that has the function of acquiring necessary data from information sources accessed by the user and recording it internally.

[0400] "User" refers to an individual who uses information and communication devices to access information sources and view content and advertisements.

[0401] An "information source" is a digital content or platform from which a user obtains information, such as a web page or application.

[0402] A "data processing device" is a computer program or hardware configuration that operates to analyze recorded data and extract user interests.

[0403] "Interest categories" refer to classifications identified by data processing devices that indicate specific categories or genres that a user is interested in.

[0404] "Personalization" is the process of tailoring the display of information and advertisements based on user interests, with the aim of providing a more relevant experience.

[0405] "Priority" refers to the criteria used to give more importance to information and advertisements related to specific interest groups than other factors, thereby ranking them higher in search results.

[0406] "Placement" refers to the design and adjustment process that determines how content and advertisements appear within an information source.

[0407] This invention is a technology that personalizes a user's web browsing experience using a system with information and communication devices. This system protects privacy by completing all data processing on the terminal rather than on a server. Specifically, it records web page data internally in response to user actions and uses that data to run a generated AI model for analysis.

[0408] The device utilizes browser functions such as IndexedDB and Web Storage API to record data such as the URLs of visited information sources, visit times, and clicked links using web browser extensions and web applications. This recorded data is analyzed by a generative AI model to extract the user's interests. The AI ​​model uses machine learning techniques to identify characteristic interest categories. For example, if high frequency of access to technology-related articles is observed, this is identified as a major area of ​​interest.

[0409] Using the analysis results, the device dynamically optimizes the internal content of the information source by manipulating the DOM with JavaScript, adjusting the display of information and advertisements based on interests. For example, when a user visits a shopping site, gadget-related products are displayed at the top based on past activity logs, and advertisements are similarly personalized.

[0410] As a result, users will see more relevant content and advertisements while maintaining their privacy, leading to a more efficient and comfortable browsing experience. An example of a prompt might be: "Show how to analyze the user's browsing history, identify categories of interest, and prioritize displaying content related to those categories."

[0411] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0412] Step 1:

[0413] The device records data about the information sources visited each time a user visits a webpage. Inputs include the URLs accessed, links clicked, and visit duration. This data is stored locally using IndexedDB or the Web Storage API. Output is the visit record stored in the database on the device.

[0414] Step 2:

[0415] The device inputs recorded data into a generating AI model to analyze user interests. Specifically, log data such as URLs and click patterns are passed to the AI ​​model as input. The AI ​​model analyzes the data using machine learning algorithms and extracts characteristic interest categories. The output is categories indicating user interests and relevance scores.

[0416] Step 3:

[0417] The device dynamically optimizes the information on the visited webpage based on the analysis results. The input is the user's interest categories obtained from the AI ​​model. By manipulating the DOM using JavaScript, the layout of content on the page is adjusted. Specifically, it changes the order of news articles and product lists to prioritize the display of highly relevant information. The output is the optimized layout of the information sources.

[0418] Step 4:

[0419] The device personalizes and displays ads based on analyzed interests. The input is also user interest data. It provides appropriate ad selection information to the ad script and filters out irrelevant ads. The output is the display of ads that match the user's interests. Specifically, ads for products and services that the user is interested in are displayed preferentially.

[0420] (Application Example 1)

[0421] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0422] The challenge lies in effectively delivering personalized commercial messages based on user interests while maintaining the protection of personal information. Traditional methods raise privacy concerns because data is transmitted to external servers, and it has been difficult to display commercial messages that accurately reflect user interests.

[0423] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0424] In this invention, the server includes means for locally recording information of information resources visited by user operations, means for operating a generation algorithm that analyzes the user's interests using the recorded information, and means for personalizing and displaying commercial messages within the information resources based on the analyzed interests. This makes it possible to effectively provide highly relevant commercial messages based on the user's interests while maintaining the user's privacy.

[0425] "User actions" refer to actions performed by users on information resources, such as browsing, selecting, and clicking.

[0426] "Information resources" refer to digital media that provide data and content, such as websites and online platforms.

[0427] "Local recording methods" refer to mechanisms that store data directly on the user's device without sending it to an external server.

[0428] A "generative algorithm" is a computational method used to analyze collected data and infer user interests and preferences.

[0429] "Dynamic optimization" refers to the process of adjusting and optimizing the placement and display of digital content in real time according to user interests.

[0430] A "commercial message" is advertising information intended to promote products or provide information about services.

[0431] "Personalized display methods" refer to methods of selecting and adjusting the content of commercial messages based on the individual interests and preferences of each user.

[0432] To implement this invention, several software modules are required on the user's terminal. The user's terminal locally records information about the information resources visited by the user's operations. Specifically, a web browser extension collects the URLs visited by the user, the links clicked, and the time spent on each site, and stores this information on the terminal.

[0433] Next, a generative algorithm running within the device analyzes the user's interests using the collected information. This uses a lightweight machine learning library such as TensorFlow Lite. Based on the data from the information resources the user has visited, this algorithm estimates characteristic areas of interest. For example, if a user frequently visits technology-related pages, technology-related content will be recognized as their primary area of ​​interest.

[0434] Based on these results, the server dynamically optimizes and personalizes the digital content and commercial messages within the information resources. In this process, it retrieves highly relevant commercial messages from services such as the Google Ads API and outputs the most relevant content based on the user's interests. For example, if the analysis reveals that the user is interested in smart home devices, the next time they browse, commercial messages related to smart homes will be displayed.

[0435] A concrete example of a prompt message would be, "Based on the themes of the web pages the user has recently visited, infer their interests and select the most relevant ad category." This allows for the delivery of personalized commercial messages in real time while protecting user privacy.

[0436] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0437] Step 1:

[0438] The user's device collects information about the information resources it visits. This input includes the URLs the user is viewing, the links they clicked, and the time spent on each resource. This data is recorded in the device's local storage and retained for later analysis.

[0439] Step 2:

[0440] The device runs the generative AI model using locally stored data. The input data used is the visit history and click data collected in Step 1. The generative AI model analyzes these data to identify the user's areas of interest as output. Specifically, it uses TensorFlow Lite to analyze data patterns and extract characteristic categories.

[0441] Step 3:

[0442] The server optimizes the content within the information resource based on the analyzed user interests. Using the output of the generative AI model, it identifies relevant digital content and dynamically adjusts its display placement. The input is the analysis results from step 2.

[0443] Step 4:

[0444] The server uses external service APIs to retrieve relevant commercial messages in order to personalize them according to the user's interests. The input is the user's area of ​​interest information. Based on this input information, the server queries the API, retrieves the commercial messages, and sends and displays them on the user's device.

[0445] Step 5:

[0446] The user's device integrates and displays commercial messages retrieved from the server with analyzed content. The final output is optimized content and commercial messages displayed on the device in a user-friendly format. The device dynamically updates the UI to improve the user experience.

[0447] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0448] This invention is a system designed to improve the user's web browsing experience, combining a sentiment engine with traditional personalization features. The system locally records information about the web pages the user visits and analyzes the user's interests by running a generative model.

[0449] Furthermore, this system incorporates an emotion engine that monitors the user's facial expressions and voice in real time. This emotion engine detects the user's emotional state through the camera and microphone and feeds that information back into the generative model. This makes it possible to display content and advertisements based on the user's emotions.

[0450] The device can dynamically adjust the content on a webpage based on this data. Specifically, if the user is excited, the webpage content will be optimized to display news and advertisements on more active themes. Conversely, if the user is relaxed, a calming design and colors will be applied to provide content that matches the user's mood.

[0451] For example, when a user visits an e-commerce site, if the emotion engine detects that the user is in a happy state, the device may prioritize displaying discount information on products the user likes or change the page to a color scheme that evokes happiness. This significantly improves the user experience and contributes to improved conversion rates.

[0452] This entire process is completed on the user's device, enabling personalization that takes into account the user's interests and emotions while protecting their privacy.

[0453] The following describes the processing flow.

[0454] Step 1:

[0455] A user accesses a web page through a web browser.

[0456] Step 2:

[0457] The device locally records data such as the URLs of the web pages the user visited, the links clicked, and the time spent on each page.

[0458] Step 3:

[0459] The device uses a camera and microphone to monitor the user's facial expressions and voice in real time, and an emotion engine analyzes the user's emotional state.

[0460] Step 4:

[0461] The generative model identifies user interests and emotions based on recorded web page data and sentiment analysis results.

[0462] Step 5:

[0463] The device dynamically adjusts the content and advertisements displayed on visited web pages based on the user's interests and emotions. For example, if the user is excited, it prioritizes displaying content on more active themes.

[0464] Step 6:

[0465] The device also adjusts the colors and themes of web pages based on the user's emotions, optimizing the overall user experience.

[0466] Step 7:

[0467] As new data is generated from user actions and emotional changes, the device continuously analyzes this data in real time and uses it to optimize content for the next visit.

[0468] (Example 2)

[0469] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0470] In today's digital environment, there is an urgent need to develop information delivery methods optimized for individual users, yet traditional methods have struggled to consider users' emotional states. This has resulted in limited user experiences and an inability to achieve a high degree of personalization.

[0471] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0472] In this invention, the server includes means for locally recording information of information sources visited by the user's operation, means for performing generative calculations to analyze the user's interests using the recorded information, and means for detecting the user's emotional state using a device that recognizes the user's facial expressions and voice. This enables advanced personalization based on the user's interests and emotions.

[0473] "Information source" is a term that refers to websites and digital content that users access.

[0474] "Local recording" refers to writing and saving information within the user's device.

[0475] "Generative computation" refers to the process of using AI technology to analyze data and identify user interests and trends.

[0476] "Personalization" refers to adjusting the content to suit the individual characteristics of each user.

[0477] A "device that recognizes facial expressions and voice" refers to a device that uses sensors such as cameras and microphones to detect the emotional state of a user.

[0478] "Emotional state" refers to information that indicates the user's psychological or emotional condition.

[0479] "Personalization" refers to providing information and services optimized for individual users.

[0480] This invention is a system for improving the user experience when a user visits an information source. The terminal locally records information about the information source visited by the user. This information includes the URL of the visited page, the time spent viewing it, and the page's metadata. Based on the collected data, the terminal uses a generative AI model to analyze the user's interests. The generative AI model processes this data to identify categories and keywords that the user is likely to be interested in.

[0481] Furthermore, the device recognizes the user's facial expressions and voice in real time through devices such as cameras and microphones, and detects their emotional state. The emotion engine feeds this information back into the generating AI model, providing information appropriate to the user's state.

[0482] For example, when a user visits an e-commerce site, if the emotion engine determines that the user is happy, the device will highlight discounts and promotions on preferred products. It can also change the page design to use colors that evoke happiness.

[0483] A concrete example of a prompt message would be, "Generate entertainment news recommendations for when the user is excited." In this way, a personalized experience based on the user's interests and emotions can be provided.

[0484] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0485] Step 1:

[0486] The device locally records information about the information sources the user visits. This includes the URL of the information source, the time spent viewing it, and page metadata. This data serves as input for subsequent analysis processes.

[0487] Step 2:

[0488] The device inputs recorded data into a generative AI model to analyze the user's interests. The generative AI model processes the content of visited information sources and past browsing history to identify categories and keywords that the user is likely to be interested in. As a result of this analysis, information related to the user's interests is output.

[0489] Step 3:

[0490] The device monitors the user's facial expressions and voice in real time through its camera and microphone. This allows it to detect the user's emotional state. The emotion engine analyzes this data to identify the user's current psychological state (e.g., joy, excitement, relaxation).

[0491] Step 4:

[0492] The device dynamically adjusts the content within the information source based on analyzed interests and emotional states. It comprehensively analyzes interest and emotional information to achieve a higher level of personalization. For example, when a user is excited, it displays information on topics relevant to their current interests.

[0493] Step 5:

[0494] The device delivers optimized, personalized content to the user. This includes personalized information and advertisements, achieving a high level of personalization based on the user's emotions and interests. This output improves the user experience and maximizes the effectiveness of the information source.

[0495] (Application Example 2)

[0496] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0497] In recent years, as personalization of information in users' web environments has become increasingly important, there is a need for systems that can quickly grasp users' emotional states and provide optimal content accordingly. However, conventional technologies are limited to personalization based solely on user interests, and there is a challenge in providing dynamic information that responds to real-time emotional states. Furthermore, there is a need for methods to optimize the user experience while protecting privacy.

[0498] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0499] In this invention, the server includes means for locally recording data of information pages visited by user operations, means for operating a generation algorithm that analyzes user interests using the recorded data, means for dynamically optimizing the display of content within the information pages based on the analyzed interests, means for using an emotion analysis engine that analyzes the user's emotional state in real time based on audio and image data, and means for adaptively selecting information and suggestions for products to be displayed based on the emotional state. This enables the provision of detailed content based on the user's emotions, thereby improving the user experience.

[0500] A "user" is an individual or group that uses the system.

[0501] "Operation" refers to the input or instructions that a user gives to a system.

[0502] "Information page" is a term that refers to websites and online content that users visit.

[0503] "Data" refers to a collection of information such as text, images, and audio obtained from an information page.

[0504] A "generative algorithm" refers to a computational method used to analyze user interests.

[0505] "Dynamic optimization" refers to the process of updating content display based on the user's current state.

[0506] "Content display" refers to visible elements such as text, images, and videos that are displayed on an information page.

[0507] An "emotion analysis engine" is a device or system that uses voice and image data to analyze a user's emotional state.

[0508] "Real-time" refers to a situation where information about the user's current state is processed instantly.

[0509] "Adaptive selection" refers to the act of dynamically choosing the optimal option based on specific criteria or conditions.

[0510] "Personalization" refers to the process of customizing content and suggestions according to the user's specific needs and interests.

[0511] This invention is a system that personalizes and optimizes the content and advertisements on an information page based on the user's browsing history and emotional state when they visit the page.

[0512] The server locally records data about the information pages that were visited. This data includes information such as which pages the user accessed and which content they viewed for a particularly long time.

[0513] The server uses OpenCV, an open-source computational library, to analyze the user's facial expressions in real time. It also uses the Google Speech-to-Text API to extract emotional information from the user's speech. These analysis results are used to understand the user's current emotional state.

[0514] Based on this data, the server runs a generation algorithm. This algorithm generates instructions to dynamically optimize content display and advertisements based on user interests. In doing so, the server utilizes a generation AI model running on a cloud platform to dynamically generate the most appropriate information and suggestions according to the user's emotional state. An example of such a prompt message might be: "Since the user is currently expressing feelings of joy, we will display special discount campaign products."

[0515] For example, if a user displays a joyful expression while using an online shopping app, a list of products related to happiness will be prioritized based on that emotional state information and their registered interests. This allows users to have a more fulfilling browsing experience through content that matches their emotions, while also maintaining their privacy.

[0516] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0517] Step 1:

[0518] The server collects and locally records data on the information pages visited by the user. Inputs include the URLs of visited pages and browsing history, and output is recorded in the user's browsing history database. Here, HTTP requests are parsed, and page titles and metadata are also saved.

[0519] Step 2:

[0520] The server analyzes the user's facial expressions in real time using a camera. The input is image data obtained from the camera, and the output is emotional information. Specifically, it uses OpenCV and image processing techniques to extract facial features and estimate emotions such as joy and surprise.

[0521] Step 3:

[0522] The server acquires user voice data through the microphone and extracts emotional information from the voice. The input is an audio signal, and the output is the result of the emotion analysis. The Google Speech-to-Text API is used to perform speech-to-text conversion, and the emotions are analyzed from that text data.

[0523] Step 4:

[0524] The server inputs the user's current interests and emotional state into a generative AI model based on collected browsing history data and sentiment analysis results. The generative AI model then generates prompt text and determines the next content to display based on this data. The output is a prompt text, which may include content such as "Relaxation products recommended."

[0525] Step 5:

[0526] The terminal dynamically optimizes the display of information page content based on the generated prompt text. The input is the prompt text, and the output is the updated page layout. The user interface is dynamically updated to display content that matches the user's emotional state.

[0527] Step 6:

[0528] Users view optimized information pages and enjoy an improved user experience. The input is the optimized page, and the output is expected to be increased user satisfaction. In this step, user responses are incorporated again into the next data collection cycle.

[0529] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0530] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0531] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0532] [Fourth Embodiment]

[0533] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0534] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0535] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0536] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0537] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0538] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0539] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0540] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0541] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0542] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0543] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0544] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0545] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0546] This invention is a system that uses a lightweight generative model operating within a browser to improve the user's web browsing experience. By completing all data processing on the device, it achieves personalized information delivery while protecting privacy.

[0547] The system has the functionality to record data on the user's device regarding the web pages they visit. This includes the URL of the visited web page, the links clicked, and the duration of the visit. It also collects user activity logs and stores this information locally in real time.

[0548] The generative model running on the device analyzes the recorded data to extract user interests. This analysis allows for the identification of characteristic content categories. For example, if a user frequently visits technology-related pages, this category is recognized as a primary area of ​​interest.

[0549] Based on the analysis results, the device personalizes the visited web pages in real time. Based on the user's interests, it adjusts the placement of news and articles on the visited web pages, prioritizing the display of highly relevant content. Furthermore, ad blocking is dynamically adjusted to display ads that best match the user's interests.

[0550] For example, when a user visits an online news platform, the system uses previously analyzed user interest information to prioritize relevant news articles and features. Simultaneously, the displayed advertisements are optimized based on the user's interests, improving advertising effectiveness.

[0551] Because this system performs all processing on the user's terminal, there is no need to transmit personal information externally, enabling efficient information provision while ensuring user privacy. This approach significantly improves the user experience and enables a more enriching browsing experience.

[0552] The following describes the processing flow.

[0553] Step 1:

[0554] Users access web pages via the internet.

[0555] Step 2:

[0556] The device records the URLs of accessed web pages, the links clicked, and the duration of visits. This data is stored in local storage.

[0557] Step 3:

[0558] The device analyzes data from local storage at regular intervals and runs a generative model. This model identifies the user's interests and extracts the main interest categories.

[0559] Step 4:

[0560] When a user visits a new webpage, the device optimizes the page content based on extracted interest categories. Specifically, it prioritizes displaying articles and information on topics of high interest to the user.

[0561] Step 5:

[0562] The device analyzes ad blockers on web pages, selects ads that match the user's interests, and adjusts its display to prioritize them.

[0563] Step 6:

[0564] When new data is generated through user actions, the device continues to collect and analyze the data to update the interest categories. This improves the accuracy of the analysis.

[0565] (Example 1)

[0566] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0567] In the modern information and communication field, providing personalized content and advertising while protecting user privacy is a critical challenge. However, traditional technologies commonly involved sending data to external servers for analysis at the expense of user privacy, which needed to be addressed. Furthermore, the lack of dynamic content tailoring based on user interests meant that the user experience was not sufficiently improved.

[0568] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0569] In this invention, the server includes means for internally recording data of information sources identified by user operations, means for operating a data processing device that analyzes user interests using the recorded data, means for dynamically optimizing information within the information sources based on the analyzed interests, and means for displaying personalized advertisements within the information sources based on the analyzed interests. This enables efficient personalized information provision and advertisement display while protecting user privacy.

[0570] An "information and communication device" is an electronic device that has the function of acquiring necessary data from information sources accessed by the user and recording it internally.

[0571] "User" refers to an individual who uses information and communication devices to access information sources and view content and advertisements.

[0572] An "information source" is a digital content or platform from which a user obtains information, such as a web page or application.

[0573] A "data processing device" is a computer program or hardware configuration that operates to analyze recorded data and extract user interests.

[0574] "Interest categories" refer to classifications identified by data processing devices that indicate specific categories or genres that a user is interested in.

[0575] "Personalization" is the process of tailoring the display of information and advertisements based on user interests, with the aim of providing a more relevant experience.

[0576] "Priority" refers to the criteria used to give more importance to information and advertisements related to specific interest groups than other factors, thereby ranking them higher in search results.

[0577] "Placement" refers to the design and adjustment process that determines how content and advertisements appear within an information source.

[0578] This invention is a technology that personalizes a user's web browsing experience using a system with information and communication devices. This system protects privacy by completing all data processing on the terminal rather than on a server. Specifically, it records web page data internally in response to user actions and uses that data to run a generated AI model for analysis.

[0579] The device utilizes browser functions such as IndexedDB and Web Storage API to record data such as the URLs of visited information sources, visit times, and clicked links using web browser extensions and web applications. This recorded data is analyzed by a generative AI model to extract the user's interests. The AI ​​model uses machine learning techniques to identify characteristic interest categories. For example, if high frequency of access to technology-related articles is observed, this is identified as a major area of ​​interest.

[0580] Using the analysis results, the device dynamically optimizes the internal content of the information source by manipulating the DOM with JavaScript, adjusting the display of information and advertisements based on interests. For example, when a user visits a shopping site, gadget-related products are displayed at the top based on past activity logs, and advertisements are similarly personalized.

[0581] As a result, users will see more relevant content and advertisements while maintaining their privacy, leading to a more efficient and comfortable browsing experience. An example of a prompt might be: "Show how to analyze the user's browsing history, identify categories of interest, and prioritize displaying content related to those categories."

[0582] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0583] Step 1:

[0584] The device records data about the information sources visited each time a user visits a webpage. Inputs include the URLs accessed, links clicked, and visit duration. This data is stored locally using IndexedDB or the Web Storage API. Output is the visit record stored in the database on the device.

[0585] Step 2:

[0586] The device inputs recorded data into a generating AI model to analyze user interests. Specifically, log data such as URLs and click patterns are passed to the AI ​​model as input. The AI ​​model analyzes the data using machine learning algorithms and extracts characteristic interest categories. The output is categories indicating user interests and relevance scores.

[0587] Step 3:

[0588] The device dynamically optimizes the information on the visited webpage based on the analysis results. The input is the user's interest categories obtained from the AI ​​model. By manipulating the DOM using JavaScript, the layout of content on the page is adjusted. Specifically, it changes the order of news articles and product lists to prioritize the display of highly relevant information. The output is the optimized layout of the information sources.

[0589] Step 4:

[0590] The device personalizes and displays ads based on analyzed interests. The input is also user interest data. It provides appropriate ad selection information to the ad script and filters out irrelevant ads. The output is the display of ads that match the user's interests. Specifically, ads for products and services that the user is interested in are displayed preferentially.

[0591] (Application Example 1)

[0592] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0593] The challenge lies in effectively delivering personalized commercial messages based on user interests while maintaining the protection of personal information. Traditional methods raise privacy concerns because data is transmitted to external servers, and it has been difficult to display commercial messages that accurately reflect user interests.

[0594] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0595] In this invention, the server includes means for locally recording information of information resources visited by user operations, means for operating a generation algorithm that analyzes the user's interests using the recorded information, and means for personalizing and displaying commercial messages within the information resources based on the analyzed interests. This makes it possible to effectively provide highly relevant commercial messages based on the user's interests while maintaining the user's privacy.

[0596] "User actions" refer to actions performed by users on information resources, such as browsing, selecting, and clicking.

[0597] "Information resources" refer to digital media that provide data and content, such as websites and online platforms.

[0598] "Local recording methods" refer to mechanisms that store data directly on the user's device without sending it to an external server.

[0599] A "generative algorithm" is a computational method used to analyze collected data and infer user interests and preferences.

[0600] "Dynamic optimization" refers to the process of adjusting and optimizing the placement and display of digital content in real time according to user interests.

[0601] A "commercial message" is advertising information intended to promote products or provide information about services.

[0602] "Personalized display methods" refer to methods of selecting and adjusting the content of commercial messages based on the individual interests and preferences of each user.

[0603] To implement this invention, several software modules are required on the user's terminal. The user's terminal locally records information about the information resources visited by the user's operations. Specifically, a web browser extension collects the URLs visited by the user, the links clicked, and the time spent on each site, and stores this information on the terminal.

[0604] Next, a generative algorithm running within the device analyzes the user's interests using the collected information. This uses a lightweight machine learning library such as TensorFlow Lite. Based on the data from the information resources the user has visited, this algorithm estimates characteristic areas of interest. For example, if a user frequently visits technology-related pages, technology-related content will be recognized as their primary area of ​​interest.

[0605] Based on these results, the server dynamically optimizes and personalizes the digital content and commercial messages within the information resources. In this process, it retrieves highly relevant commercial messages from services such as the Google Ads API and outputs the most relevant content based on the user's interests. For example, if the analysis reveals that the user is interested in smart home devices, the next time they browse, commercial messages related to smart homes will be displayed.

[0606] A concrete example of a prompt message would be, "Based on the themes of the web pages the user has recently visited, infer their interests and select the most relevant ad category." This allows for the delivery of personalized commercial messages in real time while protecting user privacy.

[0607] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0608] Step 1:

[0609] The user's device collects information about the information resources it visits. This input includes the URLs the user is viewing, the links they clicked, and the time spent on each resource. This data is recorded in the device's local storage and retained for later analysis.

[0610] Step 2:

[0611] The device runs the generative AI model using locally stored data. The input data used is the visit history and click data collected in Step 1. The generative AI model analyzes these data to identify the user's areas of interest as output. Specifically, it uses TensorFlow Lite to analyze data patterns and extract characteristic categories.

[0612] Step 3:

[0613] The server optimizes the content within the information resource based on the analyzed user interests. Using the output of the generative AI model, it identifies relevant digital content and dynamically adjusts its display placement. The input is the analysis results from step 2.

[0614] Step 4:

[0615] The server uses external service APIs to retrieve relevant commercial messages in order to personalize them according to the user's interests. The input is the user's area of ​​interest information. Based on this input information, the server queries the API, retrieves the commercial messages, and sends and displays them on the user's device.

[0616] Step 5:

[0617] The user's device integrates and displays commercial messages retrieved from the server with analyzed content. The final output is optimized content and commercial messages displayed on the device in a user-friendly format. The device dynamically updates the UI to improve the user experience.

[0618] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0619] This invention is a system designed to improve the user's web browsing experience, combining a sentiment engine with traditional personalization features. The system locally records information about the web pages the user visits and analyzes the user's interests by running a generative model.

[0620] Furthermore, this system incorporates an emotion engine that monitors the user's facial expressions and voice in real time. This emotion engine detects the user's emotional state through the camera and microphone and feeds that information back into the generative model. This makes it possible to display content and advertisements based on the user's emotions.

[0621] The device can dynamically adjust the content on a webpage based on this data. Specifically, if the user is excited, the webpage content will be optimized to display news and advertisements on more active themes. Conversely, if the user is relaxed, a calming design and colors will be applied to provide content that matches the user's mood.

[0622] For example, when a user visits an e-commerce site, if the emotion engine detects that the user is in a happy state, the device may prioritize displaying discount information on products the user likes or change the page to a color scheme that evokes happiness. This significantly improves the user experience and contributes to improved conversion rates.

[0623] This entire process is completed on the user's device, enabling personalization that takes into account the user's interests and emotions while protecting their privacy.

[0624] The following describes the processing flow.

[0625] Step 1:

[0626] A user accesses a web page through a web browser.

[0627] Step 2:

[0628] The device locally records data such as the URLs of the web pages the user visited, the links clicked, and the time spent on each page.

[0629] Step 3:

[0630] The device uses a camera and microphone to monitor the user's facial expressions and voice in real time, and an emotion engine analyzes the user's emotional state.

[0631] Step 4:

[0632] The generative model identifies user interests and emotions based on recorded web page data and sentiment analysis results.

[0633] Step 5:

[0634] The device dynamically adjusts the content and advertisements displayed on visited web pages based on the user's interests and emotions. For example, if the user is excited, it prioritizes displaying content on more active themes.

[0635] Step 6:

[0636] The device also adjusts the colors and themes of web pages based on the user's emotions, optimizing the overall user experience.

[0637] Step 7:

[0638] As new data is generated from user actions and emotional changes, the device continuously analyzes this data in real time and uses it to optimize content for the next visit.

[0639] (Example 2)

[0640] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0641] In today's digital environment, there is an urgent need to develop information delivery methods optimized for individual users, yet traditional methods have struggled to consider users' emotional states. This has resulted in limited user experiences and an inability to achieve a high degree of personalization.

[0642] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0643] In this invention, the server includes means for locally recording information of information sources visited by the user's operation, means for performing generative calculations to analyze the user's interests using the recorded information, and means for detecting the user's emotional state using a device that recognizes the user's facial expressions and voice. This enables advanced personalization based on the user's interests and emotions.

[0644] "Information source" is a term that refers to websites and digital content that users access.

[0645] "Local recording" refers to writing and saving information within the user's device.

[0646] "Generative computation" refers to the process of using AI technology to analyze data and identify user interests and trends.

[0647] "Personalization" refers to adjusting the content to suit the individual characteristics of each user.

[0648] A "device that recognizes facial expressions and voice" refers to a device that uses sensors such as cameras and microphones to detect the emotional state of a user.

[0649] "Emotional state" refers to information that indicates the user's psychological or emotional condition.

[0650] "Personalization" refers to providing information and services optimized for individual users.

[0651] This invention is a system for improving the user experience when a user visits an information source. The terminal locally records information about the information source visited by the user. This information includes the URL of the visited page, the time spent viewing it, and the page's metadata. Based on the collected data, the terminal uses a generative AI model to analyze the user's interests. The generative AI model processes this data to identify categories and keywords that the user is likely to be interested in.

[0652] Furthermore, the device recognizes the user's facial expressions and voice in real time through devices such as cameras and microphones, and detects their emotional state. The emotion engine feeds this information back into the generating AI model, providing information appropriate to the user's state.

[0653] For example, when a user visits an e-commerce site, if the emotion engine determines that the user is happy, the device will highlight discounts and promotions on preferred products. It can also change the page design to use colors that evoke happiness.

[0654] A concrete example of a prompt message would be, "Generate entertainment news recommendations for when the user is excited." In this way, a personalized experience based on the user's interests and emotions can be provided.

[0655] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0656] Step 1:

[0657] The device locally records information about the information sources the user visits. This includes the URL of the information source, the time spent viewing it, and page metadata. This data serves as input for subsequent analysis processes.

[0658] Step 2:

[0659] The device inputs recorded data into a generative AI model to analyze the user's interests. The generative AI model processes the content of visited information sources and past browsing history to identify categories and keywords that the user is likely to be interested in. As a result of this analysis, information related to the user's interests is output.

[0660] Step 3:

[0661] The device monitors the user's facial expressions and voice in real time through its camera and microphone. This allows it to detect the user's emotional state. The emotion engine analyzes this data to identify the user's current psychological state (e.g., joy, excitement, relaxation).

[0662] Step 4:

[0663] The device dynamically adjusts the content within the information source based on analyzed interests and emotional states. It comprehensively analyzes interest and emotional information to achieve a higher level of personalization. For example, when a user is excited, it displays information on topics relevant to their current interests.

[0664] Step 5:

[0665] The device delivers optimized, personalized content to the user. This includes personalized information and advertisements, achieving a high level of personalization based on the user's emotions and interests. This output improves the user experience and maximizes the effectiveness of the information source.

[0666] (Application Example 2)

[0667] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0668] In recent years, as personalization of information in users' web environments has become increasingly important, there is a need for systems that can quickly grasp users' emotional states and provide optimal content accordingly. However, conventional technologies are limited to personalization based solely on user interests, and there is a challenge in providing dynamic information that responds to real-time emotional states. Furthermore, there is a need for methods to optimize the user experience while protecting privacy.

[0669] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0670] In this invention, the server includes means for locally recording data of information pages visited by user operations, means for operating a generation algorithm that analyzes user interests using the recorded data, means for dynamically optimizing the display of content within the information pages based on the analyzed interests, means for using an emotion analysis engine that analyzes the user's emotional state in real time based on audio and image data, and means for adaptively selecting information and suggestions for products to be displayed based on the emotional state. This enables the provision of detailed content based on the user's emotions, thereby improving the user experience.

[0671] A "user" is an individual or group that uses the system.

[0672] "Operation" refers to the input or instructions that a user gives to a system.

[0673] "Information page" is a term that refers to websites and online content that users visit.

[0674] "Data" refers to a collection of information such as text, images, and audio obtained from an information page.

[0675] A "generative algorithm" refers to a computational method used to analyze user interests.

[0676] "Dynamic optimization" refers to the process of updating content display based on the user's current state.

[0677] "Content display" refers to visible elements such as text, images, and videos that are displayed on an information page.

[0678] An "emotion analysis engine" is a device or system that uses voice and image data to analyze a user's emotional state.

[0679] "Real-time" refers to a situation where information about the user's current state is processed instantly.

[0680] "Adaptive selection" refers to the act of dynamically choosing the optimal option based on specific criteria or conditions.

[0681] "Personalization" refers to the process of customizing content and suggestions according to the user's specific needs and interests.

[0682] This invention is a system that personalizes and optimizes the content and advertisements on an information page based on the user's browsing history and emotional state when they visit the page.

[0683] The server locally records data about the information pages that were visited. This data includes information such as which pages the user accessed and which content they viewed for a particularly long time.

[0684] The server uses OpenCV, an open-source computational library, to analyze the user's facial expressions in real time. It also uses the Google Speech-to-Text API to extract emotional information from the user's speech. These analysis results are used to understand the user's current emotional state.

[0685] Based on this data, the server runs a generation algorithm. This algorithm generates instructions to dynamically optimize content display and advertisements based on user interests. In doing so, the server utilizes a generation AI model running on a cloud platform to dynamically generate the most appropriate information and suggestions according to the user's emotional state. An example of such a prompt message might be: "Since the user is currently expressing feelings of joy, we will display special discount campaign products."

[0686] For example, if a user displays a joyful expression while using an online shopping app, a list of products related to happiness will be prioritized based on that emotional state information and their registered interests. This allows users to have a more fulfilling browsing experience through content that matches their emotions, while also maintaining their privacy.

[0687] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0688] Step 1:

[0689] The server collects and locally records data on the information pages visited by the user. Inputs include the URLs of visited pages and browsing history, and output is recorded in the user's browsing history database. Here, HTTP requests are parsed, and page titles and metadata are also saved.

[0690] Step 2:

[0691] The server analyzes the user's facial expressions in real time using a camera. The input is image data obtained from the camera, and the output is emotional information. Specifically, it uses OpenCV and image processing techniques to extract facial features and estimate emotions such as joy and surprise.

[0692] Step 3:

[0693] The server acquires user voice data through the microphone and extracts emotional information from the voice. The input is an audio signal, and the output is the result of the emotion analysis. The Google Speech-to-Text API is used to perform speech-to-text conversion, and the emotions are analyzed from that text data.

[0694] Step 4:

[0695] The server inputs the user's current interests and emotional state into a generative AI model based on collected browsing history data and sentiment analysis results. The generative AI model then generates prompt text and determines the next content to display based on this data. The output is a prompt text, which may include content such as "Relaxation products recommended."

[0696] Step 5:

[0697] The terminal dynamically optimizes the display of information page content based on the generated prompt text. The input is the prompt text, and the output is the updated page layout. The user interface is dynamically updated to display content that matches the user's emotional state.

[0698] Step 6:

[0699] Users view optimized information pages and enjoy an improved user experience. The input is the optimized page, and the output is expected to be increased user satisfaction. In this step, user responses are incorporated again into the next data collection cycle.

[0700] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0701] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0702] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0703] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0704] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0705] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0706] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0707] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0708] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0709] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0710] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0711] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0712] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0713] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0714] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0715] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0716] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0717] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0718] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0719] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0720] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0721] The following is further disclosed regarding the embodiments described above.

[0722] (Claim 1)

[0723] A means of locally recording information about web pages visited by user actions,

[0724] A means for running a generative model that analyzes user interests using recorded information,

[0725] A means of dynamically optimizing the display of content on a webpage based on analyzed interests,

[0726] A means of personalizing and displaying ads on a webpage based on analyzed interests,

[0727] A system that includes this.

[0728] (Claim 2)

[0729] The system according to claim 1, further comprising means for the generative model to identify and prioritize interest categories using the user's web page visit history.

[0730] (Claim 3)

[0731] The system according to claim 1, further comprising means for dynamically adjusting the layout of websites visited based on analyzed user interests to improve the user experience.

[0732] "Example 1"

[0733] (Claim 1)

[0734] An information and communication device includes means for internally recording data of information sources identified by user operations,

[0735] A means for operating a data processing device that uses recorded data to analyze user interests,

[0736] A means for dynamically optimizing information within an information source based on analyzed interests,

[0737] A means of personalizing and displaying advertisements within an information source based on analyzed interests,

[0738] A system that includes this.

[0739] (Claim 2)

[0740] The system according to claim 1, further comprising a data processing device for identifying and prioritizing areas of interest using visit records of user information sources.

[0741] (Claim 3)

[0742] The system according to claim 1, further comprising means for dynamically adjusting the placement of visited information sources based on analyzed user interests to improve the user experience.

[0743] "Application Example 1"

[0744] (Claim 1)

[0745] A means for locally recording information about information resources visited by user actions,

[0746] A means for running a generative algorithm that analyzes user interests using recorded information,

[0747] A means for dynamically optimizing the display of digital content within information resources based on analyzed interests,

[0748] A means for individually displaying commercial messages within information resources based on analyzed interests,

[0749] A means of acquiring and adjusting the display of highly relevant commercial messages based on user interests,

[0750] A system that includes this.

[0751] (Claim 2)

[0752] The system according to claim 1, further comprising a means for the generation algorithm to identify and prioritize areas of interest using the user's browsing history of information resources.

[0753] (Claim 3)

[0754] The system according to claim 1, further comprising means for dynamically adjusting the display configuration of visited information resources based on analyzed user interests to improve the user experience.

[0755] "Example 2 of combining an emotion engine"

[0756] (Claim 1)

[0757] A means for locally recording information about information sources visited by user actions,

[0758] A means for running a generative operation that analyzes user interests using recorded information,

[0759] A means for dynamically optimizing the display of content within an information source based on analyzed interests,

[0760] A means of personalizing and displaying advertisements within an information source based on analyzed interests,

[0761] A means for detecting the user's emotional state using a device that recognizes the user's facial expressions and voice,

[0762] A means for feeding back detected emotion information to the generation calculation and adjusting the content within the content source,

[0763] A system that includes this.

[0764] (Claim 2)

[0765] The system according to claim 1, further comprising means for the generation operation to identify and prioritize interest classifications using the user's information source visit history.

[0766] (Claim 3)

[0767] The system according to claim 1, further comprising means for dynamically adjusting the composition of information sources visited based on the analyzed user interests and emotional state, thereby improving the user experience.

[0768] "Application example 2 when combining with an emotional engine"

[0769] (Claim 1)

[0770] A means of locally recording data of information pages visited by user actions,

[0771] A means for running a generative algorithm that analyzes user interests using recorded data,

[0772] A means for dynamically optimizing the display of content within an information page based on analyzed interests,

[0773] A means of displaying personalized advertisements on information pages based on analyzed interests,

[0774] A means of using an emotion analysis engine that analyzes the user's emotional state in real time based on voice and image data,

[0775] A means of adaptively selecting information and suggestions for products to be displayed based on emotional state,

[0776] A system that includes this.

[0777] (Claim 2)

[0778] The system according to claim 1, further comprising a generation algorithm for identifying and prioritizing areas of interest using the user's browsing history of information pages.

[0779] (Claim 3)

[0780] The system according to claim 1, further comprising means for dynamically adjusting the design of the information pages visited based on the analyzed emotions of the user, thereby improving the user experience. [Explanation of Symbols]

[0781] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of locally recording information about web pages visited by user actions, A means for running a generative model that analyzes user interests using recorded information, A means of dynamically optimizing the display of content on a webpage based on analyzed interests, A means of personalizing and displaying ads on a webpage based on analyzed interests, A system that includes this.

2. The system according to claim 1, further comprising means for the generative model to identify and prioritize interest categories using the user's web page visit history.

3. The system according to claim 1, further comprising means for dynamically adjusting the layout of websites visited based on analyzed user interests to improve the user experience.

Images