system

Abstract

A system is provided.SOLUTION: A system comprising: means for a user to input a URL of a service or capture an image with a camera; means for analyzing the input URL or the captured image and identifying a structure or content of the service; means for converting the service into an easy-to-see and easy-to-understand UI based on the identified information; means for allowing the user to communicate a request by voice or text; means for analyzing the request and executing an appropriate operation; and means for notifying the user of completion of the executed operation.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Today, there are a wide variety of web services and applications, but many of them have unintuitive user interfaces (UIs) that are difficult to use. This makes them difficult to use, especially for users with low IT literacy or those trying to try new services, and reduces convenience. Furthermore, because these services are difficult to understand, the steps required to achieve a goal are complicated, causing frustration for many users. Furthermore, language barriers pose a further obstacle when using services offered in a foreign language. There is a need to solve these problems and make a variety of services easily accessible to all users. [Means for solving the problem]

[0005] This invention provides a means for a user to input a service URL or take a picture of the screen with a camera, and a means for analyzing the input URL or captured image to identify the structure and content of the service. Using a means for converting the identified information into an easy-to-read and understandable UI, the system allows users to operate the service intuitively. It also provides a means for a user to communicate their requests by voice or text, and a means for analyzing the requests and executing appropriate operations. Furthermore, the system provides a means for notifying the user when the executed operations are complete. This allows users to use the service without difficulty and intuitively perform the required operations.

[0006] "User" refers to a person who uses the system, enters the service URL or takes a picture of the screen with a camera, and communicates their request by voice or text.

[0007] "URL" stands for Uniform Resource Locator and is an address that indicates the location of a specific resource (web page or file) on the Web.

[0008] A "camera" is a device installed in devices such as smartphones that captures images and videos.

[0009] "Analysis" refers to the process of identifying the structure and content based on the input URL or captured image data.

[0010] "Image analysis" refers to the techniques and processes used to process captured image data and extract text and image structure from it.

[0011] "Generative AI" refers to algorithms and systems that use artificial intelligence techniques to organize and transform data into a more readable format.

[0012] "UI" stands for User Interface and refers to the design of the screen and operation methods used when a user interacts with a system or service.

[0013] "Speech recognition" refers to the technology that converts the user's speech into digital data and understands its content.

[0014] "Natural language processing" refers to the process of using AI technology to analyze human language and understand its meaning.

[0015] An "operation" refers to a series of instructions or inputs a user makes to achieve a specific goal.

[0016] "Notification" refers to a system informing a user of the results of a particular action or process. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10]1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] First, the user installs and launches the smartphone app, which is designed to assist in the use of web services and provides users with intuitive and easy operation.

[0039] Screen Input

[0040] Users can either enter the URL of the service they want to use through the app's interface or take a screenshot of the service's screen using the app's camera. For example, if a user wants to access a product page on an overseas auction site, they can enter the URL of the product page or take a screenshot of the screen.

[0041] Data transmission and analysis

[0042] The device sends the URL entered by the user and the captured image data to a server on the cloud. The server then accesses the received URL and retrieves the HTML data. It also uses image analysis technology (such as OCR) to extract the screen structure and text information from the captured image data.

[0043] The server analyzes the acquired HTML data or image analysis results to identify the page structure and elements. This includes using generative AI to recognize and classify appropriate information, such as the product name, price, and the location of the purchase button.

[0044] UI transformation

[0045] The server then converts the UI into a format that is intuitively easy for users to understand based on the analysis results. Specifically, it highlights important information and arranges action buttons to simplify user operations. The easy-to-read UI data generated here is sent to the device and displayed to the user.

[0046] Collecting user requests

[0047] Users can view the converted UI and then specify actions by voice or text, for example, "I want to buy this product."

[0048] The device uses voice recognition technology to convert the user's voice instructions into text data, and then uses natural language processing technology to analyze the instructions. The results of this analysis are sent to the server as a specific action (e.g., purchasing procedure).

[0049] Service agency

[0050] The server generates and executes the corresponding web request based on the parsed action, which may include the user logging in with their account information, clicking a purchase button, or entering payment information.

[0051] If necessary, the server will prompt the user for additional information (e.g., shipping address or payment information). The device will notify the user of this request, prompt them to enter the required information, and then send it back to the server.

[0052] Operation completion notification

[0053] When the proxy operation is complete, the server notifies the terminal of the completion. The terminal displays the received notification to the user, informing them that all operations have been completed. For example, it displays a message such as "The purchase procedure has been completed."

[0054] This allows users to easily access the services they need without having to deal with complicated operations or an unfriendly UI. This system is particularly convenient for users with low IT literacy or who face language barriers.

[0055] The processing flow will be explained below.

[0056] Step 1:

[0057] Users launch a smartphone app and enter the URL of the web service they want to use, or take a picture of the screen using the camera. For example, consider the case where a user enters the URL of a product page on an overseas auction site.

[0058] Step 2:

[0059] The device receives the URL entered by the user or the captured image data and sends it to a server on the cloud. If a URL is entered, the URL information is sent, and if an image is captured, the image data is sent.

[0060] Step 3:

[0061] The server retrieves the HTML data of the web page based on the received URL, and if image data is sent, it uses image analysis technology to extract text information and screen structure from the image.

[0062] Step 4:

[0063] The server performs analysis using the acquired HTML data and image analysis results. Specifically, it uses generative AI to identify important elements on the page (e.g., product name, price, purchase button, etc.).

[0064] Step 5:

[0065] The server generates UI data based on the identified information, converting it into a format that is easy for the user to understand. This UI data includes highlighting important information and arranging buttons for easy user operation.

[0066] Step 6:

[0067] The server transmits the generated UI data to the terminal.

[0068] Step 7:

[0069] The device displays the received UI data to the user, allowing the user to consider their next action while looking at the displayed UI.

[0070] Step 8:

[0071] The user inputs their request by voice or text, for example, "I would like to purchase this product."

[0072] Step 9:

[0073] The device uses voice recognition technology to convert the user's voice instructions into text, and then uses natural language processing technology to analyze the text data and understand it as a specific action (e.g., checkout).

[0074] Step 10:

[0075] The device then sends the analyzed user request to the server, and as a specific action, sends a purchase procedure request to the server.

[0076] Step 11:

[0077] Based on the received action instruction, the server generates a corresponding web request and performs the required operation, for example, logging in to an auction site and initiating the process of purchasing an item.

[0078] Step 12:

[0079] The server will prompt the user for additional information as needed, for example a shipping address or payment information.

[0080] Step 13:

[0081] The terminal notifies the user of a request for additional information from the server and prompts the user to input the required information.

[0082] Step 14:

[0083] The user enters any additional information requested and transmits it to the terminal.

[0084] Step 15:

[0085] The terminal transmits the additional information entered by the user to the server.

[0086] Step 16:

[0087] The server uses the received additional information to complete any necessary operations, such as entering payment information and finally completing the purchase.

[0088] Step 17:

[0089] The server notifies the terminal that the operation is complete.

[0090] Step 18:

[0091] The device displays the received completion notification to the user to let them know that the operation has been completed, for example, by displaying a message such as "Purchase procedure completed."

[0092] The above is the specific flow of processing by this system.

[0093] Example 1

[0094] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0095] Modern web services require complex user interfaces and multilingual support, which can make them difficult to use, especially for users with low IT literacy or who face language barriers.Furthermore, differences in operating procedures between different web services can make users feel stressed about procedures and operations.

[0096] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0097] In this invention, the server includes: means for a user to input service information or take a screen shot; means for transmitting the input information or the captured image to a cloud-based processing device; means for analyzing the transmitted information and using a generative AI model to identify the structure and content of the service; means for converting the identified information into an intuitive user interface that is easy for the user to understand; means for the user to communicate requests by voice or text; means for analyzing the requests using natural language processing to identify specific operations; means for generating a web request to execute the identified operations; and means for notifying the user of the completion of the executed operations. This allows users to easily perform complex operations, improving convenience, particularly for users with low IT literacy and when using international services.

[0098] "User" means any person or entity that intends to use the System and / or the Services.

[0099] "Service information" refers to the specific URL and screen information related to the web service the user wants to use.

[0100] "Means for capturing screen images" refers to a function that uses a camera installed on a mobile device such as a smartphone or tablet to capture an image of the screen of the service being used.

[0101] A "cloud processing unit" is a computer resource that runs on a remote server accessible over the Internet.

[0102] "Means for transmitting" refers to the network function for transmitting data from the user's terminal to a processing device on the cloud.

[0103] A "generative AI model" is a machine learning model that is trained to perform a specific task based on a large dataset.

[0104] "User interface" refers to the screen and operating means that users use to operate the system.

[0105] "Natural language processing" refers to techniques and methods for analyzing natural language and converting it into meaningful information.

[0106] A "web request" refers to data that requests or sends information to a server on the Internet.

[0107] "Means of notification" means the functionality that allows the system to send information or messages to the user.

[0108] This invention provides a system that helps users easily use web services through smartphone apps. Specific embodiments of this system are described below.

[0109] First, users install and launch a dedicated application on their smartphone or tablet. This application can be downloaded from various app stores, such as Google Play® and the App Store. After launching the application, users can either enter the URL of the service they want to use in the "Enter URL" field or take a screenshot using the camera function.

[0110] For example, if a user wants to purchase an item on an overseas auction site, they can enter the URL of the product page or take a photo of the product page with their camera. The app has a "Send" button, and when the user presses this button, the entered URL or the captured image is sent to a server in the cloud. The sent data is sent to the server in JSON format via an HTTPS request.

[0111] The server then accesses the received URL to retrieve the HTML data. It also uses OCR technology (e.g., Tesseract OCR) to extract text information from the captured image. The server processes this data using libraries such as Python requests, BeautifulSoup, and Selenium.

[0112] The server then uses the acquired HTML data and image analysis results to identify the page structure and elements. At this stage, a generative AI model (e.g., OpenAI (registered trademark) GPT) is used to recognize and classify relevant information such as product names, prices, and the location of purchase buttons. Based on the analysis results, a user interface (UI) is created that is intuitive and easy for users to understand. The server generates the UI using front-end libraries such as D3.js and React, and sends the generated UI data to the user's device.

[0113] The device displays a user interface based on the received data. While viewing this interface, the user can use voice or text input functions to instruct specific actions. For example, they can give voice instructions such as "I want to purchase this product," or manually enter text into a text field. Voice input is converted into text data using a speech recognition API such as Google® Cloud Speech-to-Text or IBM Watson®.

[0114] The device then uses natural language processing technology (e.g., spaCy) to analyze the user's instructions and identify specific operations. The analysis results are then sent back to the server, which then generates and executes the corresponding web requests. Specifically, the server uses tools such as Selenium to automatically operate the browser and complete the necessary procedures.

[0115] Finally, the server notifies the device that the operation is complete, and the device displays the notification to the user, for example, by displaying a message saying "Purchase procedure completed" to let the user know that all operations were completed successfully.

[0116] Here are some examples of prompts:

[0117] "User: I would like to purchase this item from Amazon.

[0118] URL:https: / / www.example.com / example-product

[0119] Camera: A screenshot of your screen has been uploaded.

[0120] This system will allow users to easily perform complicated operations, greatly improving convenience, especially for users with low IT literacy and those using international services.

[0121] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0122] Step 1: User Actions

[0123] Users install and launch a dedicated application on their smartphone or tablet. Then, they can enter the URL of the service they want to use or take a screenshot of the screen with their camera. Specifically, users download the application from Google Play or the App Store, open the application, and enter the URL in the "Enter URL" field or tap the camera icon to take a screenshot of the screen.

[0124] Input: URL or image data

[0125] Output: URLs and image data prepared within the application

[0126] Step 2: Sending data

[0127] The device sends the URL entered by the user and the captured image data to a server on the cloud. Specifically, when the user taps the "Send" button, the data is sent to the server in JSON format via an HTTPS request.

[0128] Input: URL or image data ready

[0129] Output: The URL and image data are sent to the server and accepted by the server.

[0130] Step 3: Get the URL and analyze the image

[0131] The server accesses the sent URL and retrieves the HTML data. It also uses Tesseract OCR technology to extract text information from the captured image data. Specifically, the server uses the Python requests library to send a GET request to the URL and uses Tesseract OCR to retrieve text information from the image.

[0132] Input: URL or image data sent to the server

[0133] Output: HTML data or parsed text information

[0134] Step 4: Analyzing the Page

[0135] The server analyzes the acquired HTML data and image analysis results to identify the structure and elements of the web page. At this stage, a generative AI model (e.g., OpenAI GPT) is used to recognize and classify relevant information such as the product name, price, and purchase button location. Specifically, the server parses the HTML data using BeautifulSoup or Selenium and organizes the information using generative AI.

[0136] Input: HTML data or text information

[0137] Output: Identified product information (name, price, button position, etc.)

[0138] Step 5: Transforming the UI

[0139] The server generates a user interface (UI) based on the analysis results, which is easy for users to understand intuitively. Specifically, the server visualizes the analysis results using D3.js and React, and sends the generated UI data in JSON format to the device.

[0140] Input: Identified product information

[0141] Output: Generated UI data

[0142] Step 6: Input user requests

[0143] The user can then view the converted UI and input a specific action using voice or text, such as "I want to buy this product." To achieve this, the user can tap the voice input button in the app to input voice or type directly into a text field.

[0144] Input: Generated UI data

[0145] Output: User's voice commands or text input

[0146] Step 7: Parsing the instructions

[0147] The device converts the voice instructions used into text using Google Cloud Speech-to-Text or IBM Watson, and then analyzes them using natural language processing technology (e.g., spaCy). As a result of the analysis, specific operation instructions are identified and sent to the server. Specific operations include sending the voice data to an API, converting it into text, and identifying actions based on the analyzed text.

[0148] Input: User's voice commands or text input

[0149] Output: Specific operation instructions sent to the server

[0150] Step 8: Delegate the service

[0151] The server generates and executes web requests based on the user's actions, such as logging in to an account, clicking a purchase button, or entering payment information. The server then uses Selenium to automatically perform the necessary steps.

[0152] Input: Specific operating instructions

[0153] Output: The result of the web request that was made.

[0154] Step 9: Operation completion notification

[0155] The server sends a notification to the device when the proxy operation is complete. The device displays the received notification to the user. For example, it displays a message saying "The purchase procedure has been completed." Specifically, the server sends the operation completion status to the device in JSON format, and the device receives it and displays it in the app.

[0156] Input: The result of the web request that was executed

[0157] Output: Notification to the user that the operation was completed

[0158] (Application example 1)

[0159] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0160] When using overseas web services, especially food delivery services, users often have difficulty with different languages ​​and complex UIs. Due to language barriers and a lack of IT literacy, many users are unable to use the services easily and intuitively.

[0161] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0162] In this invention, the server includes a means for accessing a URL entered by a user and acquiring HTML data, a means for analyzing captured image data to extract the screen structure, and a means for automatically translating information displayed in different languages ​​and converting it into a language that is easy for the user to understand. This allows users to intuitively use overseas food delivery services without having to deal with language or complex UIs.

[0163] "Service URL" refers to the Internet address of the web service that a User wishes to use.

[0164] "Taking a picture of the screen with a camera" refers to the act of a user using the camera function of a smartphone or other device to save the display screen of a web service as image data.

[0165] "Identified information" refers to data about the structure and content of web services obtained by analyzing URLs and image data.

[0166] "Easy-to-see and easy-to-understand UI" refers to an interface that is designed to be easy for users to understand and operate.

[0167] "Communicating requests by voice or text" refers to the act of a user using voice input or text input to communicate desired operations or instructions to the system.

[0168] "Performing appropriate operations" refers to the act of analyzing the user's request and having the system take the necessary actions on the web service based on those instructions.

[0169] "Notifying the user of the completion of an executed operation" refers to the act of notifying the user that an operation performed by the system based on the user's instructions has been completed.

[0170] "Automatically translating information displayed in different languages" refers to the process of automatically converting content displayed by a web service into a language that the user can understand, when that content is not in the user's native language.

[0171] "Providing services and products available in a specific area" refers to the act of presenting users with information about services and products available within a specific area based on the user's location or designated area.

[0172] First, the user installs and launches the smartphone app, which is designed to assist in the use of web services and provides users with intuitive and easy operation.

[0173] Screen Input

[0174] Users can either enter the URL of the service they want to use through the app interface or take a screenshot of the service screen using the camera. For example, if a user wants to use an overseas food delivery service, they can enter the URL of the service or take a screenshot of the screen.

[0175] Data transmission and analysis

[0176] The device sends the URL entered by the user or the captured image data to a server on the cloud. The server then accesses the received URL and retrieves the HTML data. It also uses image analysis technology (such as OCR) to extract the screen structure and text information from the captured image data. The server then analyzes the retrieved HTML data or image analysis results to identify the page structure and elements. This includes utilizing generative AI to recognize and classify appropriate information. For example, it identifies menu names, prices, and the position of the order button.

[0177] UI transformation

[0178] Based on the analysis results, the server converts the UI into a format that is intuitively easy for users to understand. Specifically, it highlights important information and places action buttons to simplify user operations. It also automatically translates information displayed in different languages ​​and converts it into a language that is easy for users to understand. The easy-to-read UI data generated here is sent to the device and displayed to the user.

[0179] Collecting user requests

[0180] While viewing the converted UI, the user can specify actions by voice or text. For example, the user can say, "I would like to order this dish." The device uses voice recognition technology to convert the user's voice instructions into text data, and then uses natural language processing technology to analyze the instructions. The results of this analysis are sent to the server as a specific action (e.g., placing an order).

[0181] Service agency

[0182] The server generates and executes the corresponding web request based on the parsed action. This may include logging in with the user's account information, clicking an order button, or entering payment information. If necessary, the server prompts the user for additional information (e.g., shipping address or payment information). The device notifies the user of this request, prompts them to enter the required information, and then sends it back to the server.

[0183] Operation completion notification

[0184] The server notifies the terminal when the proxy operation is complete. The terminal then displays the received notification to the user, informing them that all operations have been completed. For example, it displays a message such as "The order procedure has been completed." This allows the user to easily use the services they need without having to struggle with complicated operations or an unfriendly UI.

[0185] Examples and prompts

[0186] For example, if a user wants to use an overseas food delivery service, they can enter the following prompt into the app:

[0187] Example prompt sentence:

[0188] "https: / / example-fooddeli.com"

[0189] or

[0190] "https: / / example-image.com / screenshot.png"

[0191] This allows users to use the service intuitively without having to deal with different languages ​​or complex operations, and the system offers great convenience, especially for users with low IT literacy or who face language barriers.

[0192] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0193] Step 1:

[0194] The user launches the smartphone app and either enters the URL of the service they want to use or takes a screenshot of the service's screen using the camera. This entered URL or captured screen image becomes the input data for processing.

[0195] Input data: URL or image

[0196] Output data: Input URL or image data

[0197] Step 2:

[0198] The device sends the URL entered by the user or the captured image data to a server on the cloud. This sending operation passes the input data directly to the server.

[0199] Input data: User's URL or image data

[0200] Output data: Transmission to server completed

[0201] Step 3:

[0202] The server accesses the received URL and retrieves the HTML data. It also uses OCR technology to analyze the screen structure and text information of the captured image data. Specifically, it uses pytesseract to extract text from the image and BeautifulSoup to analyze the HTML.

[0203] Input data: URL or image data

[0204] Output data: HTML data or parsed text information

[0205] Step 4:

[0206] The server analyzes the acquired HTML data or image analysis results to identify the page structure and elements. Generative AI technology is then used to properly recognize the identified information and classify important information. For example, menu names, prices, and the location of the order button are identified.

[0207] Input data: HTML data or parsed text information

[0208] Output data: Structure and element information of identified pages

[0209] Step 5:

[0210] The server converts the UI into a format that is intuitively easy for users to understand based on the analysis results. During this process, it highlights important information and places action buttons to simplify user operations. In addition, it automatically translates information displayed in different languages ​​and converts it into a language that is easy for users to understand.

[0211] Input data: Identified page structure and element information

[0212] Output data: Transformed UI data

[0213] Step 6:

[0214] The device displays the converted UI data received from the server to the user, who can then instruct actions via voice or text while viewing the UI.

[0215] Input data: Transformed UI data

[0216] Output data: User's voice or text instructions for action

[0217] Step 7:

[0218] The device uses voice recognition technology to convert the user's voice instructions into text data, and then uses natural language processing technology to analyze the instructions. This is where libraries such as Spacy are used. The results of this analysis are sent to the server as a specific action (e.g., ordering food).

[0219] Input data: User's voice or text instructions

[0220] Output data: Parsed instructions

[0221] Step 8:

[0222] The server generates and executes the corresponding web requests based on the parsed user instructions, which may include logging in with the user's account information, clicking an order button, or entering payment information.

[0223] Input data: Parsed instructions

[0224] Output data: Status of the operation performed

[0225] Step 9:

[0226] The server notifies the terminal when the executed operation is completed. The terminal displays the received notification to the user to inform them that all operations have been completed. For example, it displays a message such as "The order procedure has been completed."

[0227] Input data: Status of the operation performed

[0228] Output data: Completion notification to the user

[0229] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0230] This invention is a system that reduces the complexity and difficulty of understanding that users face when using web services. This system includes a function that recognizes the user's emotions and provides more appropriate support by combining it with an emotion engine.

[0231] Screen Input

[0232] Users can either enter the URL of the web service they want to use through a smartphone app or take a screenshot of the service's screen using the camera. For example, if a user wants to access a product page on an overseas auction site, they can enter the URL of that product page or take a screenshot.

[0233] Data transmission and analysis

[0234] The device sends the URL entered by the user and the captured image data to a server on the cloud. If a URL is entered, the URL information is sent, and if an image is captured, the image data is sent.

[0235] The server retrieves the HTML data of the web page based on the received URL, or analyzes the image data to extract the screen structure. Image analysis technology is used to identify the text information and screen structure within the image.

[0236] UI transformation

[0237] The server analyzes the acquired HTML data and image analysis results to identify important elements on the page (e.g., product name, price, purchase button), and uses generative AI to convert them into a user-friendly UI, which is then sent to the device.

[0238] Utilizing the Emotion Engine

[0239] When users see the UI displayed on their device, they can input their request by voice or text. For example, they can give instructions such as "I would like to purchase this product." At this time, the emotion engine recognizes the user's emotions from the voice or text.

[0240] The device uses an emotion engine to analyze the user's emotional state, speech recognition technology to convert voice instructions into text, and natural language processing technology to analyze the text data and send specific actions (e.g., purchase procedures) to the server.

[0241] Service agency

[0242] Based on the parsed actions, the server generates corresponding web requests and performs service operations on behalf of the user: first, logging in with their account information, then clicking the purchase button and entering payment information.

[0243] If necessary, the server will prompt the user for additional information (e.g., shipping address or payment information). The device will notify the user of the request, prompt them to enter the required information, and then send it back to the server.

[0244] Operation completion notification

[0245] When the operation is complete, the server notifies the device. The device then notifies the user of the completion of the operation, for example by displaying a message such as "The purchase procedure has been completed."

[0246] Emotion-Based Feedback

[0247] The emotion engine continuously monitors the user's emotions while using the service and recognizes when the user is feeling stressed or confused. The device uses the information from the emotion engine to provide appropriate feedback and additional support to the user, such as detailed instructions or hints on how to operate the device, improving user convenience.

[0248] In this way, the system allows users to use the service without difficulty and provides a more comfortable experience. In particular, by combining it with an emotion engine, it is possible to take the user's emotions into consideration and provide optimal support. As a result, it is possible to provide excellent convenience even to users with low IT literacy or who experience language barriers.

[0249] The processing flow will be explained below.

[0250] Step 1:

[0251] Users launch a smartphone app and either enter the URL of the web service they want to use or take a screenshot of the screen of the service they want to use using the camera. For example, they can enter the URL or take a screenshot to access the product page of an overseas auction site.

[0252] Step 2:

[0253] The device receives the URL entered by the user or the captured image data and sends it to a server in the cloud. The URL information and image data are sent to the server securely.

[0254] Step 3:

[0255] The server retrieves the HTML data of the web page based on the received URL, or analyzes the received image data to extract the text and screen structure within the image. Image analysis technology is used to perform character recognition (OCR) and structure recognition.

[0256] Step 4:

[0257] The server performs detailed analysis of the acquired HTML data and image analysis results to identify important elements on the page (e.g., product name, price, purchase button, etc.). Generative AI is used for the analysis to classify and organize the data.

[0258] Step 5:

[0259] The server generates user interface (UI) data based on the identified information, converting it into a user-friendly format, including highlighting important information and arranging action buttons to simplify operation.

[0260] Step 6:

[0261] The server transmits the generated UI data to the terminal.

[0262] Step 7:

[0263] The device displays the received UI data to the user, who then considers what action to take next.

[0264] Step 8:

[0265] The user looks at the displayed UI and inputs their request by voice or text. For example, they can say, "I would like to purchase this product."

[0266] Step 9:

[0267] The device uses an emotion engine to recognize emotions from the user's voice or text, then uses voice recognition technology to convert the voice instructions into text and natural language processing technology to analyze the text data. This analysis then identifies specific actions (e.g., checkout).

[0268] Step 10:

[0269] The device sends the analysis results and the user's emotional state to the server.

[0270] Step 11:

[0271] The server generates a corresponding web request based on the received action instruction and performs the required operation, such as logging in to an auction site and proceeding with the process of purchasing an item.

[0272] Step 12:

[0273] The server prompts the user for additional information (e.g., shipping address or payment information) if necessary.

[0274] Step 13:

[0275] The terminal notifies the user of a request for additional information from the server and prompts the user to enter the required information.

[0276] Step 14:

[0277] The user enters any additional information requested and sends it to the terminal.

[0278] Step 15:

[0279] The terminal sends any additional information entered by the user to the server.

[0280] Step 16:

[0281] The server uses the additional information received to complete any necessary operations, such as entering payment information and finalizing the order.

[0282] Step 17:

[0283] The server notifies the terminal that the operation is complete.

[0284] Step 18:

[0285] The device receives this notification and displays it to the user, for example, "Purchase procedure completed."

[0286] Step 19:

[0287] The emotion engine continuously monitors the user's emotions while using the service and provides appropriate feedback and support to the user depending on the situation.

[0288] The above is the specific processing flow of this system. This allows users to smoothly use the services they need without having to worry about complicated procedures or an unfriendly UI. In addition, by combining it with an emotion engine, it is possible to provide optimal support that is tailored to the user's emotions.

[0289] Example 2

[0290] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0291] When users use web services, they often find the operation complicated or difficult to understand. This makes it difficult to use, especially for users with low IT literacy or who face language barriers. There is a need to solve these problems and enable users to use services smoothly. Furthermore, by taking user emotions into consideration, it is necessary to provide more appropriate support and improve the user experience.

[0292] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0293] In this invention, the server includes means for a user to input a service address or take a picture of the screen with a camera, means for analyzing the input address or the captured image and identifying the structure and content of the service, means for converting the identified information into an easy-to-view and understandable user interface, means for the user to communicate their request by voice or text, means for analyzing the user's emotional state using an emotion engine, means for analyzing the request and performing an appropriate operation, and means for notifying the user of the completion of the performed operation. This not only allows users to easily use web services, but also makes it possible to provide optimal support through emotion recognition.

[0294] "User" refers to any individual or entity that uses the Web Services or Applications.

[0295] An "address" is a series of character strings that indicates unique location information for identifying access to a web service.

[0296] A "camera" is a device for taking pictures and storing them as digital data.

[0297] "Image data" is digital information representing a still image captured by a device such as a camera.

[0298] "Analysis" is the process of processing input data (addresses or image data) to extract information to understand its structure and content.

[0299] A "user interface" is the visual and operational interface through which a user interacts with a computer system or application.

[0300] "Voice" refers to the words and instructions given by the user captured as sound.

[0301] "Text" is a document or instruction created by combining letters and symbols.

[0302] An "emotion engine" is a function or software that analyzes a user's emotional state from voice or text.

[0303] "Natural language processing" is a technology that allows computers to understand, analyze, and generate human language.

[0304] An "action" refers to a specific operation or behavior that a system performs based on a user's instructions.

[0305] "Operation completed" is a state indicating that the action instructed by the user has been executed successfully.

[0306] "Notifications" are messages or alerts that inform the user about the completion of an operation or the status of the system.

[0307] A "system" is a collection of multiple pieces of hardware and software that work together to achieve a specific function or purpose.

[0308] "Structure" refers to the internal layout and arrangement of elements of a web page or service, and their relationships to one another.

[0309] "Content" refers to information such as text, images, links, and forms on a web page or service.

[0310] This invention is a system that reduces the complexity and difficulty of understanding when users use web services. This system has the function of recognizing the user's emotions by combining it with an emotion engine and providing more appropriate support.

[0311] Hardware and software used:

[0312] The system uses a smartphone app, cloud server, emotion engine, image analysis technology, natural language processing technology, etc. By combining these technologies, it supports user operations.

[0313] The specific process flow:

[0314] The server receives the address entered by the user using the smartphone app or the captured image data. If an address is entered, the server retrieves the HTML data of the corresponding web page. If captured image data is sent, the server uses image analysis technology to identify the text information and screen structure within the image.

[0315] The server then analyzes the acquired HTML data and image analysis results to identify important elements on the page (e.g., product name, price, purchase button). Based on this information, it uses a generative AI model to convert it into a user interface that is easy for the user to understand. The converted UI data is then sent to the device.

[0316] The user looks at the new UI and inputs their request by voice or text. For example, they might say, "I want to buy this product." At this time, the emotion engine recognizes the emotion from the user's voice or text and analyzes that information.

[0317] The device uses an emotion engine to analyze the user's emotional state, speech recognition technology to convert voice instructions into text, and natural language processing technology to analyze the text data, convert it into a specific action (e.g., purchase procedure), and send it to the server.

[0318] Based on the parsed action, the server generates the appropriate web request and performs the service operation on behalf of the user. For example, the user logs in with their account information, clicks a purchase button, and enters payment information. If necessary, the server prompts the user for additional information (for example, shipping address or payment information). The user provides the required information through the app.

[0319] Once the operation is complete, the server notifies the device, which then displays a message to the user saying, "The purchase has been completed."

[0320] The emotion engine also continuously monitors the user's emotions while using the service, recognizing when the user is feeling stressed or confused. The device uses the information from the emotion engine to provide appropriate feedback and additional support to the user, such as providing detailed instructions and hints on how to operate the device, improving user convenience.

[0321] Examples:

[0322] Example prompt sentence:

[0323] URL: https: / / example-auctionsite.com / product123

[0324] Take a screenshot and transform the UI to make it easier for users to find the purchase button.

[0325] Through these steps, the system enables users to use web services efficiently and easily, and provides optimal support through emotion recognition, making it possible to provide an easy-to-use environment even for users with low IT literacy or who face language barriers.

[0326] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0327] Step 1:

[0328] The user opens the smartphone app and enters the address of the service, or takes a screenshot using the camera. The input can be in the form of a URL or a photo. For example, if a user wants to access a product page on an auction site, they can enter the URL of that product page or take a screenshot. The input data is sent to the device.

[0329] Step 2:

[0330] The device sends the URL entered by the user or the captured image data to a server on the cloud. The device converts the entered information into the appropriate format (URL or image data) and sends it to the server. Specifically, it sends the URL information or image data via an Internet connection.

[0331] Step 3:

[0332] The server retrieves the HTML data of the web page based on the received URL. If a URL is entered, the server accesses the address and downloads the HTML data. If captured image data is sent, the server uses image analysis technology to identify the text information and screen structure within the image.

[0333] Step 4:

[0334] The server analyzes the HTML data and image analysis results to identify important elements on the page (e.g., product name, price, buy button). The server extracts this information using an HTML parser and image analysis algorithms.

[0335] Step 5:

[0336] The server uses a generative AI model to convert the data into a user interface that is easy for the user to understand. Based on the analysis results, the server changes the layout to a format that is visually easy for the user to understand, and sends the new UI data to the device.

[0337] Step 6:

[0338] When users see the new UI displayed on their device, they can input their request by voice or text. For example, they can say, "I want to purchase this product." The device then receives the user's voice or text.

[0339] Step 7:

[0340] The device uses an emotion engine to analyze the user's emotional state, converts voice instructions into text using voice recognition technology, and analyzes the text data using natural language processing technology. Specific actions are then taken as a result of the analysis.

[0341] Step 8:

[0342] The terminal then sends the analyzed specific action to the server. For example, an action such as "click the purchase button" or "enter payment information" is generated and sent to the server.

[0343] Step 9:

[0344] The server generates a corresponding web request based on the action it receives: for example, logging in with a specified account, clicking a purchase button, entering payment information, etc. If necessary, the server requests additional information (e.g., a shipping address).

[0345] Step 10:

[0346] When the operation is complete, the server notifies the device, which then notifies the user by displaying a message such as "The purchase procedure has been completed."

[0347] Step 11:

[0348] The emotion engine continuously monitors the user's emotions while using the service. It recognizes when the user feels stressed or confused. The device uses the information from the emotion engine to provide appropriate feedback and additional support. For example, it can support the user's operations by displaying detailed instructions or hints on how to operate the device.

[0349] (Application example 2)

[0350] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0351] When using conventional web services, users often face complex operations and difficulties in understanding them, posing significant barriers, particularly for users with low IT literacy or who speak a different language. Furthermore, lack of appropriate support often prevents users from identifying their emotions, increasing stress and confusion and impairing the user experience. The objective of this invention is to solve these problems and provide a more intuitive and user-friendly user interface and appropriate feedback based on the user's emotions, enabling users to use web services more comfortably.

[0352] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0353] In this invention, the server includes: means for a user to input a service URL or take a picture of the screen with a camera; means for analyzing the input URL or the captured image and identifying the structure and content of the service; means for converting the identified information into an easy-to-view and understandable UI; means for the user to communicate their request by voice or text; means for analyzing the request and performing an appropriate operation; means for notifying the user of the completion of the executed operation; means for analyzing the user's emotions using an emotion recognition engine and providing feedback based on the analysis results; and means for generating and responding to the user's request with an appropriate prompt sentence using a generative AI model. This not only simplifies complex operations for the user, but also enables the user to receive appropriate support tailored to their emotions.

[0354] "Service URL" means the web address of the online service that a User wishes to access.

[0355] "Taking a photo with a camera" refers to the act of recording the screen of the service the user wants to use as a digital image.

[0356] "Analysis" is the act of taking input data, understanding its structure and content, and extracting information.

[0357] "Identification" refers to the act of extracting and identifying important elements from the analyzed information.

[0358] An "easy-to-see and understand UI" is a user interface that users can easily understand and operate.

[0359] "Voice or text request" means the act of a user giving instructions to a system using voice input or text messages.

[0360] "Performing appropriate operations" refers to the system taking specific actions in response to the user's request.

[0361] "Notifying the completion of an operation" refers to the act of notifying the user that an operation performed by the system has been completed.

[0362] An "emotion recognition engine" is a software technology for analyzing emotions from a user's voice or text.

[0363] "Providing feedback based on analysis results" refers to the act of providing appropriate information and support to users based on the analysis results of the emotion recognition engine.

[0364] A "generative AI model" is a technology that uses artificial intelligence to generate appropriate outputs in response to specific prompts.

[0365] "Generating appropriate prompts" refers to the act of creating the most appropriate questions or instructions based on the user's request.

[0366] This invention is a system that reduces the complexity and difficulty of understanding when users use web services. By combining an emotion recognition engine and a generative AI model, this system includes a function that recognizes user emotions and provides more appropriate support.

[0367] First, users enter the URL of the web service they want to use through a smartphone app, or use the camera to take a screenshot of the screen of the service they want to use. For example, if a user wants to watch content from a specific video streaming service, they can enter the URL of the content page or take a screenshot of the screen.

[0368] Next, the device sends the URL entered by the user or the captured image data to a server in the cloud. If a URL is entered, the URL information is sent, and if an image is captured, the image data is sent. The server retrieves the HTML data of the webpage based on the received URL, or analyzes the image data to extract the screen structure. The software used here is a web scraper and image analysis technology.

[0369] The server analyzes the acquired HTML data and image analysis results to identify important elements on the page (e.g., content title, play button, description), converts them into a user-friendly UI using a generative AI model, and sends the converted UI data to the device.

[0370] When users use the new UI to perform specific actions by voice or text, the application uses an emotion recognition engine to analyze the user's emotional state and converts the voice commands into text using speech recognition technology, specifically a speech recognition library and natural language processing technology.

[0371] The server generates a corresponding web request based on the analyzed action and performs the service operation on behalf of the user. First, to play content, for example, the user logs in with their account information and then clicks the play button. After playback is complete, the server notifies the device that the operation is complete.

[0372] The emotion recognition engine also continuously monitors the user's emotions while using the service, recognizing when the user is feeling stressed or confused. For example, if the user verbally requests "I want to know more," the generative AI model will generate the following prompt:

[0373] "Example prompt statement:

[0374] User input: "I want to know more"

[0375] The generative AI model prompts: "Please provide more details about the content the user is looking for."

[0376] Expected output: "Here's a detailed description of this content. Let me know if you need more information."

[0377] In this way, this system provides the detailed information and operating procedures that users need, improving user convenience. In particular, by combining it with an emotion recognition engine, it is possible to provide optimal feedback based on the user's emotional state. As a result, users can use web services more comfortably.

[0378] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0379] Step 1:

[0380] Users can either enter the URL of the web service they want to use using a smartphone app, or take a photo of the service screen using the camera. The input data is a text URL, and the photo data is an image file.

[0381] Step 2:

[0382] The device sends the URL entered in step 1 or the captured image data to a server on the cloud. The sent data is analyzed on the server side.

[0383] Step 3:

[0384] The server retrieves the HTML data of the web page based on the received URL, or analyzes the image data to extract the screen structure. Specifically, it retrieves the HTML data using a web scraper and uses image analysis technology to identify the text information and screen structure within the image.

[0385] Step 4:

[0386] The server identifies important elements on the page (e.g., content title, play button, description) from the analyzed HTML data and image analysis results. In this process, necessary information is extracted from the analyzed data and converted into an easy-to-read and understandable UI using a generative AI model.

[0387] Step 5:

[0388] The terminal displays the converted UI data sent from the server to the user, who then looks at the displayed UI and inputs their next request by voice or text.

[0389] Step 6:

[0390] The device converts input voice instructions into text using a speech recognition library and analyzes the user's emotional state using an emotion recognition engine. Here, the input data is an audio file, and the output data is text and emotional state information.

[0391] Step 7:

[0392] The server uses a generative AI model to generate appropriate feedback and prompts based on the emotional state information obtained from the emotion recognition engine, and the generated prompts are returned to the user as appropriate instructions or additional information.

[0393] Step 8:

[0394] The server performs the operation requested by the user, prompting the user for additional information as needed (e.g., the user's account information or payment information). This is where data processing and calculations are performed to generate and fulfill the web request.

[0395] Step 9:

[0396] The device notifies the user of the request sent from the server, prompts the user to enter the required information, and then sends it back to the server. This interaction allows the user to provide additional information.

[0397] Step 10:

[0398] The server confirms the completion of the operation and notifies the terminal of the completion. The terminal then displays a notification of the operation completion to the user, informing the user that the operation has been completed successfully. For example, a message such as "The playback procedure has been completed" may be displayed.

[0399] Through these steps, users are freed from cumbersome operations and can use web services while receiving appropriate feedback based on their emotions. Prompt sentence generation using a generative AI model enables users to use services intuitively.

[0400] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0401] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0402] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0403] [Second embodiment]

[0404] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0405] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0406] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0407] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0408] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0409] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0410] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0411] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0412] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0413] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0414] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0415] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0416] First, the user installs and launches the smartphone app, which is designed to assist in the use of web services and provides users with intuitive and easy operation.

[0417] Screen Input

[0418] Users can either enter the URL of the service they want to use through the app's interface or take a screenshot of the service's screen using the app's camera. For example, if a user wants to access a product page on an overseas auction site, they can enter the URL of the product page or take a screenshot of the screen.

[0419] Data transmission and analysis

[0420] The device sends the URL entered by the user and the captured image data to a server on the cloud. The server then accesses the received URL and retrieves the HTML data. It also uses image analysis technology (such as OCR) to extract the screen structure and text information from the captured image data.

[0421] The server analyzes the acquired HTML data or image analysis results to identify the page structure and elements. This includes using generative AI to recognize and classify appropriate information, such as the product name, price, and the location of the purchase button.

[0422] UI transformation

[0423] The server then converts the UI into a format that is intuitively easy for users to understand based on the analysis results. Specifically, it highlights important information and arranges action buttons to simplify user operations. The easy-to-read UI data generated here is sent to the device and displayed to the user.

[0424] Collecting user requests

[0425] Users can view the converted UI and then specify actions by voice or text, for example, "I want to buy this product."

[0426] The device uses voice recognition technology to convert the user's voice instructions into text data, and then uses natural language processing technology to analyze the instructions. The results of this analysis are sent to the server as a specific action (e.g., purchasing procedure).

[0427] Service agency

[0428] The server generates and executes the corresponding web request based on the parsed action, which may include the user logging in with their account information, clicking a purchase button, or entering payment information.

[0429] If necessary, the server will prompt the user for additional information (e.g., shipping address or payment information). The device will notify the user of this request, prompt them to enter the required information, and then send it back to the server.

[0430] Operation completion notification

[0431] When the proxy operation is complete, the server notifies the terminal of the completion. The terminal displays the received notification to the user, informing them that all operations have been completed. For example, it displays a message such as "The purchase procedure has been completed."

[0432] This allows users to easily access the services they need without having to deal with complicated operations or an unfriendly UI. This system is particularly convenient for users with low IT literacy or who face language barriers.

[0433] The processing flow will be explained below.

[0434] Step 1:

[0435] Users launch a smartphone app and enter the URL of the web service they want to use, or take a picture of the screen using the camera. For example, consider the case where a user enters the URL of a product page on an overseas auction site.

[0436] Step 2:

[0437] The device receives the URL entered by the user or the captured image data and sends it to a server on the cloud. If a URL is entered, the URL information is sent, and if an image is captured, the image data is sent.

[0438] Step 3:

[0439] The server retrieves the HTML data of the web page based on the received URL, and if image data is sent, it uses image analysis technology to extract text information and screen structure from the image.

[0440] Step 4:

[0441] The server performs analysis using the acquired HTML data and image analysis results. Specifically, it uses generative AI to identify important elements on the page (e.g., product name, price, purchase button, etc.).

[0442] Step 5:

[0443] The server generates UI data based on the identified information, converting it into a format that is easy for the user to understand. This UI data includes highlighting important information and arranging buttons for easy user operation.

[0444] Step 6:

[0445] The server transmits the generated UI data to the terminal.

[0446] Step 7:

[0447] The device displays the received UI data to the user, allowing the user to consider their next action while looking at the displayed UI.

[0448] Step 8:

[0449] The user inputs their request by voice or text, for example, "I would like to purchase this product."

[0450] Step 9:

[0451] The device uses voice recognition technology to convert the user's voice instructions into text, and then uses natural language processing technology to analyze the text data and understand it as a specific action (e.g., checkout).

[0452] Step 10:

[0453] The device then sends the analyzed user request to the server, and as a specific action, sends a purchase procedure request to the server.

[0454] Step 11:

[0455] Based on the received action instruction, the server generates a corresponding web request and performs the required operation, for example, logging in to an auction site and initiating the process of purchasing an item.

[0456] Step 12:

[0457] The server will prompt the user for additional information as needed, for example a shipping address or payment information.

[0458] Step 13:

[0459] The terminal notifies the user of a request for additional information from the server and prompts the user to input the required information.

[0460] Step 14:

[0461] The user enters any additional information requested and transmits it to the terminal.

[0462] Step 15:

[0463] The terminal transmits the additional information entered by the user to the server.

[0464] Step 16:

[0465] The server uses the received additional information to complete any necessary operations, such as entering payment information and finally completing the purchase.

[0466] Step 17:

[0467] The server notifies the terminal that the operation is complete.

[0468] Step 18:

[0469] The device displays the received completion notification to the user to let them know that the operation has been completed, for example, by displaying a message such as "Purchase procedure completed."

[0470] The above is the specific flow of processing by this system.

[0471] Example 1

[0472] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0473] Modern web services require complex user interfaces and multilingual support, which can make them difficult to use, especially for users with low IT literacy or who face language barriers.Furthermore, differences in operating procedures between different web services can make users feel stressed about procedures and operations.

[0474] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0475] In this invention, the server includes: means for a user to input service information or take a screen shot; means for transmitting the input information or the captured image to a cloud-based processing device; means for analyzing the transmitted information and using a generative AI model to identify the structure and content of the service; means for converting the identified information into an intuitive user interface that is easy for the user to understand; means for the user to communicate requests by voice or text; means for analyzing the requests using natural language processing to identify specific operations; means for generating a web request to execute the identified operations; and means for notifying the user of the completion of the executed operations. This allows users to easily perform complex operations, improving convenience, particularly for users with low IT literacy and when using international services.

[0476] "User" means any person or entity that intends to use the System and / or the Services.

[0477] "Service information" refers to the specific URL and screen information related to the web service the user wants to use.

[0478] "Means for capturing screen images" refers to a function that uses a camera installed on a mobile device such as a smartphone or tablet to capture an image of the screen of the service being used.

[0479] A "cloud processing unit" is a computer resource that runs on a remote server accessible over the Internet.

[0480] "Means for transmitting" refers to the network function for transmitting data from the user's terminal to a processing device on the cloud.

[0481] A "generative AI model" is a machine learning model that is trained to perform a specific task based on a large dataset.

[0482] "User interface" refers to the screen and operating means that users use to operate the system.

[0483] "Natural language processing" refers to techniques and methods for analyzing natural language and converting it into meaningful information.

[0484] A "web request" refers to data that requests or sends information to a server on the Internet.

[0485] "Means of notification" means the functionality that allows the system to send information or messages to the user.

[0486] This invention provides a system that helps users easily use web services through smartphone apps. Specific embodiments of this system are described below.

[0487] First, users install and launch a dedicated application on their smartphone or tablet. This application can be downloaded from various app stores, including Google Play and the App Store. After launching the application, users can either enter the URL of the service they want to use in the "Enter URL" field, or use the camera to take a screenshot.

[0488] For example, if a user wants to purchase an item on an overseas auction site, they can enter the URL of the product page or take a photo of the product page with their camera. The app has a "Send" button, and when the user presses this button, the entered URL or the captured image is sent to a server in the cloud. The sent data is sent to the server in JSON format via an HTTPS request.

[0489] The server then accesses the received URL to retrieve the HTML data. It also uses OCR technology (e.g., Tesseract OCR) to extract text information from the captured image. The server processes this data using libraries such as Python requests, BeautifulSoup, and Selenium.

[0490] The server then uses the acquired HTML data and image analysis results to identify the page structure and elements. At this stage, a generative AI model (e.g., OpenAI GPT) is used to recognize and classify relevant information such as product names, prices, and the location of purchase buttons. Based on the analysis results, a user interface (UI) is created that is intuitive and easy for users to understand. The server generates the UI using front-end libraries such as D3.js and React, and sends the generated UI data to the user's device.

[0491] The device displays a user interface based on the received data. While viewing this interface, the user can use voice or text input functions to specify specific actions. For example, they can give voice commands such as "I want to purchase this product," or manually enter text into a text field. Voice input is converted into text data using speech recognition APIs such as Google Cloud Speech-to-Text and IBM Watson.

[0492] The device then uses natural language processing technology (e.g., spaCy) to analyze the user's instructions and identify specific operations. The analysis results are then sent back to the server, which then generates and executes the corresponding web requests. Specifically, the server uses tools such as Selenium to automatically operate the browser and complete the necessary procedures.

[0493] Finally, the server notifies the device that the operation is complete, and the device displays the notification to the user, for example, by displaying a message saying "Purchase procedure completed" to let the user know that all operations were completed successfully.

[0494] Here are some examples of prompts:

[0495] "User: I would like to purchase this item from Amazon.

[0496] URL:https: / / www.example.com / example-product

[0497] Camera: A screenshot of your screen has been uploaded.

[0498] This system will allow users to easily perform complicated operations, greatly improving convenience, especially for users with low IT literacy and those using international services.

[0499] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0500] Step 1: User Actions

[0501] Users install and launch a dedicated application on their smartphone or tablet. Then, they can enter the URL of the service they want to use or take a screenshot of the screen with their camera. Specifically, users download the application from Google Play or the App Store, open the application, and enter the URL in the "Enter URL" field or tap the camera icon to take a screenshot of the screen.

[0502] Input: URL or image data

[0503] Output: URLs and image data prepared within the application

[0504] Step 2: Sending data

[0505] The device sends the URL entered by the user and the captured image data to a server on the cloud. Specifically, when the user taps the "Send" button, the data is sent to the server in JSON format via an HTTPS request.

[0506] Input: URL or image data ready

[0507] Output: The URL and image data are sent to the server and accepted by the server.

[0508] Step 3: Get the URL and analyze the image

[0509] The server accesses the sent URL and retrieves the HTML data. It also uses Tesseract OCR technology to extract text information from the captured image data. Specifically, the server uses the Python requests library to send a GET request to the URL and uses Tesseract OCR to retrieve text information from the image.

[0510] Input: URL or image data sent to the server

[0511] Output: HTML data or parsed text information

[0512] Step 4: Analyzing the Page

[0513] The server analyzes the acquired HTML data and image analysis results to identify the structure and elements of the web page. At this stage, a generative AI model (e.g., OpenAI GPT) is used to recognize and classify relevant information such as the product name, price, and purchase button location. Specifically, the server parses the HTML data using BeautifulSoup or Selenium and organizes the information using generative AI.

[0514] Input: HTML data or text information

[0515] Output: Identified product information (name, price, button position, etc.)

[0516] Step 5: Transforming the UI

[0517] The server generates a user interface (UI) based on the analysis results, which is easy for users to understand intuitively. Specifically, the server visualizes the analysis results using D3.js and React, and sends the generated UI data in JSON format to the device.

[0518] Input: Identified product information

[0519] Output: Generated UI data

[0520] Step 6: Input user requests

[0521] The user can then view the converted UI and input a specific action using voice or text, such as "I want to buy this product." To achieve this, the user can tap the voice input button in the app to input voice or type directly into a text field.

[0522] Input: Generated UI data

[0523] Output: User's voice commands or text input

[0524] Step 7: Parsing the instructions

[0525] The device converts the voice instructions used into text using Google Cloud Speech-to-Text or IBM Watson, and then analyzes them using natural language processing technology (e.g., spaCy). As a result of the analysis, specific operation instructions are identified and sent to the server. Specific operations include sending the voice data to an API, converting it into text, and identifying actions based on the analyzed text.

[0526] Input: User's voice commands or text input

[0527] Output: Specific operation instructions sent to the server

[0528] Step 8: Delegate the service

[0529] The server generates and executes web requests based on the user's actions, such as logging in to an account, clicking a purchase button, or entering payment information. The server then uses Selenium to automatically perform the necessary steps.

[0530] Input: Specific operating instructions

[0531] Output: The result of the web request that was made.

[0532] Step 9: Operation completion notification

[0533] The server sends a notification to the device when the proxy operation is complete. The device displays the received notification to the user. For example, it displays a message saying "The purchase procedure has been completed." Specifically, the server sends the operation completion status to the device in JSON format, and the device receives it and displays it in the app.

[0534] Input: The result of the web request that was executed

[0535] Output: Notification to the user that the operation was completed

[0536] (Application example 1)

[0537] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0538] When using overseas web services, especially food delivery services, users often have difficulty with different languages ​​and complex UIs. Due to language barriers and a lack of IT literacy, many users are unable to use the services easily and intuitively.

[0539] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0540] In this invention, the server includes a means for accessing a URL entered by a user and acquiring HTML data, a means for analyzing captured image data to extract the screen structure, and a means for automatically translating information displayed in different languages ​​and converting it into a language that is easy for the user to understand. This allows users to intuitively use overseas food delivery services without having to deal with language or complex UIs.

[0541] "Service URL" refers to the Internet address of the web service that a User wishes to use.

[0542] "Taking a picture of the screen with a camera" refers to the act of a user using the camera function of a smartphone or other device to save the display screen of a web service as image data.

[0543] "Identified information" refers to data about the structure and content of web services obtained by analyzing URLs and image data.

[0544] "Easy-to-see and easy-to-understand UI" refers to an interface that is designed to be easy for users to understand and operate.

[0545] "Communicating requests by voice or text" refers to the act of a user using voice input or text input to communicate desired operations or instructions to the system.

[0546] "Performing appropriate operations" refers to the act of analyzing the user's request and having the system take the necessary actions on the web service based on those instructions.

[0547] "Notifying the user of the completion of an executed operation" refers to the act of notifying the user that an operation performed by the system based on the user's instructions has been completed.

[0548] "Automatically translating information displayed in different languages" refers to the process of automatically converting content displayed by a web service into a language that the user can understand, when that content is not in the user's native language.

[0549] "Providing services and products available in a specific area" refers to the act of presenting users with information about services and products available within a specific area based on the user's location or designated area.

[0550] First, the user installs and launches the smartphone app, which is designed to assist in the use of web services and provides users with intuitive and easy operation.

[0551] Screen Input

[0552] Users can either enter the URL of the service they want to use through the app interface or take a screenshot of the service screen using the camera. For example, if a user wants to use an overseas food delivery service, they can enter the URL of the service or take a screenshot of the screen.

[0553] Data transmission and analysis

[0554] The device sends the URL entered by the user or the captured image data to a server on the cloud. The server then accesses the received URL and retrieves the HTML data. It also uses image analysis technology (such as OCR) to extract the screen structure and text information from the captured image data. The server then analyzes the retrieved HTML data or image analysis results to identify the page structure and elements. This includes utilizing generative AI to recognize and classify appropriate information. For example, it identifies menu names, prices, and the position of the order button.

[0555] UI transformation

[0556] Based on the analysis results, the server converts the UI into a format that is intuitively easy for users to understand. Specifically, it highlights important information and places action buttons to simplify user operations. It also automatically translates information displayed in different languages ​​and converts it into a language that is easy for users to understand. The easy-to-read UI data generated here is sent to the device and displayed to the user.

[0557] Collecting user requests

[0558] While viewing the converted UI, the user can specify actions by voice or text. For example, the user can say, "I would like to order this dish." The device uses voice recognition technology to convert the user's voice instructions into text data, and then uses natural language processing technology to analyze the instructions. The results of this analysis are sent to the server as a specific action (e.g., placing an order).

[0559] Service agency

[0560] The server generates and executes the corresponding web request based on the parsed action. This may include logging in with the user's account information, clicking an order button, or entering payment information. If necessary, the server prompts the user for additional information (e.g., shipping address or payment information). The device notifies the user of this request, prompts them to enter the required information, and then sends it back to the server.

[0561] Operation completion notification

[0562] The server notifies the terminal when the proxy operation is complete. The terminal then displays the received notification to the user, informing them that all operations have been completed. For example, it displays a message such as "The order procedure has been completed." This allows the user to easily use the services they need without having to struggle with complicated operations or an unfriendly UI.

[0563] Examples and prompts

[0564] For example, if a user wants to use an overseas food delivery service, they can enter the following prompt into the app:

[0565] Example prompt sentence:

[0566] "https: / / example-fooddeli.com"

[0567] or

[0568] "https: / / example-image.com / screenshot.png"

[0569] This allows users to use the service intuitively without having to deal with different languages ​​or complex operations, and the system offers great convenience, especially for users with low IT literacy or who face language barriers.

[0570] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0571] Step 1:

[0572] The user launches the smartphone app and either enters the URL of the service they want to use or takes a screenshot of the service's screen using the camera. This entered URL or captured screen image becomes the input data for processing.

[0573] Input data: URL or image

[0574] Output data: Input URL or image data

[0575] Step 2:

[0576] The device sends the URL entered by the user or the captured image data to a server on the cloud. This sending operation passes the input data directly to the server.

[0577] Input data: User's URL or image data

[0578] Output data: Transmission to server completed

[0579] Step 3:

[0580] The server accesses the received URL and retrieves the HTML data. It also uses OCR technology to analyze the screen structure and text information of the captured image data. Specifically, it uses pytesseract to extract text from the image and BeautifulSoup to analyze the HTML.

[0581] Input data: URL or image data

[0582] Output data: HTML data or parsed text information

[0583] Step 4:

[0584] The server analyzes the acquired HTML data or image analysis results to identify the page structure and elements. Generative AI technology is then used to properly recognize the identified information and classify important information. For example, menu names, prices, and the location of the order button are identified.

[0585] Input data: HTML data or parsed text information

[0586] Output data: Structure and element information of identified pages

[0587] Step 5:

[0588] The server converts the UI into a format that is intuitively easy for users to understand based on the analysis results. During this process, it highlights important information and places action buttons to simplify user operations. In addition, it automatically translates information displayed in different languages ​​and converts it into a language that is easy for users to understand.

[0589] Input data: Identified page structure and element information

[0590] Output data: Transformed UI data

[0591] Step 6:

[0592] The device displays the converted UI data received from the server to the user, who can then instruct actions via voice or text while viewing the UI.

[0593] Input data: Transformed UI data

[0594] Output data: User's voice or text instructions for action

[0595] Step 7:

[0596] The device uses voice recognition technology to convert the user's voice instructions into text data, and then uses natural language processing technology to analyze the instructions. This is where libraries such as Spacy are used. The results of this analysis are sent to the server as a specific action (e.g., ordering food).

[0597] Input data: User's voice or text instructions

[0598] Output data: Parsed instructions

[0599] Step 8:

[0600] The server generates and executes the corresponding web requests based on the parsed user instructions, which may include logging in with the user's account information, clicking an order button, or entering payment information.

[0601] Input data: Parsed instructions

[0602] Output data: Status of the operation performed

[0603] Step 9:

[0604] The server notifies the terminal when the executed operation is completed. The terminal displays the received notification to the user to inform them that all operations have been completed. For example, it displays a message such as "The order procedure has been completed."

[0605] Input data: Status of the operation performed

[0606] Output data: Completion notification to the user

[0607] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0608] This invention is a system that reduces the complexity and difficulty of understanding that users face when using web services. This system includes a function that recognizes the user's emotions and provides more appropriate support by combining it with an emotion engine.

[0609] Screen Input

[0610] Users can either enter the URL of the web service they want to use through a smartphone app or take a screenshot of the service's screen using the camera. For example, if a user wants to access a product page on an overseas auction site, they can enter the URL of that product page or take a screenshot.

[0611] Data transmission and analysis

[0612] The device sends the URL entered by the user and the captured image data to a server on the cloud. If a URL is entered, the URL information is sent, and if an image is captured, the image data is sent.

[0613] The server retrieves the HTML data of the web page based on the received URL, or analyzes the image data to extract the screen structure. Image analysis technology is used to identify the text information and screen structure within the image.

[0614] UI transformation

[0615] The server analyzes the acquired HTML data and image analysis results to identify important elements on the page (e.g., product name, price, purchase button), and uses generative AI to convert them into a user-friendly UI, which is then sent to the device.

[0616] Utilizing the Emotion Engine

[0617] When users see the UI displayed on their device, they can input their request by voice or text. For example, they can give instructions such as "I would like to purchase this product." At this time, the emotion engine recognizes the user's emotions from the voice or text.

[0618] The device uses an emotion engine to analyze the user's emotional state, speech recognition technology to convert voice instructions into text, and natural language processing technology to analyze the text data and send specific actions (e.g., purchase procedures) to the server.

[0619] Service agency

[0620] Based on the parsed actions, the server generates corresponding web requests and performs service operations on behalf of the user: first, logging in with their account information, then clicking the purchase button and entering payment information.

[0621] If necessary, the server will prompt the user for additional information (e.g., shipping address or payment information). The device will notify the user of the request, prompt them to enter the required information, and then send it back to the server.

[0622] Operation completion notification

[0623] When the operation is complete, the server notifies the device. The device then notifies the user of the completion of the operation, for example by displaying a message such as "The purchase procedure has been completed."

[0624] Emotion-Based Feedback

[0625] The emotion engine continuously monitors the user's emotions while using the service and recognizes when the user is feeling stressed or confused. The device uses the information from the emotion engine to provide appropriate feedback and additional support to the user, such as detailed instructions or hints on how to operate the device, improving user convenience.

[0626] In this way, the system allows users to use the service without difficulty and provides a more comfortable experience. In particular, by combining it with an emotion engine, it is possible to take the user's emotions into consideration and provide optimal support. As a result, it is possible to provide excellent convenience even to users with low IT literacy or who experience language barriers.

[0627] The processing flow will be explained below.

[0628] Step 1:

[0629] Users launch a smartphone app and either enter the URL of the web service they want to use or take a screenshot of the screen of the service they want to use using the camera. For example, they can enter the URL or take a screenshot to access the product page of an overseas auction site.

[0630] Step 2:

[0631] The device receives the URL entered by the user or the captured image data and sends it to a server in the cloud. The URL information and image data are sent to the server securely.

[0632] Step 3:

[0633] The server retrieves the HTML data of the web page based on the received URL, or analyzes the received image data to extract the text and screen structure within the image. Image analysis technology is used to perform character recognition (OCR) and structure recognition.

[0634] Step 4:

[0635] The server performs detailed analysis of the acquired HTML data and image analysis results to identify important elements on the page (e.g., product name, price, purchase button, etc.). Generative AI is used for the analysis to classify and organize the data.

[0636] Step 5:

[0637] The server generates user interface (UI) data based on the identified information, converting it into a user-friendly format, including highlighting important information and arranging action buttons to simplify operation.

[0638] Step 6:

[0639] The server transmits the generated UI data to the terminal.

[0640] Step 7:

[0641] The device displays the received UI data to the user, who then considers what action to take next.

[0642] Step 8:

[0643] The user looks at the displayed UI and inputs their request by voice or text. For example, they can say, "I would like to purchase this product."

[0644] Step 9:

[0645] The device uses an emotion engine to recognize emotions from the user's voice or text, then uses voice recognition technology to convert the voice instructions into text and natural language processing technology to analyze the text data. This analysis then identifies specific actions (e.g., checkout).

[0646] Step 10:

[0647] The device sends the analysis results and the user's emotional state to the server.

[0648] Step 11:

[0649] The server generates a corresponding web request based on the received action instruction and performs the required operation, such as logging in to an auction site and proceeding with the process of purchasing an item.

[0650] Step 12:

[0651] The server prompts the user for additional information (e.g., shipping address or payment information) if necessary.

[0652] Step 13:

[0653] The terminal notifies the user of a request for additional information from the server and prompts the user to enter the required information.

[0654] Step 14:

[0655] The user enters any additional information requested and sends it to the terminal.

[0656] Step 15:

[0657] The terminal sends any additional information entered by the user to the server.

[0658] Step 16:

[0659] The server uses the additional information received to complete any necessary operations, such as entering payment information and finalizing the order.

[0660] Step 17:

[0661] The server notifies the terminal that the operation is complete.

[0662] Step 18:

[0663] The device receives this notification and displays it to the user, for example, "Purchase procedure completed."

[0664] Step 19:

[0665] The emotion engine continuously monitors the user's emotions while using the service and provides appropriate feedback and support to the user depending on the situation.

[0666] The above is the specific processing flow of this system. This allows users to smoothly use the services they need without having to worry about complicated procedures or an unfriendly UI. In addition, by combining it with an emotion engine, it is possible to provide optimal support that is tailored to the user's emotions.

[0667] Example 2

[0668] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0669] When users use web services, they often find the operation complicated or difficult to understand. This makes it difficult to use, especially for users with low IT literacy or who face language barriers. There is a need to solve these problems and enable users to use services smoothly. Furthermore, by taking user emotions into consideration, it is necessary to provide more appropriate support and improve the user experience.

[0670] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0671] In this invention, the server includes means for a user to input a service address or take a picture of the screen with a camera, means for analyzing the input address or the captured image and identifying the structure and content of the service, means for converting the identified information into an easy-to-view and understandable user interface, means for the user to communicate their request by voice or text, means for analyzing the user's emotional state using an emotion engine, means for analyzing the request and performing an appropriate operation, and means for notifying the user of the completion of the performed operation. This not only allows users to easily use web services, but also makes it possible to provide optimal support through emotion recognition.

[0672] "User" refers to any individual or entity that uses the Web Services or Applications.

[0673] An "address" is a series of character strings that indicates unique location information for identifying access to a web service.

[0674] A "camera" is a device for taking pictures and storing them as digital data.

[0675] "Image data" is digital information representing a still image captured by a device such as a camera.

[0676] "Analysis" is the process of processing input data (addresses or image data) to extract information to understand its structure and content.

[0677] A "user interface" is the visual and operational interface through which a user interacts with a computer system or application.

[0678] "Voice" refers to the words and instructions given by the user captured as sound.

[0679] "Text" is a document or instruction created by combining letters and symbols.

[0680] An "emotion engine" is a function or software that analyzes a user's emotional state from voice or text.

[0681] "Natural language processing" is a technology that allows computers to understand, analyze, and generate human language.

[0682] An "action" refers to a specific operation or behavior that a system performs based on a user's instructions.

[0683] "Operation completed" is a state indicating that the action instructed by the user has been executed successfully.

[0684] "Notifications" are messages or alerts that inform the user about the completion of an operation or the status of the system.

[0685] A "system" is a collection of multiple pieces of hardware and software that work together to achieve a specific function or purpose.

[0686] "Structure" refers to the internal layout and arrangement of elements of a web page or service, and their relationships to one another.

[0687] "Content" refers to information such as text, images, links, and forms on a web page or service.

[0688] This invention is a system that reduces the complexity and difficulty of understanding when users use web services. This system has the function of recognizing the user's emotions by combining it with an emotion engine and providing more appropriate support.

[0689] Hardware and software used:

[0690] The system uses a smartphone app, cloud server, emotion engine, image analysis technology, natural language processing technology, etc. By combining these technologies, it supports user operations.

[0691] The specific process flow:

[0692] The server receives the address entered by the user using the smartphone app or the captured image data. If an address is entered, the server retrieves the HTML data of the corresponding web page. If captured image data is sent, the server uses image analysis technology to identify the text information and screen structure within the image.

[0693] The server then analyzes the acquired HTML data and image analysis results to identify important elements on the page (e.g., product name, price, purchase button). Based on this information, it uses a generative AI model to convert it into a user interface that is easy for the user to understand. The converted UI data is then sent to the device.

[0694] The user looks at the new UI and inputs their request by voice or text. For example, they might say, "I want to buy this product." At this time, the emotion engine recognizes the emotion from the user's voice or text and analyzes that information.

[0695] The device uses an emotion engine to analyze the user's emotional state, speech recognition technology to convert voice instructions into text, and natural language processing technology to analyze the text data, convert it into a specific action (e.g., purchase procedure), and send it to the server.

[0696] Based on the parsed action, the server generates the appropriate web request and performs the service operation on behalf of the user. For example, the user logs in with their account information, clicks a purchase button, and enters payment information. If necessary, the server prompts the user for additional information (for example, shipping address or payment information). The user provides the required information through the app.

[0697] Once the operation is complete, the server notifies the device, which then displays a message to the user saying, "The purchase has been completed."

[0698] The emotion engine also continuously monitors the user's emotions while using the service, recognizing when the user is feeling stressed or confused. The device uses the information from the emotion engine to provide appropriate feedback and additional support to the user, such as providing detailed instructions and hints on how to operate the device, improving user convenience.

[0699] Examples:

[0700] Example prompt sentence:

[0701] URL: https: / / example-auctionsite.com / product123

[0702] Take a screenshot and transform the UI to make it easier for users to find the purchase button.

[0703] Through these steps, the system enables users to use web services efficiently and easily, and provides optimal support through emotion recognition, making it possible to provide an easy-to-use environment even for users with low IT literacy or who face language barriers.

[0704] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0705] Step 1:

[0706] The user opens the smartphone app and enters the address of the service, or takes a screenshot using the camera. The input can be in the form of a URL or a photo. For example, if a user wants to access a product page on an auction site, they can enter the URL of that product page or take a screenshot. The input data is sent to the device.

[0707] Step 2:

[0708] The device sends the URL entered by the user or the captured image data to a server on the cloud. The device converts the entered information into the appropriate format (URL or image data) and sends it to the server. Specifically, it sends the URL information or image data via an Internet connection.

[0709] Step 3:

[0710] The server retrieves the HTML data of the web page based on the received URL. If a URL is entered, the server accesses the address and downloads the HTML data. If captured image data is sent, the server uses image analysis technology to identify the text information and screen structure within the image.

[0711] Step 4:

[0712] The server analyzes the HTML data and image analysis results to identify important elements on the page (e.g., product name, price, buy button). The server extracts this information using an HTML parser and image analysis algorithms.

[0713] Step 5:

[0714] The server uses a generative AI model to convert the data into a user interface that is easy for the user to understand. Based on the analysis results, the server changes the layout to a format that is visually easy for the user to understand, and sends the new UI data to the device.

[0715] Step 6:

[0716] When users see the new UI displayed on their device, they can input their request by voice or text. For example, they can say, "I want to purchase this product." The device then receives the user's voice or text.

[0717] Step 7:

[0718] The device uses an emotion engine to analyze the user's emotional state, converts voice instructions into text using voice recognition technology, and analyzes the text data using natural language processing technology. Specific actions are then taken as a result of the analysis.

[0719] Step 8:

[0720] The terminal then sends the analyzed specific action to the server. For example, an action such as "click the purchase button" or "enter payment information" is generated and sent to the server.

[0721] Step 9:

[0722] The server generates a corresponding web request based on the action it receives: for example, logging in with a specified account, clicking a purchase button, entering payment information, etc. If necessary, the server requests additional information (e.g., a shipping address).

[0723] Step 10:

[0724] When the operation is complete, the server notifies the device, which then notifies the user by displaying a message such as "The purchase procedure has been completed."

[0725] Step 11:

[0726] The emotion engine continuously monitors the user's emotions while using the service. It recognizes when the user feels stressed or confused. The device uses the information from the emotion engine to provide appropriate feedback and additional support. For example, it can support the user's operations by displaying detailed instructions or hints on how to operate the device.

[0727] (Application example 2)

[0728] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0729] When using conventional web services, users often face complex operations and difficulties in understanding them, posing significant barriers, particularly for users with low IT literacy or who speak a different language. Furthermore, lack of appropriate support often prevents users from identifying their emotions, increasing stress and confusion and impairing the user experience. The objective of this invention is to solve these problems and provide a more intuitive and user-friendly user interface and appropriate feedback based on the user's emotions, enabling users to use web services more comfortably.

[0730] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0731] In this invention, the server includes: means for a user to input a service URL or take a picture of the screen with a camera; means for analyzing the input URL or the captured image and identifying the structure and content of the service; means for converting the identified information into an easy-to-view and understandable UI; means for the user to communicate their request by voice or text; means for analyzing the request and performing an appropriate operation; means for notifying the user of the completion of the executed operation; means for analyzing the user's emotions using an emotion recognition engine and providing feedback based on the analysis results; and means for generating and responding to the user's request with an appropriate prompt sentence using a generative AI model. This not only simplifies complex operations for the user, but also enables the user to receive appropriate support tailored to their emotions.

[0732] "Service URL" means the web address of the online service that a User wishes to access.

[0733] "Taking a photo with a camera" refers to the act of recording the screen of the service the user wants to use as a digital image.

[0734] "Analysis" is the act of taking input data, understanding its structure and content, and extracting information.

[0735] "Identification" refers to the act of extracting and identifying important elements from the analyzed information.

[0736] An "easy-to-see and understand UI" is a user interface that users can easily understand and operate.

[0737] "Voice or text request" means the act of a user giving instructions to a system using voice input or text messages.

[0738] "Performing appropriate operations" refers to the system taking specific actions in response to the user's request.

[0739] "Notifying the completion of an operation" refers to the act of notifying the user that an operation performed by the system has been completed.

[0740] An "emotion recognition engine" is a software technology for analyzing emotions from a user's voice or text.

[0741] "Providing feedback based on analysis results" refers to the act of providing appropriate information and support to users based on the analysis results of the emotion recognition engine.

[0742] A "generative AI model" is a technology that uses artificial intelligence to generate appropriate outputs in response to specific prompts.

[0743] "Generating appropriate prompts" refers to the act of creating the most appropriate questions or instructions based on the user's request.

[0744] This invention is a system that reduces the complexity and difficulty of understanding when users use web services. By combining an emotion recognition engine and a generative AI model, this system includes a function that recognizes user emotions and provides more appropriate support.

[0745] First, users enter the URL of the web service they want to use through a smartphone app, or use the camera to take a screenshot of the screen of the service they want to use. For example, if a user wants to watch content from a specific video streaming service, they can enter the URL of the content page or take a screenshot of the screen.

[0746] Next, the device sends the URL entered by the user or the captured image data to a server in the cloud. If a URL is entered, the URL information is sent, and if an image is captured, the image data is sent. The server retrieves the HTML data of the webpage based on the received URL, or analyzes the image data to extract the screen structure. The software used here is a web scraper and image analysis technology.

[0747] The server analyzes the acquired HTML data and image analysis results to identify important elements on the page (e.g., content title, play button, description), converts them into a user-friendly UI using a generative AI model, and sends the converted UI data to the device.

[0748] When users use the new UI to perform specific actions by voice or text, the application uses an emotion recognition engine to analyze the user's emotional state and converts the voice commands into text using speech recognition technology, specifically a speech recognition library and natural language processing technology.

[0749] The server generates a corresponding web request based on the analyzed action and performs the service operation on behalf of the user. First, to play content, for example, the user logs in with their account information and then clicks the play button. After playback is complete, the server notifies the device that the operation is complete.

[0750] The emotion recognition engine also continuously monitors the user's emotions while using the service, recognizing when the user is feeling stressed or confused. For example, if the user verbally requests "I want to know more," the generative AI model will generate the following prompt:

[0751] "Example prompt statement:

[0752] User input: "I want to know more"

[0753] The generative AI model prompts: "Please provide more details about the content the user is looking for."

[0754] Expected output: "Here's a detailed description of this content. Let me know if you need more information."

[0755] In this way, this system provides the detailed information and operating procedures that users need, improving user convenience. In particular, by combining it with an emotion recognition engine, it is possible to provide optimal feedback based on the user's emotional state. As a result, users can use web services more comfortably.

[0756] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0757] Step 1:

[0758] Users can either enter the URL of the web service they want to use using a smartphone app, or take a photo of the service screen using the camera. The input data is a text URL, and the photo data is an image file.

[0759] Step 2:

[0760] The device sends the URL entered in step 1 or the captured image data to a server on the cloud. The sent data is analyzed on the server side.

[0761] Step 3:

[0762] The server retrieves the HTML data of the web page based on the received URL, or analyzes the image data to extract the screen structure. Specifically, it retrieves the HTML data using a web scraper and uses image analysis technology to identify the text information and screen structure within the image.

[0763] Step 4:

[0764] The server identifies important elements on the page (e.g., content title, play button, description) from the analyzed HTML data and image analysis results. In this process, necessary information is extracted from the analyzed data and converted into an easy-to-read and understandable UI using a generative AI model.

[0765] Step 5:

[0766] The terminal displays the converted UI data sent from the server to the user, who then looks at the displayed UI and inputs their next request by voice or text.

[0767] Step 6:

[0768] The device converts input voice instructions into text using a speech recognition library and analyzes the user's emotional state using an emotion recognition engine. Here, the input data is an audio file, and the output data is text and emotional state information.

[0769] Step 7:

[0770] The server uses a generative AI model to generate appropriate feedback and prompts based on the emotional state information obtained from the emotion recognition engine, and the generated prompts are returned to the user as appropriate instructions or additional information.

[0771] Step 8:

[0772] The server performs the operation requested by the user, prompting the user for additional information as needed (e.g., the user's account information or payment information). This is where data processing and calculations are performed to generate and fulfill the web request.

[0773] Step 9:

[0774] The device notifies the user of the request sent from the server, prompts the user to enter the required information, and then sends it back to the server. This interaction allows the user to provide additional information.

[0775] Step 10:

[0776] The server confirms the completion of the operation and notifies the terminal of the completion. The terminal then displays a notification of the operation completion to the user, informing the user that the operation has been completed successfully. For example, a message such as "The playback procedure has been completed" may be displayed.

[0777] Through these steps, users are freed from cumbersome operations and can use web services while receiving appropriate feedback based on their emotions. Prompt sentence generation using a generative AI model enables users to use services intuitively.

[0778] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0779] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0780] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0781] [Third embodiment]

[0782] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0783] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0784] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0785] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0786] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0787] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0788] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0789] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0790] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0791] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0792] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0793] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0794] First, the user installs and launches the smartphone app, which is designed to assist in the use of web services and provides users with intuitive and easy operation.

[0795] Screen Input

[0796] Users can either enter the URL of the service they want to use through the app's interface or take a screenshot of the service's screen using the app's camera. For example, if a user wants to access a product page on an overseas auction site, they can enter the URL of the product page or take a screenshot of the screen.

[0797] Data transmission and analysis

[0798] The device sends the URL entered by the user and the captured image data to a server on the cloud. The server then accesses the received URL and retrieves the HTML data. It also uses image analysis technology (such as OCR) to extract the screen structure and text information from the captured image data.

[0799] The server analyzes the acquired HTML data or image analysis results to identify the page structure and elements. This includes using generative AI to recognize and classify appropriate information, such as the product name, price, and the location of the purchase button.

[0800] UI transformation

[0801] The server then converts the UI into a format that is intuitively easy for users to understand based on the analysis results. Specifically, it highlights important information and arranges action buttons to simplify user operations. The easy-to-read UI data generated here is sent to the device and displayed to the user.

[0802] Collecting user requests

[0803] Users can view the converted UI and then specify actions by voice or text, for example, "I want to buy this product."

[0804] The device uses voice recognition technology to convert the user's voice instructions into text data, and then uses natural language processing technology to analyze the instructions. The results of this analysis are sent to the server as a specific action (e.g., purchasing procedure).

[0805] Service agency

[0806] The server generates and executes the corresponding web request based on the parsed action, which may include the user logging in with their account information, clicking a purchase button, or entering payment information.

[0807] If necessary, the server will prompt the user for additional information (e.g., shipping address or payment information). The device will notify the user of this request, prompt them to enter the required information, and then send it back to the server.

[0808] Operation completion notification

[0809] When the proxy operation is complete, the server notifies the terminal of the completion. The terminal displays the received notification to the user, informing them that all operations have been completed. For example, it displays a message such as "The purchase procedure has been completed."

[0810] This allows users to easily access the services they need without having to deal with complicated operations or an unfriendly UI. This system is particularly convenient for users with low IT literacy or who face language barriers.

[0811] The processing flow will be explained below.

[0812] Step 1:

[0813] Users launch a smartphone app and enter the URL of the web service they want to use, or take a picture of the screen using the camera. For example, consider the case where a user enters the URL of a product page on an overseas auction site.

[0814] Step 2:

[0815] The device receives the URL entered by the user or the captured image data and sends it to a server on the cloud. If a URL is entered, the URL information is sent, and if an image is captured, the image data is sent.

[0816] Step 3:

[0817] The server retrieves the HTML data of the web page based on the received URL, and if image data is sent, it uses image analysis technology to extract text information and screen structure from the image.

[0818] Step 4:

[0819] The server performs analysis using the acquired HTML data and image analysis results. Specifically, it uses generative AI to identify important elements on the page (e.g., product name, price, purchase button, etc.).

[0820] Step 5:

[0821] The server generates UI data based on the identified information, converting it into a format that is easy for the user to understand. This UI data includes highlighting important information and arranging buttons for easy user operation.

[0822] Step 6:

[0823] The server transmits the generated UI data to the terminal.

[0824] Step 7:

[0825] The device displays the received UI data to the user, allowing the user to consider their next action while looking at the displayed UI.

[0826] Step 8:

[0827] The user inputs their request by voice or text, for example, "I would like to purchase this product."

[0828] Step 9:

[0829] The device uses voice recognition technology to convert the user's voice instructions into text, and then uses natural language processing technology to analyze the text data and understand it as a specific action (e.g., checkout).

[0830] Step 10:

[0831] The device then sends the analyzed user request to the server, and as a specific action, sends a purchase procedure request to the server.

[0832] Step 11:

[0833] Based on the received action instruction, the server generates a corresponding web request and performs the required operation, for example, logging in to an auction site and initiating the process of purchasing an item.

[0834] Step 12:

[0835] The server will prompt the user for additional information as needed, for example a shipping address or payment information.

[0836] Step 13:

[0837] The terminal notifies the user of a request for additional information from the server and prompts the user to input the required information.

[0838] Step 14:

[0839] The user enters any additional information requested and transmits it to the terminal.

[0840] Step 15:

[0841] The terminal transmits the additional information entered by the user to the server.

[0842] Step 16:

[0843] The server uses the received additional information to complete any necessary operations, such as entering payment information and finally completing the purchase.

[0844] Step 17:

[0845] The server notifies the terminal that the operation is complete.

[0846] Step 18:

[0847] The device displays the received completion notification to the user to let them know that the operation has been completed, for example, by displaying a message such as "Purchase procedure completed."

[0848] The above is the specific flow of processing by this system.

[0849] Example 1

[0850] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0851] Modern web services require complex user interfaces and multilingual support, which can make them difficult to use, especially for users with low IT literacy or who face language barriers.Furthermore, differences in operating procedures between different web services can make users feel stressed about procedures and operations.

[0852] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0853] In this invention, the server includes: means for a user to input service information or take a screen shot; means for transmitting the input information or the captured image to a cloud-based processing device; means for analyzing the transmitted information and using a generative AI model to identify the structure and content of the service; means for converting the identified information into an intuitive user interface that is easy for the user to understand; means for the user to communicate requests by voice or text; means for analyzing the requests using natural language processing to identify specific operations; means for generating a web request to execute the identified operations; and means for notifying the user of the completion of the executed operations. This allows users to easily perform complex operations, improving convenience, particularly for users with low IT literacy and when using international services.

[0854] "User" means any person or entity that intends to use the System and / or the Services.

[0855] "Service information" refers to the specific URL and screen information related to the web service the user wants to use.

[0856] "Means for capturing screen images" refers to a function that uses a camera installed on a mobile device such as a smartphone or tablet to capture an image of the screen of the service being used.

[0857] A "cloud processing unit" is a computer resource that runs on a remote server accessible over the Internet.

[0858] "Means for transmitting" refers to the network function for transmitting data from the user's terminal to a processing device on the cloud.

[0859] A "generative AI model" is a machine learning model that is trained to perform a specific task based on a large dataset.

[0860] "User interface" refers to the screen and operating means that users use to operate the system.

[0861] "Natural language processing" refers to techniques and methods for analyzing natural language and converting it into meaningful information.

[0862] A "web request" refers to data that requests or sends information to a server on the Internet.

[0863] "Means of notification" means the functionality that allows the system to send information or messages to the user.

[0864] This invention provides a system that helps users easily use web services through smartphone apps. Specific embodiments of this system are described below.

[0865] First, users install and launch a dedicated application on their smartphone or tablet. This application can be downloaded from various app stores, including Google Play and the App Store. After launching the application, users can either enter the URL of the service they want to use in the "Enter URL" field, or use the camera to take a screenshot.

[0866] For example, if a user wants to purchase an item on an overseas auction site, they can enter the URL of the product page or take a photo of the product page with their camera. The app has a "Send" button, and when the user presses this button, the entered URL or the captured image is sent to a server in the cloud. The sent data is sent to the server in JSON format via an HTTPS request.

[0867] The server then accesses the received URL to retrieve the HTML data. It also uses OCR technology (e.g., Tesseract OCR) to extract text information from the captured image. The server processes this data using libraries such as Python requests, BeautifulSoup, and Selenium.

[0868] The server then uses the acquired HTML data and image analysis results to identify the page structure and elements. At this stage, a generative AI model (e.g., OpenAI GPT) is used to recognize and classify relevant information such as product names, prices, and the location of purchase buttons. Based on the analysis results, a user interface (UI) is created that is intuitive and easy for users to understand. The server generates the UI using front-end libraries such as D3.js and React, and sends the generated UI data to the user's device.

[0869] The device displays a user interface based on the received data. While viewing this interface, the user can use voice or text input functions to specify specific actions. For example, they can give voice commands such as "I want to purchase this product," or manually enter text into a text field. Voice input is converted into text data using speech recognition APIs such as Google Cloud Speech-to-Text and IBM Watson.

[0870] The device then uses natural language processing technology (e.g., spaCy) to analyze the user's instructions and identify specific operations. The analysis results are then sent back to the server, which then generates and executes the corresponding web requests. Specifically, the server uses tools such as Selenium to automatically operate the browser and complete the necessary procedures.

[0871] Finally, the server notifies the device that the operation is complete, and the device displays the notification to the user, for example, by displaying a message saying "Purchase procedure completed" to let the user know that all operations were completed successfully.

[0872] Here are some examples of prompts:

[0873] "User: I would like to purchase this item from Amazon.

[0874] URL:https: / / www.example.com / example-product

[0875] Camera: A screenshot of your screen has been uploaded.

[0876] This system will allow users to easily perform complicated operations, greatly improving convenience, especially for users with low IT literacy and those using international services.

[0877] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0878] Step 1: User Actions

[0879] Users install and launch a dedicated application on their smartphone or tablet. Then, they can enter the URL of the service they want to use or take a screenshot of the screen with their camera. Specifically, users download the application from Google Play or the App Store, open the application, and enter the URL in the "Enter URL" field or tap the camera icon to take a screenshot of the screen.

[0880] Input: URL or image data

[0881] Output: URLs and image data prepared within the application

[0882] Step 2: Sending data

[0883] The device sends the URL entered by the user and the captured image data to a server on the cloud. Specifically, when the user taps the "Send" button, the data is sent to the server in JSON format via an HTTPS request.

[0884] Input: URL or image data ready

[0885] Output: The URL and image data are sent to the server and accepted by the server.

[0886] Step 3: Get the URL and analyze the image

[0887] The server accesses the sent URL and retrieves the HTML data. It also uses Tesseract OCR technology to extract text information from the captured image data. Specifically, the server uses the Python requests library to send a GET request to the URL and uses Tesseract OCR to retrieve text information from the image.

[0888] Input: URL or image data sent to the server

[0889] Output: HTML data or parsed text information

[0890] Step 4: Analyzing the Page

[0891] The server analyzes the acquired HTML data and image analysis results to identify the structure and elements of the web page. At this stage, a generative AI model (e.g., OpenAI GPT) is used to recognize and classify relevant information such as the product name, price, and purchase button location. Specifically, the server parses the HTML data using BeautifulSoup or Selenium and organizes the information using generative AI.

[0892] Input: HTML data or text information

[0893] Output: Identified product information (name, price, button position, etc.)

[0894] Step 5: Transforming the UI

[0895] The server generates a user interface (UI) based on the analysis results, which is easy for users to understand intuitively. Specifically, the server visualizes the analysis results using D3.js and React, and sends the generated UI data in JSON format to the device.

[0896] Input: Identified product information

[0897] Output: Generated UI data

[0898] Step 6: Input user requests

[0899] The user can then view the converted UI and input a specific action using voice or text, such as "I want to buy this product." To achieve this, the user can tap the voice input button in the app to input voice or type directly into a text field.

[0900] Input: Generated UI data

[0901] Output: User's voice commands or text input

[0902] Step 7: Parsing the instructions

[0903] The device converts the voice instructions used into text using Google Cloud Speech-to-Text or IBM Watson, and then analyzes them using natural language processing technology (e.g., spaCy). As a result of the analysis, specific operation instructions are identified and sent to the server. Specific operations include sending the voice data to an API, converting it into text, and identifying actions based on the analyzed text.

[0904] Input: User's voice commands or text input

[0905] Output: Specific operation instructions sent to the server

[0906] Step 8: Delegate the service

[0907] The server generates and executes web requests based on the user's actions, such as logging in to an account, clicking a purchase button, or entering payment information. The server then uses Selenium to automatically perform the necessary steps.

[0908] Input: Specific operating instructions

[0909] Output: The result of the web request that was made.

[0910] Step 9: Operation completion notification

[0911] The server sends a notification to the device when the proxy operation is complete. The device displays the received notification to the user. For example, it displays a message saying "The purchase procedure has been completed." Specifically, the server sends the operation completion status to the device in JSON format, and the device receives it and displays it in the app.

[0912] Input: The result of the web request that was executed

[0913] Output: Notification to the user that the operation was completed

[0914] (Application example 1)

[0915] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0916] When using overseas web services, especially food delivery services, users often have difficulty with different languages ​​and complex UIs. Due to language barriers and a lack of IT literacy, many users are unable to use the services easily and intuitively.

[0917] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0918] In this invention, the server includes a means for accessing a URL entered by a user and acquiring HTML data, a means for analyzing captured image data to extract the screen structure, and a means for automatically translating information displayed in different languages ​​and converting it into a language that is easy for the user to understand. This allows users to intuitively use overseas food delivery services without having to deal with language or complex UIs.

[0919] "Service URL" refers to the Internet address of the web service that a User wishes to use.

[0920] "Taking a picture of the screen with a camera" refers to the act of a user using the camera function of a smartphone or other device to save the display screen of a web service as image data.

[0921] "Identified information" refers to data about the structure and content of web services obtained by analyzing URLs and image data.

[0922] "Easy-to-see and easy-to-understand UI" refers to an interface that is designed to be easy for users to understand and operate.

[0923] "Communicating requests by voice or text" refers to the act of a user using voice input or text input to communicate desired operations or instructions to the system.

[0924] "Performing appropriate operations" refers to the act of analyzing the user's request and having the system take the necessary actions on the web service based on those instructions.

[0925] "Notifying the user of the completion of an executed operation" refers to the act of notifying the user that an operation performed by the system based on the user's instructions has been completed.

[0926] "Automatically translating information displayed in different languages" refers to the process of automatically converting content displayed by a web service into a language that the user can understand, when that content is not in the user's native language.

[0927] "Providing services and products available in a specific area" refers to the act of presenting users with information about services and products available within a specific area based on the user's location or designated area.

[0928] First, the user installs and launches the smartphone app, which is designed to assist in the use of web services and provides users with intuitive and easy operation.

[0929] Screen Input

[0930] Users can either enter the URL of the service they want to use through the app interface or take a screenshot of the service screen using the camera. For example, if a user wants to use an overseas food delivery service, they can enter the URL of the service or take a screenshot of the screen.

[0931] Data transmission and analysis

[0932] The device sends the URL entered by the user or the captured image data to a server on the cloud. The server then accesses the received URL and retrieves the HTML data. It also uses image analysis technology (such as OCR) to extract the screen structure and text information from the captured image data. The server then analyzes the retrieved HTML data or image analysis results to identify the page structure and elements. This includes utilizing generative AI to recognize and classify appropriate information. For example, it identifies menu names, prices, and the position of the order button.

[0933] UI transformation

[0934] Based on the analysis results, the server converts the UI into a format that is intuitively easy for users to understand. Specifically, it highlights important information and places action buttons to simplify user operations. It also automatically translates information displayed in different languages ​​and converts it into a language that is easy for users to understand. The easy-to-read UI data generated here is sent to the device and displayed to the user.

[0935] Collecting user requests

[0936] While viewing the converted UI, the user can specify actions by voice or text. For example, the user can say, "I would like to order this dish." The device uses voice recognition technology to convert the user's voice instructions into text data, and then uses natural language processing technology to analyze the instructions. The results of this analysis are sent to the server as a specific action (e.g., placing an order).

[0937] Service agency

[0938] The server generates and executes the corresponding web request based on the parsed action. This may include logging in with the user's account information, clicking an order button, or entering payment information. If necessary, the server prompts the user for additional information (e.g., shipping address or payment information). The device notifies the user of this request, prompts them to enter the required information, and then sends it back to the server.

[0939] Operation completion notification

[0940] The server notifies the terminal when the proxy operation is complete. The terminal then displays the received notification to the user, informing them that all operations have been completed. For example, it displays a message such as "The order procedure has been completed." This allows the user to easily use the services they need without having to struggle with complicated operations or an unfriendly UI.

[0941] Examples and prompts

[0942] For example, if a user wants to use an overseas food delivery service, they can enter the following prompt into the app:

[0943] Example prompt sentence:

[0944] "https: / / example-fooddeli.com"

[0945] or

[0946] "https: / / example-image.com / screenshot.png"

[0947] This allows users to use the service intuitively without having to deal with different languages ​​or complex operations, and the system offers great convenience, especially for users with low IT literacy or who face language barriers.

[0948] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0949] Step 1:

[0950] The user launches the smartphone app and either enters the URL of the service they want to use or takes a screenshot of the service's screen using the camera. This entered URL or captured screen image becomes the input data for processing.

[0951] Input data: URL or image

[0952] Output data: Input URL or image data

[0953] Step 2:

[0954] The device sends the URL entered by the user or the captured image data to a server on the cloud. This sending operation passes the input data directly to the server.

[0955] Input data: User's URL or image data

[0956] Output data: Transmission to server completed

[0957] Step 3:

[0958] The server accesses the received URL and retrieves the HTML data. It also uses OCR technology to analyze the screen structure and text information of the captured image data. Specifically, it uses pytesseract to extract text from the image and BeautifulSoup to analyze the HTML.

[0959] Input data: URL or image data

[0960] Output data: HTML data or parsed text information

[0961] Step 4:

[0962] The server analyzes the acquired HTML data or image analysis results to identify the page structure and elements. Generative AI technology is then used to properly recognize the identified information and classify important information. For example, menu names, prices, and the location of the order button are identified.

[0963] Input data: HTML data or parsed text information

[0964] Output data: Structure and element information of identified pages

[0965] Step 5:

[0966] The server converts the UI into a format that is intuitively easy for users to understand based on the analysis results. During this process, it highlights important information and places action buttons to simplify user operations. In addition, it automatically translates information displayed in different languages ​​and converts it into a language that is easy for users to understand.

[0967] Input data: Identified page structure and element information

[0968] Output data: Transformed UI data

[0969] Step 6:

[0970] The device displays the converted UI data received from the server to the user, who can then instruct actions via voice or text while viewing the UI.

[0971] Input data: Transformed UI data

[0972] Output data: User's voice or text instructions for action

[0973] Step 7:

[0974] The device uses voice recognition technology to convert the user's voice instructions into text data, and then uses natural language processing technology to analyze the instructions. This is where libraries such as Spacy are used. The results of this analysis are sent to the server as a specific action (e.g., ordering food).

[0975] Input data: User's voice or text instructions

[0976] Output data: Parsed instructions

[0977] Step 8:

[0978] The server generates and executes the corresponding web requests based on the parsed user instructions, which may include logging in with the user's account information, clicking an order button, or entering payment information.

[0979] Input data: Parsed instructions

[0980] Output data: Status of the operation performed

[0981] Step 9:

[0982] The server notifies the terminal when the executed operation is completed. The terminal displays the received notification to the user to inform them that all operations have been completed. For example, it displays a message such as "The order procedure has been completed."

[0983] Input data: Status of the operation performed

[0984] Output data: Completion notification to the user

[0985] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0986] This invention is a system that reduces the complexity and difficulty of understanding that users face when using web services. This system includes a function that recognizes the user's emotions and provides more appropriate support by combining it with an emotion engine.

[0987] Screen Input

[0988] Users can either enter the URL of the web service they want to use through a smartphone app or take a screenshot of the service's screen using the camera. For example, if a user wants to access a product page on an overseas auction site, they can enter the URL of that product page or take a screenshot.

[0989] Data transmission and analysis

[0990] The device sends the URL entered by the user and the captured image data to a server on the cloud. If a URL is entered, the URL information is sent, and if an image is captured, the image data is sent.

[0991] The server retrieves the HTML data of the web page based on the received URL, or analyzes the image data to extract the screen structure. Image analysis technology is used to identify the text information and screen structure within the image.

[0992] UI transformation

[0993] The server analyzes the acquired HTML data and image analysis results to identify important elements on the page (e.g., product name, price, purchase button), and uses generative AI to convert them into a user-friendly UI, which is then sent to the device.

[0994] Utilizing the Emotion Engine

[0995] When users see the UI displayed on their device, they can input their request by voice or text. For example, they can give instructions such as "I would like to purchase this product." At this time, the emotion engine recognizes the user's emotions from the voice or text.

[0996] The device uses an emotion engine to analyze the user's emotional state, speech recognition technology to convert voice instructions into text, and natural language processing technology to analyze the text data and send specific actions (e.g., purchase procedures) to the server.

[0997] Service agency

[0998] Based on the parsed actions, the server generates corresponding web requests and performs service operations on behalf of the user: first, logging in with their account information, then clicking the purchase button and entering payment information.

[0999] If necessary, the server will prompt the user for additional information (e.g., shipping address or payment information). The device will notify the user of the request, prompt them to enter the required information, and then send it back to the server.

[1000] Operation completion notification

[1001] When the operation is complete, the server notifies the device. The device then notifies the user of the completion of the operation, for example by displaying a message such as "The purchase procedure has been completed."

[1002] Emotion-Based Feedback

[1003] The emotion engine continuously monitors the user's emotions while using the service and recognizes when the user is feeling stressed or confused. The device uses the information from the emotion engine to provide appropriate feedback and additional support to the user, such as detailed instructions or hints on how to operate the device, improving user convenience.

[1004] In this way, the system allows users to use the service without difficulty and provides a more comfortable experience. In particular, by combining it with an emotion engine, it is possible to take the user's emotions into consideration and provide optimal support. As a result, it is possible to provide excellent convenience even to users with low IT literacy or who experience language barriers.

[1005] The processing flow will be explained below.

[1006] Step 1:

[1007] Users launch a smartphone app and either enter the URL of the web service they want to use or take a screenshot of the screen of the service they want to use using the camera. For example, they can enter the URL or take a screenshot to access the product page of an overseas auction site.

[1008] Step 2:

[1009] The device receives the URL entered by the user or the captured image data and sends it to a server in the cloud. The URL information and image data are sent to the server securely.

[1010] Step 3:

[1011] The server retrieves the HTML data of the web page based on the received URL, or analyzes the received image data to extract the text and screen structure within the image. Image analysis technology is used to perform character recognition (OCR) and structure recognition.

[1012] Step 4:

[1013] The server performs detailed analysis of the acquired HTML data and image analysis results to identify important elements on the page (e.g., product name, price, purchase button, etc.). Generative AI is used for the analysis to classify and organize the data.

[1014] Step 5:

[1015] The server generates user interface (UI) data based on the identified information, converting it into a user-friendly format, including highlighting important information and arranging action buttons to simplify operation.

[1016] Step 6:

[1017] The server transmits the generated UI data to the terminal.

[1018] Step 7:

[1019] The device displays the received UI data to the user, who then considers what action to take next.

[1020] Step 8:

[1021] The user looks at the displayed UI and inputs their request by voice or text. For example, they can say, "I would like to purchase this product."

[1022] Step 9:

[1023] The device uses an emotion engine to recognize emotions from the user's voice or text, then uses voice recognition technology to convert the voice instructions into text and natural language processing technology to analyze the text data. This analysis then identifies specific actions (e.g., checkout).

[1024] Step 10:

[1025] The device sends the analysis results and the user's emotional state to the server.

[1026] Step 11:

[1027] The server generates a corresponding web request based on the received action instruction and performs the required operation, such as logging in to an auction site and proceeding with the process of purchasing an item.

[1028] Step 12:

[1029] The server prompts the user for additional information (e.g., shipping address or payment information) if necessary.

[1030] Step 13:

[1031] The terminal notifies the user of a request for additional information from the server and prompts the user to enter the required information.

[1032] Step 14:

[1033] The user enters any additional information requested and sends it to the terminal.

[1034] Step 15:

[1035] The terminal sends any additional information entered by the user to the server.

[1036] Step 16:

[1037] The server uses the additional information received to complete any necessary operations, such as entering payment information and finalizing the order.

[1038] Step 17:

[1039] The server notifies the terminal that the operation is complete.

[1040] Step 18:

[1041] The device receives this notification and displays it to the user, for example, "Purchase procedure completed."

[1042] Step 19:

[1043] The emotion engine continuously monitors the user's emotions while using the service and provides appropriate feedback and support to the user depending on the situation.

[1044] The above is the specific processing flow of this system. This allows users to smoothly use the services they need without having to worry about complicated procedures or an unfriendly UI. In addition, by combining it with an emotion engine, it is possible to provide optimal support that is tailored to the user's emotions.

[1045] Example 2

[1046] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1047] When users use web services, they often find the operation complicated or difficult to understand. This makes it difficult to use, especially for users with low IT literacy or who face language barriers. There is a need to solve these problems and enable users to use services smoothly. Furthermore, by taking user emotions into consideration, it is necessary to provide more appropriate support and improve the user experience.

[1048] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1049] In this invention, the server includes means for a user to input a service address or take a picture of the screen with a camera, means for analyzing the input address or the captured image and identifying the structure and content of the service, means for converting the identified information into an easy-to-view and understandable user interface, means for the user to communicate their request by voice or text, means for analyzing the user's emotional state using an emotion engine, means for analyzing the request and performing an appropriate operation, and means for notifying the user of the completion of the performed operation. This not only allows users to easily use web services, but also makes it possible to provide optimal support through emotion recognition.

[1050] "User" refers to any individual or entity that uses the Web Services or Applications.

[1051] An "address" is a series of character strings that indicates unique location information for identifying access to a web service.

[1052] A "camera" is a device for taking pictures and storing them as digital data.

[1053] "Image data" is digital information representing a still image captured by a device such as a camera.

[1054] "Analysis" is the process of processing input data (addresses or image data) to extract information to understand its structure and content.

[1055] A "user interface" is the visual and operational interface through which a user interacts with a computer system or application.

[1056] "Voice" refers to the words and instructions given by the user captured as sound.

[1057] "Text" is a document or instruction created by combining letters and symbols.

[1058] An "emotion engine" is a function or software that analyzes a user's emotional state from voice or text.

[1059] "Natural language processing" is a technology that allows computers to understand, analyze, and generate human language.

[1060] An "action" refers to a specific operation or behavior that a system performs based on a user's instructions.

[1061] "Operation completed" is a state indicating that the action instructed by the user has been executed successfully.

[1062] "Notifications" are messages or alerts that inform the user about the completion of an operation or the status of the system.

[1063] A "system" is a collection of multiple pieces of hardware and software that work together to achieve a specific function or purpose.

[1064] "Structure" refers to the internal layout and arrangement of elements of a web page or service, and their relationships to one another.

[1065] "Content" refers to information such as text, images, links, and forms on a web page or service.

[1066] This invention is a system that reduces the complexity and difficulty of understanding when users use web services. This system has the function of recognizing the user's emotions by combining it with an emotion engine and providing more appropriate support.

[1067] Hardware and software used:

[1068] The system uses a smartphone app, cloud server, emotion engine, image analysis technology, natural language processing technology, etc. By combining these technologies, it supports user operations.

[1069] The specific process flow:

[1070] The server receives the address entered by the user using the smartphone app or the captured image data. If an address is entered, the server retrieves the HTML data of the corresponding web page. If captured image data is sent, the server uses image analysis technology to identify the text information and screen structure within the image.

[1071] The server then analyzes the acquired HTML data and image analysis results to identify important elements on the page (e.g., product name, price, purchase button). Based on this information, it uses a generative AI model to convert it into a user interface that is easy for the user to understand. The converted UI data is then sent to the device.

[1072] The user looks at the new UI and inputs their request by voice or text. For example, they might say, "I want to buy this product." At this time, the emotion engine recognizes the emotion from the user's voice or text and analyzes that information.

[1073] The device uses an emotion engine to analyze the user's emotional state, speech recognition technology to convert voice instructions into text, and natural language processing technology to analyze the text data, convert it into a specific action (e.g., purchase procedure), and send it to the server.

[1074] Based on the parsed action, the server generates the appropriate web request and performs the service operation on behalf of the user. For example, the user logs in with their account information, clicks a purchase button, and enters payment information. If necessary, the server prompts the user for additional information (for example, shipping address or payment information). The user provides the required information through the app.

[1075] Once the operation is complete, the server notifies the device, which then displays a message to the user saying, "The purchase has been completed."

[1076] The emotion engine also continuously monitors the user's emotions while using the service, recognizing when the user is feeling stressed or confused. The device uses the information from the emotion engine to provide appropriate feedback and additional support to the user, such as providing detailed instructions and hints on how to operate the device, improving user convenience.

[1077] Examples:

[1078] Example prompt sentence:

[1079] URL: https: / / example-auctionsite.com / product123

[1080] Take a screenshot and transform the UI to make it easier for users to find the purchase button.

[1081] Through these steps, the system enables users to use web services efficiently and easily, and provides optimal support through emotion recognition, making it possible to provide an easy-to-use environment even for users with low IT literacy or who face language barriers.

[1082] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1083] Step 1:

[1084] The user opens the smartphone app and enters the address of the service, or takes a screenshot using the camera. The input can be in the form of a URL or a photo. For example, if a user wants to access a product page on an auction site, they can enter the URL of that product page or take a screenshot. The input data is sent to the device.

[1085] Step 2:

[1086] The device sends the URL entered by the user or the captured image data to a server on the cloud. The device converts the entered information into the appropriate format (URL or image data) and sends it to the server. Specifically, it sends the URL information or image data via an Internet connection.

[1087] Step 3:

[1088] The server retrieves the HTML data of the web page based on the received URL. If a URL is entered, the server accesses the address and downloads the HTML data. If captured image data is sent, the server uses image analysis technology to identify the text information and screen structure within the image.

[1089] Step 4:

[1090] The server analyzes the HTML data and image analysis results to identify important elements on the page (e.g., product name, price, buy button). The server extracts this information using an HTML parser and image analysis algorithms.

[1091] Step 5:

[1092] The server uses a generative AI model to convert the data into a user interface that is easy for the user to understand. Based on the analysis results, the server changes the layout to a format that is visually easy for the user to understand, and sends the new UI data to the device.

[1093] Step 6:

[1094] When users see the new UI displayed on their device, they can input their request by voice or text. For example, they can say, "I want to purchase this product." The device then receives the user's voice or text.

[1095] Step 7:

[1096] The device uses an emotion engine to analyze the user's emotional state, converts voice instructions into text using voice recognition technology, and analyzes the text data using natural language processing technology. Specific actions are then taken as a result of the analysis.

[1097] Step 8:

[1098] The terminal then sends the analyzed specific action to the server. For example, an action such as "click the purchase button" or "enter payment information" is generated and sent to the server.

[1099] Step 9:

[1100] The server generates a corresponding web request based on the action it receives: for example, logging in with a specified account, clicking a purchase button, entering payment information, etc. If necessary, the server requests additional information (e.g., a shipping address).

[1101] Step 10:

[1102] When the operation is complete, the server notifies the device, which then notifies the user by displaying a message such as "The purchase procedure has been completed."

[1103] Step 11:

[1104] The emotion engine continuously monitors the user's emotions while using the service. It recognizes when the user feels stressed or confused. The device uses the information from the emotion engine to provide appropriate feedback and additional support. For example, it can support the user's operations by displaying detailed instructions or hints on how to operate the device.

[1105] (Application example 2)

[1106] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1107] When using conventional web services, users often face complex operations and difficulties in understanding them, posing significant barriers, particularly for users with low IT literacy or who speak a different language. Furthermore, lack of appropriate support often prevents users from identifying their emotions, increasing stress and confusion and impairing the user experience. The objective of this invention is to solve these problems and provide a more intuitive and user-friendly user interface and appropriate feedback based on the user's emotions, enabling users to use web services more comfortably.

[1108] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1109] In this invention, the server includes: means for a user to input a service URL or take a picture of the screen with a camera; means for analyzing the input URL or the captured image and identifying the structure and content of the service; means for converting the identified information into an easy-to-view and understandable UI; means for the user to communicate their request by voice or text; means for analyzing the request and performing an appropriate operation; means for notifying the user of the completion of the executed operation; means for analyzing the user's emotions using an emotion recognition engine and providing feedback based on the analysis results; and means for generating and responding to the user's request with an appropriate prompt sentence using a generative AI model. This not only simplifies complex operations for the user, but also enables the user to receive appropriate support tailored to their emotions.

[1110] "Service URL" means the web address of the online service that a User wishes to access.

[1111] "Taking a photo with a camera" refers to the act of recording the screen of the service the user wants to use as a digital image.

[1112] "Analysis" is the act of taking input data, understanding its structure and content, and extracting information.

[1113] "Identification" refers to the act of extracting and identifying important elements from the analyzed information.

[1114] An "easy-to-see and understand UI" is a user interface that users can easily understand and operate.

[1115] "Voice or text request" means the act of a user giving instructions to a system using voice input or text messages.

[1116] "Performing appropriate operations" refers to the system taking specific actions in response to the user's request.

[1117] "Notifying the completion of an operation" refers to the act of notifying the user that an operation performed by the system has been completed.

[1118] An "emotion recognition engine" is a software technology for analyzing emotions from a user's voice or text.

[1119] "Providing feedback based on analysis results" refers to the act of providing appropriate information and support to users based on the analysis results of the emotion recognition engine.

[1120] A "generative AI model" is a technology that uses artificial intelligence to generate appropriate outputs in response to specific prompts.

[1121] "Generating appropriate prompts" refers to the act of creating the most appropriate questions or instructions based on the user's request.

[1122] This invention is a system that reduces the complexity and difficulty of understanding when users use web services. By combining an emotion recognition engine and a generative AI model, this system includes a function that recognizes user emotions and provides more appropriate support.

[1123] First, users enter the URL of the web service they want to use through a smartphone app, or use the camera to take a screenshot of the screen of the service they want to use. For example, if a user wants to watch content from a specific video streaming service, they can enter the URL of the content page or take a screenshot of the screen.

[1124] Next, the device sends the URL entered by the user or the captured image data to a server in the cloud. If a URL is entered, the URL information is sent, and if an image is captured, the image data is sent. The server retrieves the HTML data of the webpage based on the received URL, or analyzes the image data to extract the screen structure. The software used here is a web scraper and image analysis technology.

[1125] The server analyzes the acquired HTML data and image analysis results to identify important elements on the page (e.g., content title, play button, description), converts them into a user-friendly UI using a generative AI model, and sends the converted UI data to the device.

[1126] When users use the new UI to perform specific actions by voice or text, the application uses an emotion recognition engine to analyze the user's emotional state and converts the voice commands into text using speech recognition technology, specifically a speech recognition library and natural language processing technology.

[1127] The server generates a corresponding web request based on the analyzed action and performs the service operation on behalf of the user. First, to play content, for example, the user logs in with their account information and then clicks the play button. After playback is complete, the server notifies the device that the operation is complete.

[1128] The emotion recognition engine also continuously monitors the user's emotions while using the service, recognizing when the user is feeling stressed or confused. For example, if the user verbally requests "I want to know more," the generative AI model will generate the following prompt:

[1129] "Example prompt statement:

[1130] User input: "I want to know more"

[1131] The generative AI model prompts: "Please provide more details about the content the user is looking for."

[1132] Expected output: "Here's a detailed description of this content. Let me know if you need more information."

[1133] In this way, this system provides the detailed information and operating procedures that users need, improving user convenience. In particular, by combining it with an emotion recognition engine, it is possible to provide optimal feedback based on the user's emotional state. As a result, users can use web services more comfortably.

[1134] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1135] Step 1:

[1136] Users can either enter the URL of the web service they want to use using a smartphone app, or take a photo of the service screen using the camera. The input data is a text URL, and the photo data is an image file.

[1137] Step 2:

[1138] The device sends the URL entered in step 1 or the captured image data to a server on the cloud. The sent data is analyzed on the server side.

[1139] Step 3:

[1140] The server retrieves the HTML data of the web page based on the received URL, or analyzes the image data to extract the screen structure. Specifically, it retrieves the HTML data using a web scraper and uses image analysis technology to identify the text information and screen structure within the image.

[1141] Step 4:

[1142] The server identifies important elements on the page (e.g., content title, play button, description) from the analyzed HTML data and image analysis results. In this process, necessary information is extracted from the analyzed data and converted into an easy-to-read and understandable UI using a generative AI model.

[1143] Step 5:

[1144] The terminal displays the converted UI data sent from the server to the user, who then looks at the displayed UI and inputs their next request by voice or text.

[1145] Step 6:

[1146] The device converts input voice instructions into text using a speech recognition library and analyzes the user's emotional state using an emotion recognition engine. Here, the input data is an audio file, and the output data is text and emotional state information.

[1147] Step 7:

[1148] The server uses a generative AI model to generate appropriate feedback and prompts based on the emotional state information obtained from the emotion recognition engine, and the generated prompts are returned to the user as appropriate instructions or additional information.

[1149] Step 8:

[1150] The server performs the operation requested by the user, prompting the user for additional information as needed (e.g., the user's account information or payment information). This is where data processing and calculations are performed to generate and fulfill the web request.

[1151] Step 9:

[1152] The device notifies the user of the request sent from the server, prompts the user to enter the required information, and then sends it back to the server. This interaction allows the user to provide additional information.

[1153] Step 10:

[1154] The server confirms the completion of the operation and notifies the terminal of the completion. The terminal then displays a notification of the operation completion to the user, informing the user that the operation has been completed successfully. For example, a message such as "The playback procedure has been completed" may be displayed.

[1155] Through these steps, users are freed from cumbersome operations and can use web services while receiving appropriate feedback based on their emotions. Prompt sentence generation using a generative AI model enables users to use services intuitively.

[1156] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1157] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1158] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1159] [Fourth embodiment]

[1160] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1161] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1162] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1163] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1164] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1165] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1166] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1167] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1168] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1169] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1170] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1171] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1172] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1173] First, the user installs and launches the smartphone app, which is designed to assist in the use of web services and provides users with intuitive and easy operation.

[1174] Screen Input

[1175] Users can either enter the URL of the service they want to use through the app's interface or take a screenshot of the service's screen using the app's camera. For example, if a user wants to access a product page on an overseas auction site, they can enter the URL of the product page or take a screenshot of the screen.

[1176] Data transmission and analysis

[1177] The device sends the URL entered by the user and the captured image data to a server on the cloud. The server then accesses the received URL and retrieves the HTML data. It also uses image analysis technology (such as OCR) to extract the screen structure and text information from the captured image data.

[1178] The server analyzes the acquired HTML data or image analysis results to identify the page structure and elements. This includes using generative AI to recognize and classify appropriate information, such as the product name, price, and the location of the purchase button.

[1179] UI transformation

[1180] The server then converts the UI into a format that is intuitively easy for users to understand based on the analysis results. Specifically, it highlights important information and arranges action buttons to simplify user operations. The easy-to-read UI data generated here is sent to the device and displayed to the user.

[1181] Collecting user requests

[1182] Users can view the converted UI and then specify actions by voice or text, for example, "I want to buy this product."

[1183] The device uses voice recognition technology to convert the user's voice instructions into text data, and then uses natural language processing technology to analyze the instructions. The results of this analysis are sent to the server as a specific action (e.g., purchasing procedure).

[1184] Service agency

[1185] The server generates and executes the corresponding web request based on the parsed action, which may include the user logging in with their account information, clicking a purchase button, or entering payment information.

[1186] If necessary, the server will prompt the user for additional information (e.g., shipping address or payment information). The device will notify the user of this request, prompt them to enter the required information, and then send it back to the server.

[1187] Operation completion notification

[1188] When the proxy operation is complete, the server notifies the terminal of the completion. The terminal displays the received notification to the user, informing them that all operations have been completed. For example, it displays a message such as "The purchase procedure has been completed."

[1189] This allows users to easily access the services they need without having to deal with complicated operations or an unfriendly UI. This system is particularly convenient for users with low IT literacy or who face language barriers.

[1190] The processing flow will be explained below.

[1191] Step 1:

[1192] Users launch a smartphone app and enter the URL of the web service they want to use, or take a picture of the screen using the camera. For example, consider the case where a user enters the URL of a product page on an overseas auction site.

[1193] Step 2:

[1194] The device receives the URL entered by the user or the captured image data and sends it to a server on the cloud. If a URL is entered, the URL information is sent, and if an image is captured, the image data is sent.

[1195] Step 3:

[1196] The server retrieves the HTML data of the web page based on the received URL, and if image data is sent, it uses image analysis technology to extract text information and screen structure from the image.

[1197] Step 4:

[1198] The server performs analysis using the acquired HTML data and image analysis results. Specifically, it uses generative AI to identify important elements on the page (e.g., product name, price, purchase button, etc.).

[1199] Step 5:

[1200] The server generates UI data based on the identified information, converting it into a format that is easy for the user to understand. This UI data includes highlighting important information and arranging buttons for easy user operation.

[1201] Step 6:

[1202] The server transmits the generated UI data to the terminal.

[1203] Step 7:

[1204] The device displays the received UI data to the user, allowing the user to consider their next action while looking at the displayed UI.

[1205] Step 8:

[1206] The user inputs their request by voice or text, for example, "I would like to purchase this product."

[1207] Step 9:

[1208] The device uses voice recognition technology to convert the user's voice instructions into text, and then uses natural language processing technology to analyze the text data and understand it as a specific action (e.g., checkout).

[1209] Step 10:

[1210] The device then sends the analyzed user request to the server, and as a specific action, sends a purchase procedure request to the server.

[1211] Step 11:

[1212] Based on the received action instruction, the server generates a corresponding web request and performs the required operation, for example, logging in to an auction site and initiating the process of purchasing an item.

[1213] Step 12:

[1214] The server will prompt the user for additional information as needed, for example a shipping address or payment information.

[1215] Step 13:

[1216] The terminal notifies the user of a request for additional information from the server and prompts the user to input the required information.

[1217] Step 14:

[1218] The user enters any additional information requested and transmits it to the terminal.

[1219] Step 15:

[1220] The terminal transmits the additional information entered by the user to the server.

[1221] Step 16:

[1222] The server uses the received additional information to complete any necessary operations, such as entering payment information and finally completing the purchase.

[1223] Step 17:

[1224] The server notifies the terminal that the operation is complete.

[1225] Step 18:

[1226] The device displays the received completion notification to the user to let them know that the operation has been completed, for example, by displaying a message such as "Purchase procedure completed."

[1227] The above is the specific flow of processing by this system.

[1228] Example 1

[1229] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1230] Modern web services require complex user interfaces and multilingual support, which can make them difficult to use, especially for users with low IT literacy or who face language barriers.Furthermore, differences in operating procedures between different web services can make users feel stressed about procedures and operations.

[1231] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1232] In this invention, the server includes: means for a user to input service information or take a screen shot; means for transmitting the input information or the captured image to a cloud-based processing device; means for analyzing the transmitted information and using a generative AI model to identify the structure and content of the service; means for converting the identified information into an intuitive user interface that is easy for the user to understand; means for the user to communicate requests by voice or text; means for analyzing the requests using natural language processing to identify specific operations; means for generating a web request to execute the identified operations; and means for notifying the user of the completion of the executed operations. This allows users to easily perform complex operations, improving convenience, particularly for users with low IT literacy and when using international services.

[1233] "User" means any person or entity that intends to use the System and / or the Services.

[1234] "Service information" refers to the specific URL and screen information related to the web service the user wants to use.

[1235] "Means for capturing screen images" refers to a function that uses a camera installed on a mobile device such as a smartphone or tablet to capture an image of the screen of the service being used.

[1236] A "cloud processing unit" is a computer resource that runs on a remote server accessible over the Internet.

[1237] "Means for transmitting" refers to the network function for transmitting data from the user's terminal to a processing device on the cloud.

[1238] A "generative AI model" is a machine learning model that is trained to perform a specific task based on a large dataset.

[1239] "User interface" refers to the screen and operating means that users use to operate the system.

[1240] "Natural language processing" refers to techniques and methods for analyzing natural language and converting it into meaningful information.

[1241] A "web request" refers to data that requests or sends information to a server on the Internet.

[1242] "Means of notification" means the functionality that allows the system to send information or messages to the user.

[1243] This invention provides a system that helps users easily use web services through smartphone apps. Specific embodiments of this system are described below.

[1244] First, users install and launch a dedicated application on their smartphone or tablet. This application can be downloaded from various app stores, including Google Play and the App Store. After launching the application, users can either enter the URL of the service they want to use in the "Enter URL" field, or use the camera to take a screenshot.

[1245] For example, if a user wants to purchase an item on an overseas auction site, they can enter the URL of the product page or take a photo of the product page with their camera. The app has a "Send" button, and when the user presses this button, the entered URL or the captured image is sent to a server in the cloud. The sent data is sent to the server in JSON format via an HTTPS request.

[1246] The server then accesses the received URL to retrieve the HTML data. It also uses OCR technology (e.g., Tesseract OCR) to extract text information from the captured image. The server processes this data using libraries such as Python requests, BeautifulSoup, and Selenium.

[1247] The server then uses the acquired HTML data and image analysis results to identify the page structure and elements. At this stage, a generative AI model (e.g., OpenAI GPT) is used to recognize and classify relevant information such as product names, prices, and the location of purchase buttons. Based on the analysis results, a user interface (UI) is created that is intuitive and easy for users to understand. The server generates the UI using front-end libraries such as D3.js and React, and sends the generated UI data to the user's device.

[1248] The device displays a user interface based on the received data. While viewing this interface, the user can use voice or text input functions to specify specific actions. For example, they can give voice commands such as "I want to purchase this product," or manually enter text into a text field. Voice input is converted into text data using speech recognition APIs such as Google Cloud Speech-to-Text and IBM Watson.

[1249] The device then uses natural language processing technology (e.g., spaCy) to analyze the user's instructions and identify specific operations. The analysis results are then sent back to the server, which then generates and executes the corresponding web requests. Specifically, the server uses tools such as Selenium to automatically operate the browser and complete the necessary procedures.

[1250] Finally, the server notifies the device that the operation is complete, and the device displays the notification to the user, for example, by displaying a message saying "Purchase procedure completed" to let the user know that all operations were completed successfully.

[1251] Here are some examples of prompts:

[1252] "User: I would like to purchase this item from Amazon.

[1253] URL:https: / / www.example.com / example-product

[1254] Camera: A screenshot of your screen has been uploaded.

[1255] This system will allow users to easily perform complicated operations, greatly improving convenience, especially for users with low IT literacy and those using international services.

[1256] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1257] Step 1: User Actions

[1258] Users install and launch a dedicated application on their smartphone or tablet. Then, they can enter the URL of the service they want to use or take a screenshot of the screen with their camera. Specifically, users download the application from Google Play or the App Store, open the application, and enter the URL in the "Enter URL" field or tap the camera icon to take a screenshot of the screen.

[1259] Input: URL or image data

[1260] Output: URLs and image data prepared within the application

[1261] Step 2: Sending data

[1262] The device sends the URL entered by the user and the captured image data to a server on the cloud. Specifically, when the user taps the "Send" button, the data is sent to the server in JSON format via an HTTPS request.

[1263] Input: URL or image data ready

[1264] Output: The URL and image data are sent to the server and accepted by the server.

[1265] Step 3: Get the URL and analyze the image

[1266] The server accesses the sent URL and retrieves the HTML data. It also uses Tesseract OCR technology to extract text information from the captured image data. Specifically, the server uses the Python requests library to send a GET request to the URL and uses Tesseract OCR to retrieve text information from the image.

[1267] Input: URL or image data sent to the server

[1268] Output: HTML data or parsed text information

[1269] Step 4: Analyzing the Page

[1270] The server analyzes the acquired HTML data and image analysis results to identify the structure and elements of the web page. At this stage, a generative AI model (e.g., OpenAI GPT) is used to recognize and classify relevant information such as the product name, price, and purchase button location. Specifically, the server parses the HTML data using BeautifulSoup or Selenium and organizes the information using generative AI.

[1271] Input: HTML data or text information

[1272] Output: Identified product information (name, price, button position, etc.)

[1273] Step 5: Transforming the UI

[1274] The server generates a user interface (UI) based on the analysis results, which is easy for users to understand intuitively. Specifically, the server visualizes the analysis results using D3.js and React, and sends the generated UI data in JSON format to the device.

[1275] Input: Identified product information

[1276] Output: Generated UI data

[1277] Step 6: Input user requests

[1278] The user can then view the converted UI and input a specific action using voice or text, such as "I want to buy this product." To achieve this, the user can tap the voice input button in the app to input voice or type directly into a text field.

[1279] Input: Generated UI data

[1280] Output: User's voice commands or text input

[1281] Step 7: Parsing the instructions

[1282] The device converts the voice instructions used into text using Google Cloud Speech-to-Text or IBM Watson, and then analyzes them using natural language processing technology (e.g., spaCy). As a result of the analysis, specific operation instructions are identified and sent to the server. Specific operations include sending the voice data to an API, converting it into text, and identifying actions based on the analyzed text.

[1283] Input: User's voice commands or text input

[1284] Output: Specific operation instructions sent to the server

[1285] Step 8: Delegate the service

[1286] The server generates and executes web requests based on the user's actions, such as logging in to an account, clicking a purchase button, or entering payment information. The server then uses Selenium to automatically perform the necessary steps.

[1287] Input: Specific operating instructions

[1288] Output: The result of the web request that was made.

[1289] Step 9: Operation completion notification

[1290] The server sends a notification to the device when the proxy operation is complete. The device displays the received notification to the user. For example, it displays a message saying "The purchase procedure has been completed." Specifically, the server sends the operation completion status to the device in JSON format, and the device receives it and displays it in the app.

[1291] Input: The result of the web request that was executed

[1292] Output: Notification to the user that the operation was completed

[1293] (Application example 1)

[1294] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1295] When using overseas web services, especially food delivery services, users often have difficulty with different languages ​​and complex UIs. Due to language barriers and a lack of IT literacy, many users are unable to use the services easily and intuitively.

[1296] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1297] In this invention, the server includes a means for accessing a URL entered by a user and acquiring HTML data, a means for analyzing captured image data to extract the screen structure, and a means for automatically translating information displayed in different languages ​​and converting it into a language that is easy for the user to understand. This allows users to intuitively use overseas food delivery services without having to deal with language or complex UIs.

[1298] "Service URL" refers to the Internet address of the web service that a User wishes to use.

[1299] "Taking a picture of the screen with a camera" refers to the act of a user using the camera function of a smartphone or other device to save the display screen of a web service as image data.

[1300] "Identified information" refers to data about the structure and content of web services obtained by analyzing URLs and image data.

[1301] "Easy-to-see and easy-to-understand UI" refers to an interface that is designed to be easy for users to understand and operate.

[1302] "Communicating requests by voice or text" refers to the act of a user using voice input or text input to communicate desired operations or instructions to the system.

[1303] "Performing appropriate operations" refers to the act of analyzing the user's request and having the system take the necessary actions on the web service based on those instructions.

[1304] "Notifying the user of the completion of an executed operation" refers to the act of notifying the user that an operation performed by the system based on the user's instructions has been completed.

[1305] "Automatically translating information displayed in different languages" refers to the process of automatically converting content displayed by a web service into a language that the user can understand, when that content is not in the user's native language.

[1306] "Providing services and products available in a specific area" refers to the act of presenting users with information about services and products available within a specific area based on the user's location or designated area.

[1307] First, the user installs and launches the smartphone app, which is designed to assist in the use of web services and provides users with intuitive and easy operation.

[1308] Screen Input

[1309] Users can either enter the URL of the service they want to use through the app interface or take a screenshot of the service screen using the camera. For example, if a user wants to use an overseas food delivery service, they can enter the URL of the service or take a screenshot of the screen.

[1310] Data transmission and analysis

[1311] The device sends the URL entered by the user or the captured image data to a server on the cloud. The server then accesses the received URL and retrieves the HTML data. It also uses image analysis technology (such as OCR) to extract the screen structure and text information from the captured image data. The server then analyzes the retrieved HTML data or image analysis results to identify the page structure and elements. This includes utilizing generative AI to recognize and classify appropriate information. For example, it identifies menu names, prices, and the position of the order button.

[1312] UI transformation

[1313] Based on the analysis results, the server converts the UI into a format that is intuitively easy for users to understand. Specifically, it highlights important information and places action buttons to simplify user operations. It also automatically translates information displayed in different languages ​​and converts it into a language that is easy for users to understand. The easy-to-read UI data generated here is sent to the device and displayed to the user.

[1314] Collecting user requests

[1315] While viewing the converted UI, the user can specify actions by voice or text. For example, the user can say, "I would like to order this dish." The device uses voice recognition technology to convert the user's voice instructions into text data, and then uses natural language processing technology to analyze the instructions. The results of this analysis are sent to the server as a specific action (e.g., placing an order).

[1316] Service agency

[1317] The server generates and executes the corresponding web request based on the parsed action. This may include logging in with the user's account information, clicking an order button, or entering payment information. If necessary, the server prompts the user for additional information (e.g., shipping address or payment information). The device notifies the user of this request, prompts them to enter the required information, and then sends it back to the server.

[1318] Operation completion notification

[1319] The server notifies the terminal when the proxy operation is complete. The terminal then displays the received notification to the user, informing them that all operations have been completed. For example, it displays a message such as "The order procedure has been completed." This allows the user to easily use the services they need without having to struggle with complicated operations or an unfriendly UI.

[1320] Examples and prompts

[1321] For example, if a user wants to use an overseas food delivery service, they can enter the following prompt into the app:

[1322] Example prompt sentence:

[1323] "https: / / example-fooddeli.com"

[1324] or

[1325] "https: / / example-image.com / screenshot.png"

[1326] This allows users to use the service intuitively without having to deal with different languages ​​or complex operations, and the system offers great convenience, especially for users with low IT literacy or who face language barriers.

[1327] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1328] Step 1:

[1329] The user launches the smartphone app and either enters the URL of the service they want to use or takes a screenshot of the service's screen using the camera. This entered URL or captured screen image becomes the input data for processing.

[1330] Input data: URL or image

[1331] Output data: Input URL or image data

[1332] Step 2:

[1333] The device sends the URL entered by the user or the captured image data to a server on the cloud. This sending operation passes the input data directly to the server.

[1334] Input data: User's URL or image data

[1335] Output data: Transmission to server completed

[1336] Step 3:

[1337] The server accesses the received URL and retrieves the HTML data. It also uses OCR technology to analyze the screen structure and text information of the captured image data. Specifically, it uses pytesseract to extract text from the image and BeautifulSoup to analyze the HTML.

[1338] Input data: URL or image data

[1339] Output data: HTML data or parsed text information

[1340] Step 4:

[1341] The server analyzes the acquired HTML data or image analysis results to identify the page structure and elements. Generative AI technology is then used to properly recognize the identified information and classify important information. For example, menu names, prices, and the location of the order button are identified.

[1342] Input data: HTML data or parsed text information

[1343] Output data: Structure and element information of identified pages

[1344] Step 5:

[1345] The server converts the UI into a format that is intuitively easy for users to understand based on the analysis results. During this process, it highlights important information and places action buttons to simplify user operations. In addition, it automatically translates information displayed in different languages ​​and converts it into a language that is easy for users to understand.

[1346] Input data: Identified page structure and element information

[1347] Output data: Transformed UI data

[1348] Step 6:

[1349] The device displays the converted UI data received from the server to the user, who can then instruct actions via voice or text while viewing the UI.

[1350] Input data: Transformed UI data

[1351] Output data: User's voice or text instructions for action

[1352] Step 7:

[1353] The device uses voice recognition technology to convert the user's voice instructions into text data, and then uses natural language processing technology to analyze the instructions. This is where libraries such as Spacy are used. The results of this analysis are sent to the server as a specific action (e.g., ordering food).

[1354] Input data: User's voice or text instructions

[1355] Output data: Parsed instructions

[1356] Step 8:

[1357] The server generates and executes the corresponding web requests based on the parsed user instructions, which may include logging in with the user's account information, clicking an order button, or entering payment information.

[1358] Input data: Parsed instructions

[1359] Output data: Status of the operation performed

[1360] Step 9:

[1361] The server notifies the terminal when the executed operation is completed. The terminal displays the received notification to the user to inform them that all operations have been completed. For example, it displays a message such as "The order procedure has been completed."

[1362] Input data: Status of the operation performed

[1363] Output data: Completion notification to the user

[1364] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1365] This invention is a system that reduces the complexity and difficulty of understanding that users face when using web services. This system includes a function that recognizes the user's emotions and provides more appropriate support by combining it with an emotion engine.

[1366] Screen Input

[1367] Users can either enter the URL of the web service they want to use through a smartphone app or take a screenshot of the service's screen using the camera. For example, if a user wants to access a product page on an overseas auction site, they can enter the URL of that product page or take a screenshot.

[1368] Data transmission and analysis

[1369] The device sends the URL entered by the user and the captured image data to a server on the cloud. If a URL is entered, the URL information is sent, and if an image is captured, the image data is sent.

[1370] The server retrieves the HTML data of the web page based on the received URL, or analyzes the image data to extract the screen structure. Image analysis technology is used to identify the text information and screen structure within the image.

[1371] UI transformation

[1372] The server analyzes the acquired HTML data and image analysis results to identify important elements on the page (e.g., product name, price, purchase button), and uses generative AI to convert them into a user-friendly UI, which is then sent to the device.

[1373] Utilizing the Emotion Engine

[1374] When users see the UI displayed on their device, they can input their request by voice or text. For example, they can give instructions such as "I would like to purchase this product." At this time, the emotion engine recognizes the user's emotions from the voice or text.

[1375] The device uses an emotion engine to analyze the user's emotional state, speech recognition technology to convert voice instructions into text, and natural language processing technology to analyze the text data and send specific actions (e.g., purchase procedures) to the server.

[1376] Service agency

[1377] Based on the parsed actions, the server generates corresponding web requests and performs service operations on behalf of the user: first, logging in with their account information, then clicking the purchase button and entering payment information.

[1378] If necessary, the server will prompt the user for additional information (e.g., shipping address or payment information). The device will notify the user of the request, prompt them to enter the required information, and then send it back to the server.

[1379] Operation completion notification

[1380] When the operation is complete, the server notifies the device. The device then notifies the user of the completion of the operation, for example by displaying a message such as "The purchase procedure has been completed."

[1381] Emotion-Based Feedback

[1382] The emotion engine continuously monitors the user's emotions while using the service and recognizes when the user is feeling stressed or confused. The device uses the information from the emotion engine to provide appropriate feedback and additional support to the user, such as detailed instructions or hints on how to operate the device, improving user convenience.

[1383] In this way, the system allows users to use the service without difficulty and provides a more comfortable experience. In particular, by combining it with an emotion engine, it is possible to take the user's emotions into consideration and provide optimal support. As a result, it is possible to provide excellent convenience even to users with low IT literacy or who experience language barriers.

[1384] The processing flow will be explained below.

[1385] Step 1:

[1386] Users launch a smartphone app and either enter the URL of the web service they want to use or take a screenshot of the screen of the service they want to use using the camera. For example, they can enter the URL or take a screenshot to access the product page of an overseas auction site.

[1387] Step 2:

[1388] The device receives the URL entered by the user or the captured image data and sends it to a server in the cloud. The URL information and image data are sent to the server securely.

[1389] Step 3:

[1390] The server retrieves the HTML data of the web page based on the received URL, or analyzes the received image data to extract the text and screen structure within the image. Image analysis technology is used to perform character recognition (OCR) and structure recognition.

[1391] Step 4:

[1392] The server performs detailed analysis of the acquired HTML data and image analysis results to identify important elements on the page (e.g., product name, price, purchase button, etc.). Generative AI is used for the analysis to classify and organize the data.

[1393] Step 5:

[1394] The server generates user interface (UI) data based on the identified information, converting it into a user-friendly format, including highlighting important information and arranging action buttons to simplify operation.

[1395] Step 6:

[1396] The server transmits the generated UI data to the terminal.

[1397] Step 7:

[1398] The device displays the received UI data to the user, who then considers what action to take next.

[1399] Step 8:

[1400] The user looks at the displayed UI and inputs their request by voice or text. For example, they can say, "I would like to purchase this product."

[1401] Step 9:

[1402] The device uses an emotion engine to recognize emotions from the user's voice or text, then uses voice recognition technology to convert the voice instructions into text and natural language processing technology to analyze the text data. This analysis then identifies specific actions (e.g., checkout).

[1403] Step 10:

[1404] The device sends the analysis results and the user's emotional state to the server.

[1405] Step 11:

[1406] The server generates a corresponding web request based on the received action instruction and performs the required operation, such as logging in to an auction site and proceeding with the process of purchasing an item.

[1407] Step 12:

[1408] The server prompts the user for additional information (e.g., shipping address or payment information) if necessary.

[1409] Step 13:

[1410] The terminal notifies the user of a request for additional information from the server and prompts the user to enter the required information.

[1411] Step 14:

[1412] The user enters any additional information requested and sends it to the terminal.

[1413] Step 15:

[1414] The terminal sends any additional information entered by the user to the server.

[1415] Step 16:

[1416] The server uses the additional information received to complete any necessary operations, such as entering payment information and finalizing the order.

[1417] Step 17:

[1418] The server notifies the terminal that the operation is complete.

[1419] Step 18:

[1420] The device receives this notification and displays it to the user, for example, "Purchase procedure completed."

[1421] Step 19:

[1422] The emotion engine continuously monitors the user's emotions while using the service and provides appropriate feedback and support to the user depending on the situation.

[1423] The above is the specific processing flow of this system. This allows users to smoothly use the services they need without having to worry about complicated procedures or an unfriendly UI. In addition, by combining it with an emotion engine, it is possible to provide optimal support that is tailored to the user's emotions.

[1424] Example 2

[1425] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1426] When users use web services, they often find the operation complicated or difficult to understand. This makes it difficult to use, especially for users with low IT literacy or who face language barriers. There is a need to solve these problems and enable users to use services smoothly. Furthermore, by taking user emotions into consideration, it is necessary to provide more appropriate support and improve the user experience.

[1427] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1428] In this invention, the server includes means for a user to input a service address or take a picture of the screen with a camera, means for analyzing the input address or the captured image and identifying the structure and content of the service, means for converting the identified information into an easy-to-view and understandable user interface, means for the user to communicate their request by voice or text, means for analyzing the user's emotional state using an emotion engine, means for analyzing the request and performing an appropriate operation, and means for notifying the user of the completion of the performed operation. This not only allows users to easily use web services, but also makes it possible to provide optimal support through emotion recognition.

[1429] "User" refers to any individual or entity that uses the Web Services or Applications.

[1430] An "address" is a series of character strings that indicates unique location information for identifying access to a web service.

[1431] A "camera" is a device for taking pictures and storing them as digital data.

[1432] "Image data" is digital information representing a still image captured by a device such as a camera.

[1433] "Analysis" is the process of processing input data (addresses or image data) to extract information to understand its structure and content.

[1434] A "user interface" is the visual and operational interface through which a user interacts with a computer system or application.

[1435] "Voice" refers to the words and instructions given by the user captured as sound.

[1436] "Text" is a document or instruction created by combining letters and symbols.

[1437] An "emotion engine" is a function or software that analyzes a user's emotional state from voice or text.

[1438] "Natural language processing" is a technology that allows computers to understand, analyze, and generate human language.

[1439] An "action" refers to a specific operation or behavior that a system performs based on a user's instructions.

[1440] "Operation completed" is a state indicating that the action instructed by the user has been executed successfully.

[1441] "Notifications" are messages or alerts that inform the user about the completion of an operation or the status of the system.

[1442] A "system" is a collection of multiple pieces of hardware and software that work together to achieve a specific function or purpose.

[1443] "Structure" refers to the internal layout and arrangement of elements of a web page or service, and their relationships to one another.

[1444] "Content" refers to information such as text, images, links, and forms on a web page or service.

[1445] This invention is a system that reduces the complexity and difficulty of understanding when users use web services. This system has the function of recognizing the user's emotions by combining it with an emotion engine and providing more appropriate support.

[1446] Hardware and software used:

[1447] The system uses a smartphone app, cloud server, emotion engine, image analysis technology, natural language processing technology, etc. By combining these technologies, it supports user operations.

[1448] The specific process flow:

[1449] The server receives the address entered by the user using the smartphone app or the captured image data. If an address is entered, the server retrieves the HTML data of the corresponding web page. If captured image data is sent, the server uses image analysis technology to identify the text information and screen structure within the image.

[1450] The server then analyzes the acquired HTML data and image analysis results to identify important elements on the page (e.g., product name, price, purchase button). Based on this information, it uses a generative AI model to convert it into a user interface that is easy for the user to understand. The converted UI data is then sent to the device.

[1451] The user looks at the new UI and inputs their request by voice or text. For example, they might say, "I want to buy this product." At this time, the emotion engine recognizes the emotion from the user's voice or text and analyzes that information.

[1452] The device uses an emotion engine to analyze the user's emotional state, speech recognition technology to convert voice instructions into text, and natural language processing technology to analyze the text data, convert it into a specific action (e.g., purchase procedure), and send it to the server.

[1453] Based on the parsed action, the server generates the appropriate web request and performs the service operation on behalf of the user. For example, the user logs in with their account information, clicks a purchase button, and enters payment information. If necessary, the server prompts the user for additional information (for example, shipping address or payment information). The user provides the required information through the app.

[1454] Once the operation is complete, the server notifies the device, which then displays a message to the user saying, "The purchase has been completed."

[1455] The emotion engine also continuously monitors the user's emotions while using the service, recognizing when the user is feeling stressed or confused. The device uses the information from the emotion engine to provide appropriate feedback and additional support to the user, such as providing detailed instructions and hints on how to operate the device, improving user convenience.

[1456] Examples:

[1457] Example prompt sentence:

[1458] URL: https: / / example-auctionsite.com / product123

[1459] Take a screenshot and transform the UI to make it easier for users to find the purchase button.

[1460] Through these steps, the system enables users to use web services efficiently and easily, and provides optimal support through emotion recognition, making it possible to provide an easy-to-use environment even for users with low IT literacy or who face language barriers.

[1461] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1462] Step 1:

[1463] The user opens the smartphone app and enters the address of the service, or takes a screenshot using the camera. The input can be in the form of a URL or a photo. For example, if a user wants to access a product page on an auction site, they can enter the URL of that product page or take a screenshot. The input data is sent to the device.

[1464] Step 2:

[1465] The device sends the URL entered by the user or the captured image data to a server on the cloud. The device converts the entered information into the appropriate format (URL or image data) and sends it to the server. Specifically, it sends the URL information or image data via an Internet connection.

[1466] Step 3:

[1467] The server retrieves the HTML data of the web page based on the received URL. If a URL is entered, the server accesses the address and downloads the HTML data. If captured image data is sent, the server uses image analysis technology to identify the text information and screen structure within the image.

[1468] Step 4:

[1469] The server analyzes the HTML data and image analysis results to identify important elements on the page (e.g., product name, price, buy button). The server extracts this information using an HTML parser and image analysis algorithms.

[1470] Step 5:

[1471] The server uses a generative AI model to convert the data into a user interface that is easy for the user to understand. Based on the analysis results, the server changes the layout to a format that is visually easy for the user to understand, and sends the new UI data to the device.

[1472] Step 6:

[1473] When users see the new UI displayed on their device, they can input their request by voice or text. For example, they can say, "I want to purchase this product." The device then receives the user's voice or text.

[1474] Step 7:

[1475] The device uses an emotion engine to analyze the user's emotional state, converts voice instructions into text using voice recognition technology, and analyzes the text data using natural language processing technology. Specific actions are then taken as a result of the analysis.

[1476] Step 8:

[1477] The terminal then sends the analyzed specific action to the server. For example, an action such as "click the purchase button" or "enter payment information" is generated and sent to the server.

[1478] Step 9:

[1479] The server generates a corresponding web request based on the action it receives: for example, logging in with a specified account, clicking a purchase button, entering payment information, etc. If necessary, the server requests additional information (e.g., a shipping address).

[1480] Step 10:

[1481] When the operation is complete, the server notifies the device, which then notifies the user by displaying a message such as "The purchase procedure has been completed."

[1482] Step 11:

[1483] The emotion engine continuously monitors the user's emotions while using the service. It recognizes when the user feels stressed or confused. The device uses the information from the emotion engine to provide appropriate feedback and additional support. For example, it can support the user's operations by displaying detailed instructions or hints on how to operate the device.

[1484] (Application example 2)

[1485] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1486] When using conventional web services, users often face complex operations and difficulties in understanding them, posing significant barriers, particularly for users with low IT literacy or who speak a different language. Furthermore, lack of appropriate support often prevents users from identifying their emotions, increasing stress and confusion and impairing the user experience. The objective of this invention is to solve these problems and provide a more intuitive and user-friendly user interface and appropriate feedback based on the user's emotions, enabling users to use web services more comfortably.

[1487] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1488] In this invention, the server includes: means for a user to input a service URL or take a picture of the screen with a camera; means for analyzing the input URL or the captured image and identifying the structure and content of the service; means for converting the identified information into an easy-to-view and understandable UI; means for the user to communicate their request by voice or text; means for analyzing the request and performing an appropriate operation; means for notifying the user of the completion of the executed operation; means for analyzing the user's emotions using an emotion recognition engine and providing feedback based on the analysis results; and means for generating and responding to the user's request with an appropriate prompt sentence using a generative AI model. This not only simplifies complex operations for the user, but also enables the user to receive appropriate support tailored to their emotions.

[1489] "Service URL" means the web address of the online service that a User wishes to access.

[1490] "Taking a photo with a camera" refers to the act of recording the screen of the service the user wants to use as a digital image.

[1491] "Analysis" is the act of taking input data, understanding its structure and content, and extracting information.

[1492] "Identification" refers to the act of extracting and identifying important elements from the analyzed information.

[1493] An "easy-to-see and understand UI" is a user interface that users can easily understand and operate.

[1494] "Voice or text request" means the act of a user giving instructions to a system using voice input or text messages.

[1495] "Performing appropriate operations" refers to the system taking specific actions in response to the user's request.

[1496] "Notifying the completion of an operation" refers to the act of notifying the user that an operation performed by the system has been completed.

[1497] An "emotion recognition engine" is a software technology for analyzing emotions from a user's voice or text.

[1498] "Providing feedback based on analysis results" refers to the act of providing appropriate information and support to users based on the analysis results of the emotion recognition engine.

[1499] A "generative AI model" is a technology that uses artificial intelligence to generate appropriate outputs in response to specific prompts.

[1500] "Generating appropriate prompts" refers to the act of creating the most appropriate questions or instructions based on the user's request.

[1501] This invention is a system that reduces the complexity and difficulty of understanding when users use web services. By combining an emotion recognition engine and a generative AI model, this system includes a function that recognizes user emotions and provides more appropriate support.

[1502] First, users enter the URL of the web service they want to use through a smartphone app, or use the camera to take a screenshot of the screen of the service they want to use. For example, if a user wants to watch content from a specific video streaming service, they can enter the URL of the content page or take a screenshot of the screen.

[1503] Next, the device sends the URL entered by the user or the captured image data to a server in the cloud. If a URL is entered, the URL information is sent, and if an image is captured, the image data is sent. The server retrieves the HTML data of the webpage based on the received URL, or analyzes the image data to extract the screen structure. The software used here is a web scraper and image analysis technology.

[1504] The server analyzes the acquired HTML data and image analysis results to identify important elements on the page (e.g., content title, play button, description), converts them into a user-friendly UI using a generative AI model, and sends the converted UI data to the device.

[1505] When users use the new UI to perform specific actions by voice or text, the application uses an emotion recognition engine to analyze the user's emotional state and converts the voice commands into text using speech recognition technology, specifically a speech recognition library and natural language processing technology.

[1506] The server generates a corresponding web request based on the analyzed action and performs the service operation on behalf of the user. First, to play content, for example, the user logs in with their account information and then clicks the play button. After playback is complete, the server notifies the device that the operation is complete.

[1507] The emotion recognition engine also continuously monitors the user's emotions while using the service, recognizing when the user is feeling stressed or confused. For example, if the user verbally requests "I want to know more," the generative AI model will generate the following prompt:

[1508] "Example prompt statement:

[1509] User input: "I want to know more"

[1510] The generative AI model prompts: "Please provide more details about the content the user is looking for."

[1511] Expected output: "Here's a detailed description of this content. Let me know if you need more information."

[1512] In this way, this system provides the detailed information and operating procedures that users need, improving user convenience. In particular, by combining it with an emotion recognition engine, it is possible to provide optimal feedback based on the user's emotional state. As a result, users can use web services more comfortably.

[1513] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1514] Step 1:

[1515] Users can either enter the URL of the web service they want to use using a smartphone app, or take a photo of the service screen using the camera. The input data is a text URL, and the photo data is an image file.

[1516] Step 2:

[1517] The device sends the URL entered in step 1 or the captured image data to a server on the cloud. The sent data is analyzed on the server side.

[1518] Step 3:

[1519] The server retrieves the HTML data of the web page based on the received URL, or analyzes the image data to extract the screen structure. Specifically, it retrieves the HTML data using a web scraper and uses image analysis technology to identify the text information and screen structure within the image.

[1520] Step 4:

[1521] The server identifies important elements on the page (e.g., content title, play button, description) from the analyzed HTML data and image analysis results. In this process, necessary information is extracted from the analyzed data and converted into an easy-to-read and understandable UI using a generative AI model.

[1522] Step 5:

[1523] The terminal displays the converted UI data sent from the server to the user, who then looks at the displayed UI and inputs their next request by voice or text.

[1524] Step 6:

[1525] The device converts input voice instructions into text using a speech recognition library and analyzes the user's emotional state using an emotion recognition engine. Here, the input data is an audio file, and the output data is text and emotional state information.

[1526] Step 7:

[1527] The server uses a generative AI model to generate appropriate feedback and prompts based on the emotional state information obtained from the emotion recognition engine, and the generated prompts are returned to the user as appropriate instructions or additional information.

[1528] Step 8:

[1529] The server performs the operation requested by the user, prompting the user for additional information as needed (e.g., the user's account information or payment information). This is where data processing and calculations are performed to generate and fulfill the web request.

[1530] Step 9:

[1531] The device notifies the user of the request sent from the server, prompts the user to enter the required information, and then sends it back to the server. This interaction allows the user to provide additional information.

[1532] Step 10:

[1533] The server confirms the completion of the operation and notifies the terminal of the completion. The terminal then displays a notification of the operation completion to the user, informing the user that the operation has been completed successfully. For example, a message such as "The playback procedure has been completed" may be displayed.

[1534] Through these steps, users are freed from cumbersome operations and can use web services while receiving appropriate feedback based on their emotions. Prompt sentence generation using a generative AI model enables users to use services intuitively.

[1535] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1536] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1537] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1538] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1539] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1540] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1541] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1542] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1543] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1544] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1545] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1546] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1547] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1548] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1549] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1550] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1551] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1552] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1553] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1554] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1555] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1556] The following is further disclosed regarding the above embodiment.

[1557] (Claim 1)

[1558] A means for the user to enter the service URL or take a screen capture;

[1559] A means for analyzing the input URL or the captured image and identifying the structure and content of the service;

[1560] A means for converting the specified information into an easy-to-view and easy-to-understand UI;

[1561] A means for users to communicate their requests by voice or text;

[1562] means for analyzing said request and performing appropriate operations;

[1563] means for notifying a user of completion of said performed operation.

[1564] (Claim 2)

[1565] A means to access the URL entered by the user and obtain HTML data,

[1566] 10. The system of claim 1, further comprising means for analyzing the captured image data to extract a screen structure.

[1567] (Claim 3)

[1568] 2. The system according to claim 1, further comprising means for analyzing a user's request by natural language processing and converting the request into a specific action.

[1569] "Example 1"

[1570] (Claim 1)

[1571] A means for the user to enter information about the service or take a screen shot;

[1572] means for transmitting the input information or the captured image to a processing device on the cloud;

[1573] means for analyzing the transmitted information and utilizing a generative AI model to identify the structure and content of the service;

[1574] A means for converting the specified information into an intuitive and user-friendly user interface;

[1575] A means for users to communicate their requests by voice or text;

[1576] a means for analyzing the request by natural language processing and specifying a specific operation;

[1577] means for generating a web request to perform the identified operation;

[1578] means for notifying a user of completion of said performed operation.

[1579] (Claim 2)

[1580] a means for retrieving data from a web page based on information entered by a user;

[1581] 10. The system of claim 1, further comprising means for analyzing the captured image data to extract a screen structure.

[1582] (Claim 3)

[1583] 2. The system according to claim 1, further comprising means for analyzing a user's request by natural language processing and converting the request into a specific action.

[1584] "Application Example 1"

[1585] (Claim 1)

[1586] A means for the user to enter the service URL or take a screen capture;

[1587] A means for analyzing the input URL or the captured image and identifying the structure and content of the service;

[1588] A means for converting the specified information into an easy-to-view and easy-to-understand UI;

[1589] A means for users to communicate their requests by voice or text;

[1590] means for analyzing said request and performing appropriate operations;

[1591] means for notifying a user of completion of the performed operation;

[1592] A means to automatically translate information displayed in different languages ​​and convert it into a language that is easy for the user to understand;

[1593] A system that includes a means for providing users with services and products available in a specific area based on the translated information.

[1594] (Claim 2)

[1595] A means to access the URL entered by the user and obtain HTML data,

[1596] 10. The system of claim 1, further comprising means for analyzing the captured image data to extract a screen structure.

[1597] (Claim 3)

[1598] 2. The system according to claim 1, further comprising means for analyzing a user's request by natural language processing and converting the request into a specific action.

[1599] "Example 2: Combining Emotion Engines"

[1600] (Claim 1)

[1601] A means for the user to enter the address of the service or take a screen capture;

[1602] A means for analyzing the input address or the captured image to identify the structure and content of the service;

[1603] A means for converting the specified information into an easy-to-view and easy-to-understand user interface;

[1604] A means for users to communicate their requests by voice or text;

[1605] means for analyzing the emotional state of a user using an emotion engine;

[1606] means for analyzing said request and performing appropriate operations;

[1607] means for notifying a user of completion of said performed operation.

[1608] (Claim 2)

[1609] means for accessing the address entered by the user and retrieving the hypertext markup language data;

[1610] 10. The system of claim 1, further comprising means for analyzing the captured image data to extract a screen structure.

[1611] (Claim 3)

[1612] 2. The system according to claim 1, further comprising means for analyzing a user's request by natural language processing and converting the request into a specific action.

[1613] "Application example 2 when combining emotion engines"

[1614] (Claim 1)

[1615] A means for the user to enter the service URL or take a screen capture;

[1616] A means for analyzing the input URL or the captured image and identifying the structure and content of the service;

[1617] A means for converting the specified information into an easy-to-view and easy-to-understand UI;

[1618] A means for users to communicate their requests by voice or text;

[1619] means for analyzing said request and performing appropriate operations;

[1620] means for notifying a user of completion of the performed operation;

[1621] A means for analyzing the user's emotions using an emotion recognition engine and providing feedback based on the analysis results;

[1622] A system including means for generating and responding to a user's request with an appropriate prompt using a generative AI model.

[1623] (Claim 2)

[1624] A means to access the URL entered by the user and obtain HTML data,

[1625] 10. The system of claim 1, further comprising means for analyzing the captured image data to extract a screen structure.

[1626] (Claim 3)

[1627] A means of analyzing user requests using natural language processing and converting them into specific actions,

[1628] 10. The system of claim 1, further comprising means for converting a user interface into a user-friendly UI using a generative AI model. [Explanation of symbols]

[1629] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for the user to enter the service URL or take a screen capture; A means for analyzing the input URL or the captured image and identifying the structure and content of the service; A means for converting the specified information into an easy-to-view and easy-to-understand UI; A means for users to communicate their requests by voice or text; means for analyzing said request and performing appropriate operations; and means for notifying a user of completion of said performed operation.

2. A means to access the URL entered by the user and obtain HTML data, 2. The system according to claim 1, further comprising means for analyzing the captured image data to extract a screen structure.

3. 2. The system according to claim 1, further comprising means for analyzing a user's request by natural language processing and converting the request into a specific action.

Images