system

Abstract

Provide a system. [Solution] a means for accepting a URL specified by a user; A means for retrieving a web page at a specified URL; A means for analyzing the HTML source of the retrieved web page and extracting key text elements; A means for transmitting the extracted text elements to a speech synthesis API and obtaining speech data; means for combining the acquired audio data into a continuously playable audio file; a means for providing the generated audio file to a user; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] For users who have difficulty using the Internet, such as the elderly and the visually impaired, traditional web pages that are primarily text and images are difficult to access. This creates an information gap, preventing these users from fully utilizing useful information on the Internet. The present invention solves this problem by converting web pages rich in text and image information into an audio format, making them more accessible even for users who have difficulty using the Internet. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing a system including: means for accepting a URL specified by a user; means for retrieving the web page at the specified URL; means for analyzing the HTML source of the retrieved web page and extracting key text elements; means for transmitting the extracted text elements to a speech synthesis API to retrieve audio data; means for combining the retrieved audio data into an audio file that can be played continuously; and means for providing the generated audio file to a user. Specifically, the system extracts text elements such as headings, paragraphs, links, and lists from the analyzed HTML source, and rearranges and arranges them in an appropriate order based on their content and importance to provide useful information in audio format.

[0006] A "user" is someone who wishes to use the system to play web pages audibly.

[0007] A "URL" refers to the address of a web page, and is information that identifies the location of the page that a user is trying to access.

[0008] A "server" is a computer system that receives a user's request, retrieves the specified web page, and performs analysis and speech conversion.

[0009] "HTML source" is the markup language code that makes up a web page, and is data that contains information for describing the content and structure of the page.

[0010] "Text elements" refer to the written information within a web page and include different content elements such as headings, paragraphs, links, lists, etc.

[0011] A "speech synthesis API" is an application programming interface for converting text data into speech data.

[0012] "Audio Data" means a digital audio file that represents text in audio form, generated by a speech synthesis API.

[0013] An "audio file" is a digital file containing continuously playable audio data, which is the final output format provided to the user.

[0014] "Parsing" is the process of identifying and extracting key text elements from the HTML source of a web page.

[0015] "Extraction" is the process of extracting specific text elements such as headings, paragraphs, links, and lists from the parsed HTML source.

[0016] "Sorting" is a process of rearranging the extracted text elements in an appropriate order based on their contents and importance. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The system of the present invention provides voice-based web page access that is easy to use for the elderly, the visually impaired, etc. A specific embodiment of this system will be described below.

[0039] System Overview

[0040] When a user accesses the system and wants to listen to the contents of a web page by audio, they first use a device (PC, smartphone, tablet, etc.). The user opens a web browser, accesses the system, and enters the URL of the web page they want to play by audio. This information is sent to the server, which retrieves the web page at the specified URL, converts it into audio format, and provides it to the user.

[0041] Program processing

[0042] Page retrieval and analysis

[0043] The server receives a request sent by a user and retrieves the web page at the specified URL. Typically, the HTTP request is used to retrieve the HTML source of the web page. This HTML source becomes the data used to analyze the content of the web page.

[0044] Text Extraction and Structure Recognition

[0045] The server analyzes the retrieved HTML source and extracts key text elements such as headings, paragraphs, links, and lists. It uses an HTML parser to generate a Document Object Model (DOM) and extracts the necessary text elements based on this. These extracted text elements are important for speech synthesis.

[0046] Preparing for speech synthesis

[0047] The server sends the extracted text elements to a speech synthesis API (e.g., Google® Text-to-Speech API or Amazon Polly), which retrieves the corresponding audio data for each text element and combines them into a format that can be played back continuously.

[0048] Creation and provision of voice homepages

[0049] Once the final audio file is generated, the server stores it for delivery to the user. When the user accesses a specific URL, the server streams the audio file and delivers it to the user.

[0050] Specific examples

[0051] If elderly user Tanaka finds a cooking recipe on a web page but finds the text too small to read, this system can help. Tanaka accesses the system's page using his smartphone browser and enters a URL such as "https: / / example.com / recipes / spaghetti." The server retrieves the page at this URL, analyzes the HTML source, and extracts text elements such as "How to make spaghetti," "Step 1: Boil the pasta," and "Step 2: Make the sauce." These text elements are sent to a speech synthesis API, which obtains the audio data and then compiles it into a single audio file. By playing this audio file on his smartphone, Tanaka can listen to the recipe aloud.

[0052] In this way, by converting web pages rich in text and image information into audio format, even users who have difficulty using the Internet can easily access information, contributing to bridging the digital divide.

[0053] The processing flow will be explained below.

[0054] Step 1:

[0055] The user accesses the system interface using a terminal, enters the URL of the web page they want to listen to, requests speech conversion, and clicks the submit button.

[0056] Step 2:

[0057] The device sends the entered URL information to the server, requesting the server to retrieve the web page via an HTTP request.

[0058] Step 3:

[0059] The server retrieves the specified web page based on the URL received from the user. The server sends an HTTP request to retrieve the HTML source code of the web page.

[0060] Step 4:

[0061] The HTML source code retrieved by the server is analyzed. An HTML parser is used to analyze the structure of the web page and generate a DOM (Document Object Model).

[0062] Step 5:

[0063] The server extracts the main text elements from the parsed DOM. It identifies and extracts elements such as headings (h1, h2, h3...), paragraphs (p), links (a), lists (ul, ol), etc.

[0064] Step 6:

[0065] The server converts the extracted text elements into a format for sending to the speech synthesis API. The text elements are formatted into a format that the API can process.

[0066] Step 7:

[0067] The server sends each text element to the speech synthesis API and receives the corresponding audio data. For example, the server sends the text "Step 1: Boil the pasta" to the API and receives the audio data in response.

[0068] Step 8:

[0069] The server combines each acquired audio data into a format that can be played back continuously. The acquired individual audio data is concatenated in order to generate a single continuous audio file.

[0070] Step 9:

[0071] The server saves the generated audio file on the server. It generates an access URL for the audio file and prepares it so that it can be provided to the user.

[0072] Step 10:

[0073] The server returns the access URL for the audio file to the user, and sends a response to the device containing a link for the user to access the file.

[0074] Step 11:

[0075] The user accesses the provided URL on their device and plays the audio website. The user can then stream the audio file on their smartphone or PC and listen to the contents of the web page.

[0076] Through this series of processes, text-based web pages are converted into audio-based pages and presented to the user in an easy-to-use format.

[0077] Example 1

[0078] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0079] There is a need to make web page information easily accessible to the elderly and visually impaired. However, many current web pages rely on visual information provision, which causes access barriers for these users. It is particularly difficult to obtain information when the text is small and difficult to read, or when the layout is visually complex. Therefore, a system that provides web page content using audio is needed.

[0080] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0081] In this invention, the server includes means for accepting a URL specified by a user, means for retrieving the web page of the specified URL, means for analyzing the markup language source of the retrieved web page and extracting key text elements, means for transmitting the extracted text elements to a speech synthesis service and retrieving audio data, means for combining the retrieved audio data into an audio file that can be played continuously, and means for providing the generated audio file to the user. This allows even the elderly and visually impaired to hear the contents of the web page aloud, making it easier to access information.

[0082] "User" refers to an individual or organization that uses the System.

[0083] "URL" stands for Uniform Resource Locator and is an address that specifies the location of a web resource.

[0084] A "web page" is a document in HTML format that is published on the Internet and refers to the content that is displayed using a browser.

[0085] An "HTTP request" is a communication protocol that allows a client (user's browser) to request a specific resource from a server.

[0086] "HTML source" refers to text data that represents the structure and content of a web page written in a markup language.

[0087] An "HTML parser" is a software tool for analyzing HTML source and generating a DOM (Document Object Model).

[0088] "DOM" is an abbreviation for Document Object Model, and is a model that represents the structure of a web page in a tree format.

[0089] "Text elements" refer to text data such as headings, paragraphs, links, and lists that make up the content of a web page.

[0090] "Speech synthesis service" refers to APIs and software for converting input text data into voice data.

[0091] "Audio Data" refers to digital data in audio format generated by a speech synthesis service.

[0092] An "audio file" is a digital file for storing audio data, saved in a format that allows for continuous playback.

[0093] "Streaming playback" is a method of playing audio files in real time, where playback begins before the file is completely downloaded.

[0094] The system of the present invention allows elderly people, visually impaired people, etc. to use voice to access the contents of web pages. A specific embodiment of this system will be described below.

[0095] A user accesses the system using a device (PC, smartphone, tablet, etc.). First, the user opens the device's web browser and accesses the system's web page. Next, the user enters the URL of the web page they want to play by voice and presses the "Submit" button. This information is sent to the server.

[0096] The server receives a request sent by a user and retrieves the web page at the specified URL using an HTTP request. It then analyzes the retrieved HTML source and generates a Document Object Model (DOM) using an HTML parser (e.g., BeautifulSoup or jsoup). The server then extracts key text elements such as headings, paragraphs, links, and lists from the generated DOM.

[0097] The server then sends the extracted text elements to a speech synthesis service (e.g., Google Text-to-Speech API or Amazon Polly). The service generates the corresponding audio data and returns it to the server. The server then compiles the audio data into a continuous, playable audio file and saves it as a single audio file. Finally, the server provides the saved audio file at a specific URL, which users can access to stream the audio file.

[0098] As a concrete example, consider the case where an elderly person named Tanaka browses a recipe website on his smartphone. Tanaka accesses the system's page using his smartphone's browser, enters the URL "https: / / example.com / recipes / spaghetti" and submits it. The server retrieves the web page at that URL, analyzes the HTML source, and extracts text elements such as headings and paragraphs. These text elements are then sent to a speech synthesis service, which obtains the audio data and compiles it into a single audio file. Tanaka can then stream this audio file and listen to the recipe aloud.

[0099] Examples of prompts that give specific instructions to a generative AI model include:

[0100] 1. "Design a system that uses speech synthesis technology to provide audible content from web pages for the elderly."

[0101] 2. "Explain how to parse the contents of a web page retrieved from a specific URL and convert only the key text elements into audio data."

[0102] 3. "What are the steps in the process of converting text extracted from an HTML source into audio format using the Google Text-to-Speech API?"

[0103] As described above, this system allows elderly people and people with visual impairments to easily access web page content by voice. This system lowers the barriers to using the Internet and makes it easier to access information, thereby contributing to bridging the digital divide.

[0104] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0105] Step 1:

[0106] A user accesses the system using a terminal and enters the URL of the web page they want to play aloud. As input, the user opens a web browser, enters the URL in the designated input field, and presses the submit button. As output, the URL is sent to the server.

[0107] Specific behavior:

[0108] The user opens a browser on their PC or smartphone.

[0109] Enter "https: / / example.com" in the URL input field.

[0110] Click the "Send" button.

[0111] Step 2:

[0112] The server receives a URL request sent by a user and retrieves the web page at the specified URL. As input, it receives the URL sent by the user. As output, it retrieves the HTML source of the web page at the specified URL.

[0113] Specific behavior:

[0114] The server sends an HTTP GET request based on the URL received from the user.

[0115] The server receives the HTML source as a response.

[0116] Step 3:

[0117] The server parses the HTML source and generates a DOM (Document Object Model). It uses the HTML source as input and generates a DOM tree as output.

[0118] Specific behavior:

[0119] The server generates a DOM from the HTML source using an HTML parser (e.g., BeautifulSoup).

[0120] Step 4:

[0121] The server extracts the main text elements (headings, paragraphs, links, lists, etc.) from the generated DOM. It uses the generated DOM as input. It gets a list of extracted text elements as output.

[0122] Specific behavior:

[0123] The server extracts elements such as h1 tags, p tags, a tags, and ul tags from the DOM.

[0124] The extracted elements are compiled into a text list.

[0125] Step 5:

[0126] The server sends the extracted text elements to a speech synthesis service to obtain speech data. As input, it uses the extracted text elements. As output, it obtains the corresponding speech data.

[0127] Specific behavior:

[0128] The server sends the text data to a speech synthesis service (e.g., Google Text-to-Speech API).

[0129] The server receives the audio data from the service.

[0130] Step 6:

[0131] The server compiles the acquired audio data into a continuously playable audio file and saves the audio file. As input, it uses the acquired audio data. As output, it generates and saves a continuously playable audio file.

[0132] Specific behavior:

[0133] The server combines the audio data corresponding to each text element into a single audio file.

[0134] Save the merged audio file (e.g. audio / output.mp3).

[0135] Step 7:

[0136] The server provides the stored audio file at a specific URL so that it can be streamed when accessed by the user. As input, it uses the stored audio file. As output, it generates and provides a streaming playback URL that the user can access.

[0137] Specific behavior:

[0138] The server generates a URL to access the audio file (e.g., https: / / example.com / audio / output.mp3).

[0139] When a user accesses the URL, the server streams the audio file.

[0140] By following the above processing steps in order, this system is able to provide the contents of web pages to elderly people and people with visual impairments by voice.

[0141] (Application example 1)

[0142] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0143] The purpose of this invention is to enable elderly people and people with visual impairments to easily obtain product information in physical stores. In particular, the objective is to improve information accessibility by providing a means for visually impaired people and elderly people to listen to product descriptions and guidance by audio. Furthermore, conventional methods mainly use text information, which creates a visual barrier, and there is a need to eliminate this.

[0144] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0145] In this invention, the server includes means for accepting a URL specified by a user, means for retrieving the web page of the specified URL, means for analyzing the HTML source of the retrieved web page and extracting key text elements, means for transmitting the extracted text elements to a speech synthesis API to retrieve voice data, means for combining the retrieved voice data into a continuously playable voice file, means for providing the generated voice file to the user, and application means installed on smart glasses for providing product information in a physical store in voice format, thereby enabling visually impaired people and the elderly to easily obtain product information in a physical store by voice.

[0146] "Means for accepting a URL specified by a user" refers to a means for accepting a URL entered by a user to obtain information on a specific web page, and for the entire system to start processing based on that URL.

[0147] "Means for obtaining a web page at a specified URL" refers to means for accessing a URL entered by a user and obtaining the HTML source of the corresponding web page.

[0148] "Means for analyzing the HTML source of the acquired web page and extracting major text elements" refers to means for analyzing the acquired HTML source and extracting major text elements such as headings, paragraphs, links, and lists from it.

[0149] "Means for sending the extracted text elements to a speech synthesis API and obtaining speech data" refers to means for sending the extracted text elements to a speech synthesis API to convert them into speech, and obtaining the resulting speech data.

[0150] The "means for combining acquired audio data into a continuously reproducible audio file" refers to a means for combining acquired audio data into one continuous audio file.

[0151] The "means for providing the generated audio file to the user" refers to a means for providing the generated audio file in a form that the user can access.

[0152] "Application means installed on smart glasses for providing product information in audio format in physical stores" refers to a means for installing and executing an application on smart glasses for providing product information and guidance in audio format in physical stores.

[0153] The present invention provides a system that enables visually impaired people and elderly people to easily obtain product information by voice in a brick-and-mortar store, and provides voice guidance using smart glasses. The following describes an embodiment of the present invention.

[0154] System Configuration

[0155] The system includes a server, smart glasses, and a network communication means.

[0156] server

[0157] The server has the following functions:

[0158] 1. A means of accepting a URL specified by the user

[0159] The server receives the URL of the product information page sent from the smart glasses.

[0160] 2. How to get a web page from a specified URL

[0161] The server sends an HTTP request to the specified URL and retrieves the HTML source of the corresponding web page.

[0162] 3. A method for analyzing the HTML source of a retrieved web page and extracting key text elements

[0163] An HTML parser (e.g., BeautifulSoup) is used to parse the HTML source of the web page and extract key text elements such as headings, paragraphs, links, and lists.

[0164] 4. A method for sending the extracted text elements to a speech synthesis API and obtaining the speech data

[0165] The extracted text elements are sent to the Google Text-to-Speech API to obtain the audio data.

[0166] 5. A means of combining the captured audio data into a continuously playable audio file

[0167] The individual audio data is combined into one continuous audio file.

[0168] 6. How to provide the generated audio file to the user

[0169] The completed audio file is sent to the smart glasses so that the user can listen to it.

[0170] Smart Glasses

[0171] The smart glasses have the following features:

[0172] 1. How to enter a specified URL

[0173] The URL for product information is obtained by reading the product tag or QR code (registered trademark).

[0174] 2. How to send a URL to the server

[0175] Send the obtained URL to the server.

[0176] 3. A means to receive and play audio files from the server

[0177] The audio file received from the server is played to the user.

[0178] Hardware and software used

[0179] Smart glasses (e.g., Google Glass (registered trademark))

[0180] Server (e.g. Google Cloud Platform)

[0181] BeautifulSoup for HTML parsing

[0182] Google Text-to-Speech API for speech synthesis

[0183] Specific examples

[0184] For example, if a visually impaired customer is looking for a television in a physical store, they can use this system to obtain product information by voice. When the customer approaches a television wearing smart glasses and scans the QR code, the URL corresponding to that product is sent to the server. The server then analyzes the webpage based on the URL and extracts key text information. This information is converted into audio data using a speech synthesis API and sent back to the smart glasses. The smart glasses then play the received audio file, allowing the customer to listen to the television's product information aloud.

[0185] Prompt Sentence Examples

[0186] Get the page content of "https: / / example.com / store-product-page", extract headings, paragraphs, lists, etc., and convert them into Japanese audio files. The generated audio files are intended to enable elderly and visually impaired people to easily listen to product information in physical stores.

[0187] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0188] Step 1:

[0189] When a user searches for a product in a physical store, they use the smart glasses to read the QR code on the product, which retrieves the URL of the product information page, and the smart glasses then send the URL to the server.

[0190] Input: URL of product information read from QR code

[0191] Output: URL sent to the server

[0192] Specific operation: The smart glasses' camera is used to scan the QR code and the read URL is sent to the server via the network.

[0193] Step 2:

[0194] Based on the URL received from the smart glasses, the server sends an HTTP request to retrieve the HTML source of the specified web page.

[0195] Input: URL sent from smart glasses

[0196] Output: HTML source of the retrieved web page

[0197] What happens: The server makes an HTTP GET request to the URL to retrieve the HTML source of the web page.

[0198] Step 3:

[0199] The server parses the retrieved HTML source and extracts key text elements such as headings, paragraphs, links, and lists.

[0200] Input: HTML source of the web page

[0201] Output: Extracted main text elements

[0202] Specific behavior: Analyzes HTML source using an HTML parser such as BeautifulSoup and extracts the main text elements. Extracts text data for each element such as headings, paragraphs, and lists.

[0203] Step 4:

[0204] The server sends the extracted text elements to the Google Text-to-Speech API, which converts them into audio data.

[0205] Input: Extracted main text elements

[0206] Output: Audio data

[0207] Specific operation: Sends text data to the Google Text-to-Speech API and receives audio data as a response. Audio data is generated for each text element.

[0208] Step 5:

[0209] The server combines the acquired audio data into a continuously playable audio file.

[0210] Input: Audio data

[0211] Output: A combined audio file that can be played continuously

[0212] What it does: Concatenates multiple audio data files into a single audio file, allowing users to listen to information seamlessly.

[0213] Step 6:

[0214] The server sends the generated audio file to the smart glasses and provides it to the user.

[0215] Input: Concatenated audio files

[0216] Output: Audio file sent to smart glasses

[0217] Specific operation: The server sends the audio file to the smart glasses via the network, and the smart glasses receive and play it.

[0218] Specific examples

[0219] If a visually impaired customer is looking for a television in a physical store, they can put on their smart glasses, approach a television product, and scan the QR code. The URL of this QR code is sent to a server, which then retrieves the HTML source of the webpage from the URL. The server analyzes the HTML source and extracts headings, paragraphs, lists, etc., then converts this into audio data and concatenates it into a playable audio file. Finally, this audio file is sent to the smart glasses, allowing the customer to listen to detailed information about the television.

[0220] Prompt Sentence Examples

[0221] Get the page content of "https: / / example.com / store-product-page", extract headings, paragraphs, lists, etc., and convert them into Japanese audio files. The generated audio files are intended to enable elderly and visually impaired people to easily listen to product information in physical stores.

[0222] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0223] The system of the present invention provides easy-to-use voice-based web page access for elderly people, visually impaired people, and others. By recognizing the user's emotions and reflecting them in the voice output, the system provides a more natural and easy-to-listen voice. A specific embodiment of this system is described below.

[0224] System Overview

[0225] When a user accesses the system and wants to listen to the contents of a web page aloud, they first use a device (PC, smartphone, tablet, etc.). The user opens a web browser, accesses the system, and enters the URL of the web page they want to play aloud. This information is sent to the server, which retrieves the web page at the specified URL, converts it into audio format, and provides it to the user. The system also uses an emotion engine to recognize the user's emotions and apply them to the audio output.

[0226] Program processing

[0227] Page retrieval and analysis

[0228] The server receives a request sent by a user and retrieves the web page at the specified URL. Typically, the HTTP request is used to retrieve the HTML source of the web page. This HTML source becomes the data used to analyze the content of the web page.

[0229] Text Extraction and Structure Recognition

[0230] The server analyzes the retrieved HTML source and extracts key text elements such as headings, paragraphs, links, and lists. It uses an HTML parser to generate a Document Object Model (DOM) and extracts the necessary text elements based on this. These extracted text elements are important for speech synthesis.

[0231] Preparing for speech synthesis

[0232] The server sends the extracted text elements to a speech synthesis API (such as Google Text-to-Speech API or Amazon Polly), which retrieves the corresponding audio data for each text element and combines them into a format that can be played back continuously. At this point, an emotion engine is used to simultaneously process information for analyzing the user's emotions.

[0233] Emotion Recognition and Voice Adjustment

[0234] The emotion engine receives the user's input voice, text, and non-verbal input information (facial expressions, gestures, etc.). This allows it to analyze the user's emotions and apply emotional expressions to the acquired voice data. For example, if the user is expressing a happy emotion, the voice data generated by the speech synthesis API will reflect joy and happiness.

[0235] Creation and provision of voice homepages

[0236] Once the final audio file is generated, the server stores it for delivery to the user. When the user accesses a specific URL, the server streams the audio file and delivers it to the user.

[0237] Specific examples

[0238] For example, if an elderly person named Tanaka finds a recipe on a web page but finds the text too small to read, this system can help. Tanaka accesses the system's page using a smartphone browser and enters a URL such as "https: / / example.com / recipes / spaghetti." The server retrieves the page and analyzes the HTML source to extract text elements such as "How to make spaghetti," "Step 1: Boil the pasta," and "Step 2: Make the sauce." These text elements are then sent to a speech synthesis API, which acquires the audio data and compiles it into a single audio file. At the same time, the emotion engine recognizes Tanaka's emotions from her input voice and facial expressions and adjusts the audio data based on those emotions. By playing this audio file on her smartphone, Tanaka can hear the recipe aloud, with the audio appropriately tailored to her current emotions.

[0239] In this way, web pages rich in text and images can be converted into audio format, and the user's emotions can be taken into consideration, making it easier for users who have difficulty using the Internet to access information. Furthermore, the more user-friendly audio output further promotes understanding and use of information.

[0240] The processing flow will be explained below.

[0241] Step 1:

[0242] The user accesses the system interface using a terminal, enters the URL of the web page they want to listen to, requests speech conversion, and clicks the submit button.

[0243] Step 2:

[0244] The device sends the entered URL information to the server, requesting the server to retrieve the web page via an HTTP request.

[0245] Step 3:

[0246] The server retrieves the specified web page based on the URL received from the user. The server sends an HTTP request to retrieve the HTML source code of the web page.

[0247] Step 4:

[0248] The HTML source code retrieved by the server is analyzed. An HTML parser is used to analyze the structure of the web page and generate a DOM (Document Object Model).

[0249] Step 5:

[0250] The server extracts the main text elements from the parsed DOM. It identifies and extracts elements such as headings (h1, h2, h3...), paragraphs (p), links (a), lists (ul, ol), etc.

[0251] Step 6:

[0252] The server converts the extracted text elements into a format for sending to the speech synthesis API. The text elements are formatted into a format that the API can process.

[0253] Step 7:

[0254] The server sends each text element to the speech synthesis API and receives the corresponding audio data. For example, the server sends the text "Step 1: Boil the pasta" to the API and receives the audio data in response.

[0255] Step 8:

[0256] The server activates the emotion engine based on the voice data obtained from the speech synthesis API. The emotion engine receives the user's input voice, text, and non-verbal input information (facial expressions, gestures, etc.).

[0257] Step 9:

[0258] The emotion engine analyzes the user's emotions and adjusts the voice data based on the analysis, for example, adjusting the tone and intonation of the voice to reflect emotions of joy or sadness.

[0259] Step 10:

[0260] The server combines the emotion-adjusted audio data into a continuous, playable format. Each adjusted audio data is concatenated in order to generate a single continuous audio file.

[0261] Step 11:

[0262] The server saves the generated audio file on the server. It generates an access URL for the audio file and prepares it so that it can be provided to the user.

[0263] Step 12:

[0264] The server returns the access URL for the audio file to the user, and sends a response to the device containing a link for the user to access the file.

[0265] Step 13:

[0266] The user accesses the provided URL on their device and plays the audio website. The user can then stream the audio file on their smartphone or PC and listen to the contents of the web page.

[0267] Through this series of processes, text-based web pages are converted into voice-based text, and the voice is adjusted to match the user's emotions, providing information in a more natural and easy-to-listen voice.

[0268] Example 2

[0269] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0270] For the elderly and visually impaired, reading text information on the Internet can be difficult, often resulting in visual stress. Furthermore, conventional text-to-speech systems struggle to provide natural-sounding speech that reflects the user's emotions. This can lead to a mechanical and unfamiliar experience. Furthermore, there are only a limited number of systems that can provide speech that reflects the user's emotions, limiting their range of use.

[0271] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0272] In this invention, the server includes means for accepting a URL specified by a user, means for retrieving the web page at the specified URL, means for analyzing the HTML source of the retrieved web page and extracting key text elements, means for transmitting the extracted text elements to a speech synthesis API and retrieving voice data, means for recognizing the user's emotions, means for applying emotional expressions to the retrieved voice data, means for combining the retrieved voice data into a continuously playable voice file, and means for providing the generated voice file to the user. This enables natural voice reading that takes the user's emotions into consideration. This allows a wide range of users, including the elderly and visually impaired, to receive information more easily.

[0273] "User" refers to a person who wishes to use the system to have a web page read aloud.

[0274] "URL" refers to a Uniform Resource Locator that indicates the address of a web page.

[0275] "Means" refers to a device or program provided to realize a specific function.

[0276] "Web page" refers to a document in HTML format that is published on the Internet.

[0277] "HTML source" refers to the code written in HyperText Markup Language that makes up the content of a Web page.

[0278] "Text elements" refers to textual information such as headings, paragraphs, links, and lists in HTML source.

[0279] "Speech synthesis API" refers to an application programming interface for converting text data into speech data.

[0280] "Audio data" refers to text information that has been converted into audio format.

[0281] "Means for recognizing emotions" refers to sensing technologies and algorithms for analyzing a user's emotional state.

[0282] "Means for applying emotional expressions" refers to technology that adjusts the intonation and tone of voice data based on the analyzed emotions.

[0283] "Audio file" refers to a file that contains audio data in a format that can be played continuously.

[0284] "Means of providing" refers to the technology and infrastructure required to transmit or stream the generated audio file to the user.

[0285] The system of the present invention provides voice-based web page access that is easy to use for elderly people, visually impaired people, etc. This system recognizes the user's emotions and reflects them in the voice output, providing a more natural and easy-to-listen voice.

[0286] Hardware and Software Configuration

[0287] server

[0288] The server has the following features:

[0289] Accepts a URL specified by the user.

[0290] The web page at the specified URL is retrieved via an HTTP request.

[0291] The HTML source of the retrieved web page is analyzed to extract the main text elements. For the specific analysis, an HTML parser such as BeautifulSoup (a Python library) is used.

[0292] The extracted text elements are sent to a speech synthesis API such as Google Text-to-Speech API or Amazon Polly to obtain audio data.

[0293] An emotion engine (e.g., Microsoft® Cognitive Services Emotion API) to recognize user emotions.

[0294] Emotional expressions are applied to the audio data, and the adjusted audio data is compiled into an audio file that can be played continuously.

[0295] The generated audio file is saved for provision to the user, and a dedicated URL is generated.

[0296] Terminal

[0297] Users use devices such as PCs, smartphones, and tablets. These devices have a web browser installed. Users access the system through the browser and enter a URL.

[0298] Specific examples

[0299] For example, Mr. Tanaka, an elderly person, accesses the system using a web browser and enters a URL such as "https: / / example.com / recipes / spaghetti."

[0300] 1. The user (Mr. Tanaka) opens a browser on his / her PC, smartphone, or tablet and accesses the system's web page.

[0301] 2. Tanaka enters the URL of the web page she wants to play aloud (for example, "https: / / example.com / recipes / spaghetti").

[0302] 3. The server retrieves the web page at the specified URL through an HTTP request and parses the HTML source.

[0303] 4. The server extracts key text elements such as headings, paragraphs, links, and lists from the parsed HTML source.

[0304] 5. The extracted text elements are sent to a speech synthesis API, such as Google Text-to-Speech API or Amazon Polly, to obtain the corresponding audio data.

[0305] 6. The emotion engine analyzes Tanaka's emotions. For example, if Tanaka says "I'm looking forward to it" in the input voice, the voice is analyzed and the emotion is identified.

[0306] 7. Tanaka's emotions are reflected in the acquired voice data, and adjusted voice data is generated.

[0307] 8. The server compiles the generated audio data into a format that can be played continuously, and generates a single audio file.

[0308] 9. The final audio file is saved to the server and a unique URL (e.g., "https: / / example.com / voice / spaghetti") is generated.

[0309] 10. By accessing the dedicated URL, Tanaka can stream the audio file and hear the contents of the web page in audio, which appropriately reflects Tanaka's emotions.

[0310] Prompt Sentence Examples

[0311] Below are some example prompts to be input to the generative AI model:

[0312] "Play a web page specified by the user's URL (e.g., https: / / example.com / recipes / spaghetti) aloud, providing audio that reflects the user's emotions."

[0313] This system makes it possible to provide speech synthesis that takes into consideration the emotions of users, including the elderly and visually impaired, even on web pages with a wealth of text and image information. This makes it easier for users who have difficulty using the Internet to access information, and promotes understanding of the information through friendly speech output.

[0314] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0315] Step 1:

[0316] The user uses a terminal to open a web browser. They access a web page on the system and enter the URL of the web page they want to play aloud (e.g., "https: / / example.com / recipes / spaghetti"). The entered URL is sent to the server as a request. Input: The URL entered by the user. Output: The URL request to the server.

[0317] Step 2:

[0318] The terminal sends an HTTP request containing the URL information entered by the user to the server. The server receives the request and retrieves the web page at the specified URL. In fact, it downloads the HTML source of the web page using an HTTP GET request. Input: URL request. Output: Retrieved HTML source.

[0319] Step 3:

[0320] The server parses the retrieved HTML source. It uses an HTML parser (e.g., BeautifulSoup) to convert the HTML source into a Document Object Model (DOM) and extracts major text elements such as headings, paragraphs, links, and lists. Input: HTML source. Output: Extracted text elements.

[0321] Step 4:

[0322] The server sends the extracted text elements to a speech synthesis API (for example, Google Text-to-Speech API or Amazon Polly) to obtain audio data. Specifically, each text element is sent individually to the API to obtain audio data for each. Input: Text elements. Output: Audio data.

[0323] Step 5:

[0324] The server uses an emotion engine to analyze the user's emotions. This emotion information is generated based on the user's input voice, text, and non-verbal information (facial expressions, gestures, etc.). Input: Information about the user's emotions. Output: Analyzed emotion data.

[0325] Step 6:

[0326] The server applies emotional expressions to the acquired voice data. Based on the analyzed emotional data, it adjusts the intonation and tone of the voice data. This generates a voice that reflects the user's emotions. Input: Voice data, emotional data. Output: Adjusted voice data.

[0327] Step 7:

[0328] The server combines the adjusted audio data into a single audio file. It concatenates the audio data for multiple text elements into a single file to create a continuously playable audio file. Input: Adjusted audio data. Output: Audio file.

[0329] Step 8:

[0330] The server saves the generated audio file and generates a URL that users can access. Users can stream the audio file by accessing the URL. Input: Audio file. Output: Accessible URL.

[0331] This specific process allows users to listen to web pages in a natural voice that reflects their emotions, providing an environment where even the elderly and visually impaired can easily access information.

[0332] (Application example 2)

[0333] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0334] Users with visual or physical disabilities often have difficulty accessing and understanding the content of web pages. Furthermore, conventional text-to-speech systems do not provide natural-sounding speech output that reflects the user's emotions, making it difficult for them to understand the audio information. This makes it particularly difficult for users with visual or physical disabilities to use online shopping sites.

[0335] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0336] In this invention, the server includes means for accepting a URL specified by a user, means for retrieving the web page of the specified URL, means for analyzing the HTML source of the retrieved web page and extracting key text elements, means for transmitting the extracted text elements to a speech synthesis API and retrieving voice data, means for combining the retrieved voice data into a continuously playable voice file, means for recognizing the user's emotion and adjusting the voice output based on the emotion, and means for providing the generated voice file to the user, thereby enabling users with visual or physical disabilities to understand the content of the web page through natural voice that corresponds to their emotion.

[0337] The "specified URL" is the address of the web page that the user wants to access.

[0338] "Getting a web page" means downloading the HTML source from the specified URL.

[0339] "HTML source" is data written in a markup language that describes the structure and content of a web page.

[0340] A "text element" is an element in the HTML source that contains text information, such as a heading, paragraph, link, or list.

[0341] A "speech synthesis API" is an application program interface used to convert text data into speech data.

[0342] "Audio data" means playable sound data generated using a speech synthesis API.

[0343] An "audio file" is audio data combined into a format that can be played back continuously.

[0344] "Emotion recognition" refers to analyzing a user's current emotional state from their voice or other input.

[0345] "Adjusting the audio output" refers to changing the tone and rate of the generated audio based on the perceived user emotion.

[0346] This invention is a system that enables users with visual or physical disabilities to access web pages by voice. The system retrieves a specified URL, analyzes its HTML source to extract important text elements, and generates voice data using a speech synthesis API. It also recognizes the user's emotions and adjusts the generated voice output to provide it to the user.

[0347] Overall system configuration

[0348] The system mainly consists of the following steps:

[0349] 1. Accepts a URL specified by the user.

[0350] 2. The server retrieves the web page at the specified URL.

[0351] 3. Analyze the obtained HTML source and extract the main text elements.

[0352] 4. The extracted text elements are sent to the speech synthesis API to obtain the speech data.

[0353] 5. Combine the audio data into a continuously playable audio file.

[0354] 6. Recognize user emotions and adjust voice output based on those emotions.

[0355] 7. Provide the final audio file to the user.

[0356] Hardware and software used

[0357] Hardware: The system can be accessed from devices such as PCs, smartphones, and tablets.

[0358] Software: Use "Google Text-to-Speech API" or "Amazon Polly" for speech synthesis. Use an HTML parser such as "BeautifulSoup" for HTML analysis. Use the "emotion_recognition" module for emotion recognition.

[0359] How it works

[0360] 1. URL acceptance

[0361] Users access the system from a device such as a PC or smartphone and enter the specified URL, which is then sent to the server.

[0362] 2. Fetching a Web Page

[0363] The server sends an HTTP request to the specified URL and retrieves the HTML source of the page, using the "requests" library.

[0364] 3. HTML source analysis

[0365] The server parses the HTML source and extracts the main text elements (headings, paragraphs, links, lists, etc.). This analysis is done using "BeautifulSoup."

[0366] 4. Generating Audio Data

[0367] The extracted text elements are sent to a speech synthesis API such as the Google Text-to-Speech API to generate audio data.

[0368] 5. Generate audio files

[0369] The acquired audio data is combined into a format that can be played continuously, and an audio file is generated.

[0370] 6. Emotion recognition and voice output adjustment

[0371] It analyzes emotions from the user's voice input and other non-verbal information and adjusts the generated voice data accordingly, providing a more natural and understandable voice output.

[0372] 7. Providing audio files

[0373] The final generated audio file is provided in a user-accessible format.

[0374] Specific examples

[0375] For example, if a visually impaired user wants to search for a new product on an online shopping site, they can use this system to easily obtain information. When the user enters the URL "https: / / example.com / product / smartphone" into the system, the system analyzes the page and reads out information such as "latest smartphone," "price: 50,000 yen," and "equipped with a high-quality camera." At the same time, it analyzes the user's voice input to recognize emotions and provides information in a tone that corresponds to that emotion.

[0376] Prompt Sentence Examples

[0377] Example prompt:

[0378] A user wants to check product details on an online shopping website. The URL is "https: / / example.com / product / smartphone." The user utters "Tell me about my new smartphone." The system retrieves the webpage, analyzes key text elements, and provides an emotionally-aware speech readout.

[0379] Expected results:

[0380] Information such as "This is the latest smartphone. It costs 50,000 yen and has a high-quality camera" is read out to the user in a tone that reflects their emotions.

[0381] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0382] Step 1:

[0383] Enter the URL specified by the user.

[0384] A user accesses the system using a device such as a PC or smartphone and enters a specified URL. The user's input information is sent to the server. In this step, the input is the "specified URL" and the output is sending the URL to the server. Specifically, the user enters the URL into the input form from the device's browser and clicks the submit button.

[0385] Step 2:

[0386] The server retrieves the web page at the specified URL.

[0387] The server sends an HTTP request to retrieve the HTML source of the web page based on the URL received from the user. The input is the "specified URL" and the output is the "HTML source of the web page." This process uses the "requests" library, specifically requests.get() to send the HTTP request.

[0388] Step 3:

[0389] The obtained HTML source is analyzed and the main text elements are extracted.

[0390] The server parses the retrieved HTML source using the "BeautifulSoup" library and extracts key text elements such as headings, paragraphs, links, and lists. The input is the "HTML source of the web page" and the output is the "extracted text elements." Specifically, it parses the HTML source using BeautifulSoup and extracts the required text elements using the find_all() method.

[0391] Step 4:

[0392] The extracted text elements are sent to a speech synthesis API to obtain speech data.

[0393] The server sends the extracted text elements to a speech synthesis API such as "Google Text-to-Speech API" or "Amazon Polly" to obtain the audio data. The input is the extracted text elements, and the output is audio data. Specifically, the text data is sent to the speech synthesis API endpoint, and audio data is received as a response.

[0394] Step 5:

[0395] The acquired audio data is combined into a continuously playable audio file.

[0396] The server combines the acquired audio data into a format that can be played continuously to generate an audio file. The input is "audio data" and the output is an "audio file." Specifically, the audio data is combined in order and saved as a single audio file.

[0397] Step 6:

[0398] To recognize a user's emotions and adjust audio output based on the emotions.

[0399] The server analyzes the user's voice input and non-verbal information to recognize the user's emotions. It uses the "emotion_recognition" module to perform emotion analysis and adjusts the tone and speed of the voice data based on the results. The input is an audio file and emotional information, and the output is an adjusted audio file. Specifically, it applies an emotion recognition algorithm and changes the parameters of the speech synthesis API according to the user's emotions.

[0400] Step 7:

[0401] The final audio file is provided to the user.

[0402] The server then provides the final audio file to the user in streaming format. The input is the "adjusted audio file" and the output is the "audio provided to the user." Specifically, the audio file is sent as an HTTP response and is played back in real time on the device.

[0403] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0404] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0405] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0406] [Second embodiment]

[0407] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0408] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0409] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0410] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0411] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0412] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0413] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0414] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0415] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0416] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0417] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0418] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0419] The system of the present invention provides voice-based web page access that is easy to use for the elderly, the visually impaired, etc. A specific embodiment of this system will be described below.

[0420] System Overview

[0421] When a user accesses the system and wants to listen to the contents of a web page by audio, they first use a device (PC, smartphone, tablet, etc.). The user opens a web browser, accesses the system, and enters the URL of the web page they want to play by audio. This information is sent to the server, which retrieves the web page at the specified URL, converts it into audio format, and provides it to the user.

[0422] Program processing

[0423] Page retrieval and analysis

[0424] The server receives a request sent by a user and retrieves the web page at the specified URL. Typically, the HTTP request is used to retrieve the HTML source of the web page. This HTML source becomes the data used to analyze the content of the web page.

[0425] Text Extraction and Structure Recognition

[0426] The server analyzes the retrieved HTML source and extracts key text elements such as headings, paragraphs, links, and lists. It uses an HTML parser to generate a Document Object Model (DOM) and extracts the necessary text elements based on this. These extracted text elements are important for speech synthesis.

[0427] Preparing for speech synthesis

[0428] The server sends the extracted text elements to a speech synthesis API (e.g., Google Text-to-Speech API or Amazon Polly), which retrieves the corresponding audio data for each text element and combines them into a format that can be played back continuously.

[0429] Creation and provision of voice homepages

[0430] Once the final audio file is generated, the server stores it for delivery to the user. When the user accesses a specific URL, the server streams the audio file and delivers it to the user.

[0431] Specific examples

[0432] If elderly user Tanaka finds a cooking recipe on a web page but finds the text too small to read, this system can help. Tanaka accesses the system's page using his smartphone browser and enters a URL such as "https: / / example.com / recipes / spaghetti." The server retrieves the page at this URL, analyzes the HTML source, and extracts text elements such as "How to make spaghetti," "Step 1: Boil the pasta," and "Step 2: Make the sauce." These text elements are sent to a speech synthesis API, which obtains the audio data and then compiles it into a single audio file. By playing this audio file on his smartphone, Tanaka can listen to the recipe aloud.

[0433] In this way, by converting web pages rich in text and image information into audio format, even users who have difficulty using the Internet can easily access information, contributing to bridging the digital divide.

[0434] The processing flow will be explained below.

[0435] Step 1:

[0436] The user accesses the system interface using a terminal, enters the URL of the web page they want to listen to, requests speech conversion, and clicks the submit button.

[0437] Step 2:

[0438] The device sends the entered URL information to the server, requesting the server to retrieve the web page via an HTTP request.

[0439] Step 3:

[0440] The server retrieves the specified web page based on the URL received from the user. The server sends an HTTP request to retrieve the HTML source code of the web page.

[0441] Step 4:

[0442] The HTML source code retrieved by the server is analyzed. An HTML parser is used to analyze the structure of the web page and generate a DOM (Document Object Model).

[0443] Step 5:

[0444] The server extracts the main text elements from the parsed DOM. It identifies and extracts elements such as headings (h1, h2, h3...), paragraphs (p), links (a), lists (ul, ol), etc.

[0445] Step 6:

[0446] The server converts the extracted text elements into a format for sending to the speech synthesis API. The text elements are formatted into a format that the API can process.

[0447] Step 7:

[0448] The server sends each text element to the speech synthesis API and receives the corresponding audio data. For example, the server sends the text "Step 1: Boil the pasta" to the API and receives the audio data in response.

[0449] Step 8:

[0450] The server combines each acquired audio data into a format that can be played back continuously. The acquired individual audio data is concatenated in order to generate a single continuous audio file.

[0451] Step 9:

[0452] The server saves the generated audio file on the server. It generates an access URL for the audio file and prepares it so that it can be provided to the user.

[0453] Step 10:

[0454] The server returns the access URL for the audio file to the user, and sends a response to the device containing a link for the user to access the file.

[0455] Step 11:

[0456] The user accesses the provided URL on their device and plays the audio website. The user can then stream the audio file on their smartphone or PC and listen to the contents of the web page.

[0457] Through this series of processes, text-based web pages are converted into audio-based pages and presented to the user in an easy-to-use format.

[0458] Example 1

[0459] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0460] There is a need to make web page information easily accessible to the elderly and visually impaired. However, many current web pages rely on visual information provision, which causes access barriers for these users. It is particularly difficult to obtain information when the text is small and difficult to read, or when the layout is visually complex. Therefore, a system that provides web page content using audio is needed.

[0461] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0462] In this invention, the server includes means for accepting a URL specified by a user, means for retrieving the web page of the specified URL, means for analyzing the markup language source of the retrieved web page and extracting key text elements, means for transmitting the extracted text elements to a speech synthesis service and retrieving audio data, means for combining the retrieved audio data into an audio file that can be played continuously, and means for providing the generated audio file to the user. This allows even the elderly and visually impaired to hear the contents of the web page aloud, making it easier to access information.

[0463] "User" refers to an individual or organization that uses the System.

[0464] "URL" stands for Uniform Resource Locator and is an address that specifies the location of a web resource.

[0465] A "web page" is a document in HTML format that is published on the Internet and refers to the content that is displayed using a browser.

[0466] An "HTTP request" is a communication protocol that allows a client (user's browser) to request a specific resource from a server.

[0467] "HTML source" refers to text data that represents the structure and content of a web page written in a markup language.

[0468] An "HTML parser" is a software tool for analyzing HTML source and generating a DOM (Document Object Model).

[0469] "DOM" is an abbreviation for Document Object Model, and is a model that represents the structure of a web page in a tree format.

[0470] "Text elements" refer to text data such as headings, paragraphs, links, and lists that make up the content of a web page.

[0471] "Speech synthesis service" refers to APIs and software for converting input text data into voice data.

[0472] "Audio Data" refers to digital data in audio format generated by a speech synthesis service.

[0473] An "audio file" is a digital file for storing audio data, saved in a format that allows for continuous playback.

[0474] "Streaming playback" is a method of playing audio files in real time, where playback begins before the file is completely downloaded.

[0475] The system of the present invention allows elderly people, visually impaired people, etc. to use voice to access the contents of web pages. A specific embodiment of this system will be described below.

[0476] A user accesses the system using a device (PC, smartphone, tablet, etc.). First, the user opens the device's web browser and accesses the system's web page. Next, the user enters the URL of the web page they want to play by voice and presses the "Submit" button. This information is sent to the server.

[0477] The server receives a request sent by a user and retrieves the web page at the specified URL using an HTTP request. It then analyzes the retrieved HTML source and generates a Document Object Model (DOM) using an HTML parser (e.g., BeautifulSoup or jsoup). The server then extracts key text elements such as headings, paragraphs, links, and lists from the generated DOM.

[0478] The server then sends the extracted text elements to a speech synthesis service (e.g., Google Text-to-Speech API or Amazon Polly). The service generates the corresponding audio data and returns it to the server. The server then compiles the audio data into a continuous, playable audio file and saves it as a single audio file. Finally, the server provides the saved audio file at a specific URL, which users can access to stream the audio file.

[0479] As a concrete example, consider the case where an elderly person named Tanaka browses a recipe website on his smartphone. Tanaka accesses the system's page using his smartphone's browser, enters the URL "https: / / example.com / recipes / spaghetti" and submits it. The server retrieves the web page at that URL, analyzes the HTML source, and extracts text elements such as headings and paragraphs. These text elements are then sent to a speech synthesis service, which obtains the audio data and compiles it into a single audio file. Tanaka can then stream this audio file and listen to the recipe aloud.

[0480] Examples of prompts that give specific instructions to a generative AI model include:

[0481] 1. "Design a system that uses speech synthesis technology to provide audible content from web pages for the elderly."

[0482] 2. "Explain how to parse the contents of a web page retrieved from a specific URL and convert only the key text elements into audio data."

[0483] 3. "What are the steps in the process of converting text extracted from an HTML source into audio format using the Google Text-to-Speech API?"

[0484] As described above, this system allows elderly people and people with visual impairments to easily access web page content by voice. This system lowers the barriers to using the Internet and makes it easier to access information, thereby contributing to bridging the digital divide.

[0485] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0486] Step 1:

[0487] A user accesses the system using a terminal and enters the URL of the web page they want to play aloud. As input, the user opens a web browser, enters the URL in the designated input field, and presses the submit button. As output, the URL is sent to the server.

[0488] Specific behavior:

[0489] The user opens a browser on their PC or smartphone.

[0490] Enter "https: / / example.com" in the URL input field.

[0491] Click the "Send" button.

[0492] Step 2:

[0493] The server receives a URL request sent by a user and retrieves the web page at the specified URL. As input, it receives the URL sent by the user. As output, it retrieves the HTML source of the web page at the specified URL.

[0494] Specific behavior:

[0495] The server sends an HTTP GET request based on the URL received from the user.

[0496] The server receives the HTML source as a response.

[0497] Step 3:

[0498] The server parses the HTML source and generates a DOM (Document Object Model). It uses the HTML source as input and generates a DOM tree as output.

[0499] Specific behavior:

[0500] The server generates a DOM from the HTML source using an HTML parser (e.g., BeautifulSoup).

[0501] Step 4:

[0502] The server extracts the main text elements (headings, paragraphs, links, lists, etc.) from the generated DOM. It uses the generated DOM as input. It gets a list of extracted text elements as output.

[0503] Specific behavior:

[0504] The server extracts elements such as h1 tags, p tags, a tags, and ul tags from the DOM.

[0505] The extracted elements are compiled into a text list.

[0506] Step 5:

[0507] The server sends the extracted text elements to a speech synthesis service to obtain speech data. As input, it uses the extracted text elements. As output, it obtains the corresponding speech data.

[0508] Specific behavior:

[0509] The server sends the text data to a speech synthesis service (e.g., Google Text-to-Speech API).

[0510] The server receives the audio data from the service.

[0511] Step 6:

[0512] The server compiles the acquired audio data into a continuously playable audio file and saves the audio file. As input, it uses the acquired audio data. As output, it generates and saves a continuously playable audio file.

[0513] Specific behavior:

[0514] The server combines the audio data corresponding to each text element into a single audio file.

[0515] Save the merged audio file (e.g. audio / output.mp3).

[0516] Step 7:

[0517] The server provides the stored audio file at a specific URL so that it can be streamed when accessed by the user. As input, it uses the stored audio file. As output, it generates and provides a streaming playback URL that the user can access.

[0518] Specific behavior:

[0519] The server generates a URL to access the audio file (e.g., https: / / example.com / audio / output.mp3).

[0520] When a user accesses the URL, the server streams the audio file.

[0521] By following the above processing steps in order, this system is able to provide the contents of web pages to elderly people and people with visual impairments by voice.

[0522] (Application example 1)

[0523] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0524] The purpose of this invention is to enable elderly people and people with visual impairments to easily obtain product information in physical stores. In particular, the objective is to improve information accessibility by providing a means for visually impaired people and elderly people to listen to product descriptions and guidance by audio. Furthermore, conventional methods mainly use text information, which creates a visual barrier, and there is a need to eliminate this.

[0525] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0526] In this invention, the server includes means for accepting a URL specified by a user, means for retrieving the web page of the specified URL, means for analyzing the HTML source of the retrieved web page and extracting key text elements, means for transmitting the extracted text elements to a speech synthesis API to retrieve voice data, means for combining the retrieved voice data into a continuously playable voice file, means for providing the generated voice file to the user, and application means installed on smart glasses for providing product information in a physical store in voice format, thereby enabling visually impaired people and the elderly to easily obtain product information in a physical store by voice.

[0527] "Means for accepting a URL specified by a user" refers to a means for accepting a URL entered by a user to obtain information on a specific web page, and for the entire system to start processing based on that URL.

[0528] "Means for obtaining a web page at a specified URL" refers to means for accessing a URL entered by a user and obtaining the HTML source of the corresponding web page.

[0529] "Means for analyzing the HTML source of the acquired web page and extracting major text elements" refers to means for analyzing the acquired HTML source and extracting major text elements such as headings, paragraphs, links, and lists from it.

[0530] "Means for sending the extracted text elements to a speech synthesis API and obtaining speech data" refers to means for sending the extracted text elements to a speech synthesis API to convert them into speech, and obtaining the resulting speech data.

[0531] The "means for combining acquired audio data into a continuously reproducible audio file" refers to a means for combining acquired audio data into one continuous audio file.

[0532] The "means for providing the generated audio file to the user" refers to a means for providing the generated audio file in a form that the user can access.

[0533] "Application means installed on smart glasses for providing product information in audio format in physical stores" refers to a means for installing and executing an application on smart glasses for providing product information and guidance in audio format in physical stores.

[0534] The present invention provides a system that enables visually impaired people and elderly people to easily obtain product information by voice in a brick-and-mortar store, and provides voice guidance using smart glasses. The following describes an embodiment of the present invention.

[0535] System Configuration

[0536] The system includes a server, smart glasses, and a network communication means.

[0537] server

[0538] The server has the following functions:

[0539] 1. A means of accepting a URL specified by the user

[0540] The server receives the URL of the product information page sent from the smart glasses.

[0541] 2. How to get a web page from a specified URL

[0542] The server sends an HTTP request to the specified URL and retrieves the HTML source of the corresponding web page.

[0543] 3. A method for analyzing the HTML source of a retrieved web page and extracting key text elements

[0544] An HTML parser (e.g., BeautifulSoup) is used to parse the HTML source of the web page and extract key text elements such as headings, paragraphs, links, and lists.

[0545] 4. A method for sending the extracted text elements to a speech synthesis API and obtaining the speech data

[0546] The extracted text elements are sent to the Google Text-to-Speech API to obtain the audio data.

[0547] 5. A means of combining the captured audio data into a continuously playable audio file

[0548] The individual audio data is combined into one continuous audio file.

[0549] 6. How to provide the generated audio file to the user

[0550] The completed audio file is sent to the smart glasses so that the user can listen to it.

[0551] Smart Glasses

[0552] The smart glasses have the following features:

[0553] 1. How to enter a specified URL

[0554] The URL for product information is obtained by reading the product tag or QR code.

[0555] 2. How to send a URL to the server

[0556] Send the obtained URL to the server.

[0557] 3. A means to receive and play audio files from the server

[0558] The audio file received from the server is played to the user.

[0559] Hardware and software used

[0560] Smart glasses (e.g. Google Glass)

[0561] Server (e.g. Google Cloud Platform)

[0562] BeautifulSoup for HTML parsing

[0563] Google Text-to-Speech API for speech synthesis

[0564] Specific examples

[0565] For example, if a visually impaired customer is looking for a television in a physical store, they can use this system to obtain product information by voice. When the customer approaches a television wearing smart glasses and scans the QR code, the URL corresponding to that product is sent to the server. The server then analyzes the webpage based on the URL and extracts key text information. This information is converted into audio data using a speech synthesis API and sent back to the smart glasses. The smart glasses then play the received audio file, allowing the customer to listen to the television's product information aloud.

[0566] Prompt Sentence Examples

[0567] Get the page content of "https: / / example.com / store-product-page", extract headings, paragraphs, lists, etc., and convert them into Japanese audio files. The generated audio files are intended to enable elderly and visually impaired people to easily listen to product information in physical stores.

[0568] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0569] Step 1:

[0570] When a user searches for a product in a physical store, they use the smart glasses to read the QR code on the product, which retrieves the URL of the product information page, and the smart glasses then send the URL to the server.

[0571] Input: URL of product information read from QR code

[0572] Output: URL sent to the server

[0573] Specific operation: The smart glasses' camera is used to scan the QR code and the read URL is sent to the server via the network.

[0574] Step 2:

[0575] Based on the URL received from the smart glasses, the server sends an HTTP request to retrieve the HTML source of the specified web page.

[0576] Input: URL sent from smart glasses

[0577] Output: HTML source of the retrieved web page

[0578] What happens: The server makes an HTTP GET request to the URL to retrieve the HTML source of the web page.

[0579] Step 3:

[0580] The server parses the retrieved HTML source and extracts key text elements such as headings, paragraphs, links, and lists.

[0581] Input: HTML source of the web page

[0582] Output: Extracted main text elements

[0583] Specific behavior: Analyzes HTML source using an HTML parser such as BeautifulSoup and extracts the main text elements. Extracts text data for each element such as headings, paragraphs, and lists.

[0584] Step 4:

[0585] The server sends the extracted text elements to the Google Text-to-Speech API, which converts them into audio data.

[0586] Input: Extracted main text elements

[0587] Output: Audio data

[0588] Specific operation: Sends text data to the Google Text-to-Speech API and receives audio data as a response. Audio data is generated for each text element.

[0589] Step 5:

[0590] The server combines the acquired audio data into a continuously playable audio file.

[0591] Input: Audio data

[0592] Output: A combined audio file that can be played continuously

[0593] What it does: Concatenates multiple audio data files into a single audio file, allowing users to listen to information seamlessly.

[0594] Step 6:

[0595] The server sends the generated audio file to the smart glasses and provides it to the user.

[0596] Input: Concatenated audio files

[0597] Output: Audio file sent to smart glasses

[0598] Specific operation: The server sends the audio file to the smart glasses via the network, and the smart glasses receive and play it.

[0599] Specific examples

[0600] If a visually impaired customer is looking for a television in a physical store, they can put on their smart glasses, approach a television product, and scan the QR code. The URL of this QR code is sent to a server, which then retrieves the HTML source of the webpage from the URL. The server analyzes the HTML source and extracts headings, paragraphs, lists, etc., then converts this into audio data and concatenates it into a playable audio file. Finally, this audio file is sent to the smart glasses, allowing the customer to listen to detailed information about the television.

[0601] Prompt Sentence Examples

[0602] Get the page content of "https: / / example.com / store-product-page", extract headings, paragraphs, lists, etc., and convert them into Japanese audio files. The generated audio files are intended to enable elderly and visually impaired people to easily listen to product information in physical stores.

[0603] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0604] The system of the present invention provides easy-to-use voice-based web page access for elderly people, visually impaired people, and others. By recognizing the user's emotions and reflecting them in the voice output, the system provides a more natural and easy-to-listen voice. A specific embodiment of this system is described below.

[0605] System Overview

[0606] When a user accesses the system and wants to listen to the contents of a web page aloud, they first use a device (PC, smartphone, tablet, etc.). The user opens a web browser, accesses the system, and enters the URL of the web page they want to play aloud. This information is sent to the server, which retrieves the web page at the specified URL, converts it into audio format, and provides it to the user. The system also uses an emotion engine to recognize the user's emotions and apply them to the audio output.

[0607] Program processing

[0608] Page retrieval and analysis

[0609] The server receives a request sent by a user and retrieves the web page at the specified URL. Typically, the HTTP request is used to retrieve the HTML source of the web page. This HTML source becomes the data used to analyze the content of the web page.

[0610] Text Extraction and Structure Recognition

[0611] The server analyzes the retrieved HTML source and extracts key text elements such as headings, paragraphs, links, and lists. It uses an HTML parser to generate a Document Object Model (DOM) and extracts the necessary text elements based on this. These extracted text elements are important for speech synthesis.

[0612] Preparing for speech synthesis

[0613] The server sends the extracted text elements to a speech synthesis API (such as Google Text-to-Speech API or Amazon Polly), which retrieves the corresponding audio data for each text element and combines them into a format that can be played back continuously. At this point, an emotion engine is used to simultaneously process information for analyzing the user's emotions.

[0614] Emotion Recognition and Voice Adjustment

[0615] The emotion engine receives the user's input voice, text, and non-verbal input information (facial expressions, gestures, etc.). This allows it to analyze the user's emotions and apply emotional expressions to the acquired voice data. For example, if the user is expressing a happy emotion, the voice data generated by the speech synthesis API will reflect joy and happiness.

[0616] Creation and provision of voice homepages

[0617] Once the final audio file is generated, the server stores it for delivery to the user. When the user accesses a specific URL, the server streams the audio file and delivers it to the user.

[0618] Specific examples

[0619] For example, if an elderly person named Tanaka finds a recipe on a web page but finds the text too small to read, this system can help. Tanaka accesses the system's page using a smartphone browser and enters a URL such as "https: / / example.com / recipes / spaghetti." The server retrieves the page and analyzes the HTML source to extract text elements such as "How to make spaghetti," "Step 1: Boil the pasta," and "Step 2: Make the sauce." These text elements are then sent to a speech synthesis API, which acquires the audio data and compiles it into a single audio file. At the same time, the emotion engine recognizes Tanaka's emotions from her input voice and facial expressions and adjusts the audio data based on those emotions. By playing this audio file on her smartphone, Tanaka can hear the recipe aloud, with the audio appropriately tailored to her current emotions.

[0620] In this way, web pages rich in text and images can be converted into audio format, and the user's emotions can be taken into consideration, making it easier for users who have difficulty using the Internet to access information. Furthermore, the more user-friendly audio output further promotes understanding and use of information.

[0621] The processing flow will be explained below.

[0622] Step 1:

[0623] The user accesses the system interface using a terminal, enters the URL of the web page they want to listen to, requests speech conversion, and clicks the submit button.

[0624] Step 2:

[0625] The device sends the entered URL information to the server, requesting the server to retrieve the web page via an HTTP request.

[0626] Step 3:

[0627] The server retrieves the specified web page based on the URL received from the user. The server sends an HTTP request to retrieve the HTML source code of the web page.

[0628] Step 4:

[0629] The HTML source code retrieved by the server is analyzed. An HTML parser is used to analyze the structure of the web page and generate a DOM (Document Object Model).

[0630] Step 5:

[0631] The server extracts the main text elements from the parsed DOM. It identifies and extracts elements such as headings (h1, h2, h3...), paragraphs (p), links (a), lists (ul, ol), etc.

[0632] Step 6:

[0633] The server converts the extracted text elements into a format for sending to the speech synthesis API. The text elements are formatted into a format that the API can process.

[0634] Step 7:

[0635] The server sends each text element to the speech synthesis API and receives the corresponding audio data. For example, the server sends the text "Step 1: Boil the pasta" to the API and receives the audio data in response.

[0636] Step 8:

[0637] The server activates the emotion engine based on the voice data obtained from the speech synthesis API. The emotion engine receives the user's input voice, text, and non-verbal input information (facial expressions, gestures, etc.).

[0638] Step 9:

[0639] The emotion engine analyzes the user's emotions and adjusts the voice data based on the analysis, for example, adjusting the tone and intonation of the voice to reflect emotions of joy or sadness.

[0640] Step 10:

[0641] The server combines the emotion-adjusted audio data into a continuous, playable format. Each adjusted audio data is concatenated in order to generate a single continuous audio file.

[0642] Step 11:

[0643] The server saves the generated audio file on the server. It generates an access URL for the audio file and prepares it so that it can be provided to the user.

[0644] Step 12:

[0645] The server returns the access URL for the audio file to the user, and sends a response to the device containing a link for the user to access the file.

[0646] Step 13:

[0647] The user accesses the provided URL on their device and plays the audio website. The user can then stream the audio file on their smartphone or PC and listen to the contents of the web page.

[0648] Through this series of processes, text-based web pages are converted into voice-based text, and the voice is adjusted to match the user's emotions, providing information in a more natural and easy-to-listen voice.

[0649] Example 2

[0650] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0651] For the elderly and visually impaired, reading text information on the Internet can be difficult, often resulting in visual stress. Furthermore, conventional text-to-speech systems struggle to provide natural-sounding speech that reflects the user's emotions. This can lead to a mechanical and unfamiliar experience. Furthermore, there are only a limited number of systems that can provide speech that reflects the user's emotions, limiting their range of use.

[0652] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0653] In this invention, the server includes means for accepting a URL specified by a user, means for retrieving the web page at the specified URL, means for analyzing the HTML source of the retrieved web page and extracting key text elements, means for transmitting the extracted text elements to a speech synthesis API and retrieving voice data, means for recognizing the user's emotions, means for applying emotional expressions to the retrieved voice data, means for combining the retrieved voice data into a continuously playable voice file, and means for providing the generated voice file to the user. This enables natural voice reading that takes the user's emotions into consideration. This allows a wide range of users, including the elderly and visually impaired, to receive information more easily.

[0654] "User" refers to a person who wishes to use the system to have a web page read aloud.

[0655] "URL" refers to a Uniform Resource Locator that indicates the address of a web page.

[0656] "Means" refers to a device or program provided to realize a specific function.

[0657] "Web page" refers to a document in HTML format that is published on the Internet.

[0658] "HTML source" refers to the code written in HyperText Markup Language that makes up the content of a Web page.

[0659] "Text elements" refers to textual information such as headings, paragraphs, links, and lists in HTML source.

[0660] "Speech synthesis API" refers to an application programming interface for converting text data into speech data.

[0661] "Audio data" refers to text information that has been converted into audio format.

[0662] "Means for recognizing emotions" refers to sensing technologies and algorithms for analyzing a user's emotional state.

[0663] "Means for applying emotional expressions" refers to technology that adjusts the intonation and tone of voice data based on the analyzed emotions.

[0664] "Audio file" refers to a file that contains audio data in a format that can be played continuously.

[0665] "Means of providing" refers to the technology and infrastructure required to transmit or stream the generated audio file to the user.

[0666] The system of the present invention provides voice-based web page access that is easy to use for elderly people, visually impaired people, etc. This system recognizes the user's emotions and reflects them in the voice output, providing a more natural and easy-to-listen voice.

[0667] Hardware and Software Configuration

[0668] server

[0669] The server has the following features:

[0670] Accepts a URL specified by the user.

[0671] The web page at the specified URL is retrieved via an HTTP request.

[0672] The HTML source of the retrieved web page is analyzed to extract the main text elements. For the specific analysis, an HTML parser such as BeautifulSoup (a Python library) is used.

[0673] The extracted text elements are sent to a speech synthesis API such as Google Text-to-Speech API or Amazon Polly to obtain audio data.

[0674] An emotion engine to recognize user emotions (e.g., Microsoft Cognitive Services' Emotion API).

[0675] Emotional expressions are applied to the audio data, and the adjusted audio data is compiled into an audio file that can be played continuously.

[0676] The generated audio file is saved for provision to the user, and a dedicated URL is generated.

[0677] Terminal

[0678] Users use devices such as PCs, smartphones, and tablets. These devices have a web browser installed. Users access the system through the browser and enter a URL.

[0679] Specific examples

[0680] For example, Mr. Tanaka, an elderly person, accesses the system using a web browser and enters a URL such as "https: / / example.com / recipes / spaghetti."

[0681] 1. The user (Mr. Tanaka) opens a browser on his / her PC, smartphone, or tablet and accesses the system's web page.

[0682] 2. Tanaka enters the URL of the web page she wants to play aloud (for example, "https: / / example.com / recipes / spaghetti").

[0683] 3. The server retrieves the web page at the specified URL through an HTTP request and parses the HTML source.

[0684] 4. The server extracts key text elements such as headings, paragraphs, links, and lists from the parsed HTML source.

[0685] 5. The extracted text elements are sent to a speech synthesis API, such as Google Text-to-Speech API or Amazon Polly, to obtain the corresponding audio data.

[0686] 6. The emotion engine analyzes Tanaka's emotions. For example, if Tanaka says "I'm looking forward to it" in the input voice, the voice is analyzed and the emotion is identified.

[0687] 7. Tanaka's emotions are reflected in the acquired voice data, and adjusted voice data is generated.

[0688] 8. The server compiles the generated audio data into a format that can be played continuously, and generates a single audio file.

[0689] 9. The final audio file is saved to the server and a unique URL (e.g., "https: / / example.com / voice / spaghetti") is generated.

[0690] 10. By accessing the dedicated URL, Tanaka can stream the audio file and hear the contents of the web page in audio, which appropriately reflects Tanaka's emotions.

[0691] Prompt Sentence Examples

[0692] Below are some example prompts to be input to the generative AI model:

[0693] "Play a web page specified by the user's URL (e.g., https: / / example.com / recipes / spaghetti) aloud, providing audio that reflects the user's emotions."

[0694] This system makes it possible to provide speech synthesis that takes into consideration the emotions of users, including the elderly and visually impaired, even on web pages with a wealth of text and image information. This makes it easier for users who have difficulty using the Internet to access information, and promotes understanding of the information through friendly speech output.

[0695] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0696] Step 1:

[0697] The user uses a terminal to open a web browser. They access a web page on the system and enter the URL of the web page they want to play aloud (e.g., "https: / / example.com / recipes / spaghetti"). The entered URL is sent to the server as a request. Input: The URL entered by the user. Output: The URL request to the server.

[0698] Step 2:

[0699] The terminal sends an HTTP request containing the URL information entered by the user to the server. The server receives the request and retrieves the web page at the specified URL. In fact, it downloads the HTML source of the web page using an HTTP GET request. Input: URL request. Output: Retrieved HTML source.

[0700] Step 3:

[0701] The server parses the retrieved HTML source. It uses an HTML parser (e.g., BeautifulSoup) to convert the HTML source into a Document Object Model (DOM) and extracts major text elements such as headings, paragraphs, links, and lists. Input: HTML source. Output: Extracted text elements.

[0702] Step 4:

[0703] The server sends the extracted text elements to a speech synthesis API (for example, Google Text-to-Speech API or Amazon Polly) to obtain audio data. Specifically, each text element is sent individually to the API to obtain audio data for each. Input: Text elements. Output: Audio data.

[0704] Step 5:

[0705] The server uses an emotion engine to analyze the user's emotions. This emotion information is generated based on the user's input voice, text, and non-verbal information (facial expressions, gestures, etc.). Input: Information about the user's emotions. Output: Analyzed emotion data.

[0706] Step 6:

[0707] The server applies emotional expressions to the acquired voice data. Based on the analyzed emotional data, it adjusts the intonation and tone of the voice data. This generates a voice that reflects the user's emotions. Input: Voice data, emotional data. Output: Adjusted voice data.

[0708] Step 7:

[0709] The server combines the adjusted audio data into a single audio file. It concatenates the audio data for multiple text elements into a single file to create a continuously playable audio file. Input: Adjusted audio data. Output: Audio file.

[0710] Step 8:

[0711] The server saves the generated audio file and generates a URL that users can access. Users can stream the audio file by accessing the URL. Input: Audio file. Output: Accessible URL.

[0712] This specific process allows users to listen to web pages in a natural voice that reflects their emotions, providing an environment where even the elderly and visually impaired can easily access information.

[0713] (Application example 2)

[0714] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0715] Users with visual or physical disabilities often have difficulty accessing and understanding the content of web pages. Furthermore, conventional text-to-speech systems do not provide natural-sounding speech output that reflects the user's emotions, making it difficult for them to understand the audio information. This makes it particularly difficult for users with visual or physical disabilities to use online shopping sites.

[0716] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0717] In this invention, the server includes means for accepting a URL specified by a user, means for retrieving the web page of the specified URL, means for analyzing the HTML source of the retrieved web page and extracting key text elements, means for transmitting the extracted text elements to a speech synthesis API and retrieving voice data, means for combining the retrieved voice data into a continuously playable voice file, means for recognizing the user's emotion and adjusting the voice output based on the emotion, and means for providing the generated voice file to the user, thereby enabling users with visual or physical disabilities to understand the content of the web page through natural voice that corresponds to their emotion.

[0718] The "specified URL" is the address of the web page that the user wants to access.

[0719] "Getting a web page" means downloading the HTML source from the specified URL.

[0720] "HTML source" is data written in a markup language that describes the structure and content of a web page.

[0721] A "text element" is an element in the HTML source that contains text information, such as a heading, paragraph, link, or list.

[0722] A "speech synthesis API" is an application program interface used to convert text data into speech data.

[0723] "Audio data" means playable sound data generated using a speech synthesis API.

[0724] An "audio file" is audio data combined into a format that can be played back continuously.

[0725] "Emotion recognition" refers to analyzing a user's current emotional state from their voice or other input.

[0726] "Adjusting the audio output" refers to changing the tone and rate of the generated audio based on the perceived user emotion.

[0727] This invention is a system that enables users with visual or physical disabilities to access web pages by voice. The system retrieves a specified URL, analyzes its HTML source to extract important text elements, and generates voice data using a speech synthesis API. It also recognizes the user's emotions and adjusts the generated voice output to provide it to the user.

[0728] Overall system configuration

[0729] The system mainly consists of the following steps:

[0730] 1. Accepts a URL specified by the user.

[0731] 2. The server retrieves the web page at the specified URL.

[0732] 3. Analyze the obtained HTML source and extract the main text elements.

[0733] 4. The extracted text elements are sent to the speech synthesis API to obtain the speech data.

[0734] 5. Combine the audio data into a continuously playable audio file.

[0735] 6. Recognize user emotions and adjust voice output based on those emotions.

[0736] 7. Provide the final audio file to the user.

[0737] Hardware and software used

[0738] Hardware: The system can be accessed from devices such as PCs, smartphones, and tablets.

[0739] Software: Use "Google Text-to-Speech API" or "Amazon Polly" for speech synthesis. Use an HTML parser such as "BeautifulSoup" for HTML analysis. Use the "emotion_recognition" module for emotion recognition.

[0740] How it works

[0741] 1. URL acceptance

[0742] Users access the system from a device such as a PC or smartphone and enter the specified URL, which is then sent to the server.

[0743] 2. Fetching a Web Page

[0744] The server sends an HTTP request to the specified URL and retrieves the HTML source of the page, using the "requests" library.

[0745] 3. HTML source analysis

[0746] The server parses the HTML source and extracts the main text elements (headings, paragraphs, links, lists, etc.). This analysis is done using "BeautifulSoup."

[0747] 4. Generating Audio Data

[0748] The extracted text elements are sent to a speech synthesis API such as the Google Text-to-Speech API to generate audio data.

[0749] 5. Generate audio files

[0750] The acquired audio data is combined into a format that can be played continuously, and an audio file is generated.

[0751] 6. Emotion recognition and voice output adjustment

[0752] It analyzes emotions from the user's voice input and other non-verbal information and adjusts the generated voice data accordingly, providing a more natural and understandable voice output.

[0753] 7. Providing audio files

[0754] The final generated audio file is provided in a user-accessible format.

[0755] Specific examples

[0756] For example, if a visually impaired user wants to search for a new product on an online shopping site, they can use this system to easily obtain information. When the user enters the URL "https: / / example.com / product / smartphone" into the system, the system analyzes the page and reads out information such as "latest smartphone," "price: 50,000 yen," and "equipped with a high-quality camera." At the same time, it analyzes the user's voice input to recognize emotions and provides information in a tone that corresponds to that emotion.

[0757] Prompt Sentence Examples

[0758] Example prompt:

[0759] A user wants to check product details on an online shopping website. The URL is "https: / / example.com / product / smartphone." The user utters "Tell me about my new smartphone." The system retrieves the webpage, analyzes key text elements, and provides an emotionally-aware speech readout.

[0760] Expected results:

[0761] Information such as "This is the latest smartphone. It costs 50,000 yen and has a high-quality camera" is read out to the user in a tone that reflects their emotions.

[0762] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0763] Step 1:

[0764] Enter the URL specified by the user.

[0765] A user accesses the system using a device such as a PC or smartphone and enters a specified URL. The user's input information is sent to the server. In this step, the input is the "specified URL" and the output is sending the URL to the server. Specifically, the user enters the URL into the input form from the device's browser and clicks the submit button.

[0766] Step 2:

[0767] The server retrieves the web page at the specified URL.

[0768] The server sends an HTTP request to retrieve the HTML source of the web page based on the URL received from the user. The input is the "specified URL" and the output is the "HTML source of the web page." This process uses the "requests" library, specifically requests.get() to send the HTTP request.

[0769] Step 3:

[0770] The obtained HTML source is analyzed and the main text elements are extracted.

[0771] The server parses the retrieved HTML source using the "BeautifulSoup" library and extracts key text elements such as headings, paragraphs, links, and lists. The input is the "HTML source of the web page" and the output is the "extracted text elements." Specifically, it parses the HTML source using BeautifulSoup and extracts the required text elements using the find_all() method.

[0772] Step 4:

[0773] The extracted text elements are sent to a speech synthesis API to obtain speech data.

[0774] The server sends the extracted text elements to a speech synthesis API such as "Google Text-to-Speech API" or "Amazon Polly" to obtain the audio data. The input is the extracted text elements, and the output is audio data. Specifically, the text data is sent to the speech synthesis API endpoint, and audio data is received as a response.

[0775] Step 5:

[0776] The acquired audio data is combined into a continuously playable audio file.

[0777] The server combines the acquired audio data into a format that can be played continuously to generate an audio file. The input is "audio data" and the output is an "audio file." Specifically, the audio data is combined in order and saved as a single audio file.

[0778] Step 6:

[0779] To recognize a user's emotions and adjust audio output based on the emotions.

[0780] The server analyzes the user's voice input and non-verbal information to recognize the user's emotions. It uses the "emotion_recognition" module to perform emotion analysis and adjusts the tone and speed of the voice data based on the results. The input is an audio file and emotional information, and the output is an adjusted audio file. Specifically, it applies an emotion recognition algorithm and changes the parameters of the speech synthesis API according to the user's emotions.

[0781] Step 7:

[0782] The final audio file is provided to the user.

[0783] The server then provides the final audio file to the user in streaming format. The input is the "adjusted audio file" and the output is the "audio provided to the user." Specifically, the audio file is sent as an HTTP response and is played back in real time on the device.

[0784] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0785] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0786] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0787] [Third embodiment]

[0788] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0789] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0790] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0791] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0792] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0793] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0794] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0795] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0796] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0797] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0798] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0799] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0800] The system of the present invention provides voice-based web page access that is easy to use for the elderly, the visually impaired, etc. A specific embodiment of this system will be described below.

[0801] System Overview

[0802] When a user accesses the system and wants to listen to the contents of a web page by audio, they first use a device (PC, smartphone, tablet, etc.). The user opens a web browser, accesses the system, and enters the URL of the web page they want to play by audio. This information is sent to the server, which retrieves the web page at the specified URL, converts it into audio format, and provides it to the user.

[0803] Program processing

[0804] Page retrieval and analysis

[0805] The server receives a request sent by a user and retrieves the web page at the specified URL. Typically, the HTTP request is used to retrieve the HTML source of the web page. This HTML source becomes the data used to analyze the content of the web page.

[0806] Text Extraction and Structure Recognition

[0807] The server analyzes the retrieved HTML source and extracts key text elements such as headings, paragraphs, links, and lists. It uses an HTML parser to generate a Document Object Model (DOM) and extracts the necessary text elements based on this. These extracted text elements are important for speech synthesis.

[0808] Preparing for speech synthesis

[0809] The server sends the extracted text elements to a speech synthesis API (e.g., Google Text-to-Speech API or Amazon Polly), which retrieves the corresponding audio data for each text element and combines them into a format that can be played back continuously.

[0810] Creation and provision of voice homepages

[0811] Once the final audio file is generated, the server stores it for delivery to the user. When the user accesses a specific URL, the server streams the audio file and delivers it to the user.

[0812] Specific examples

[0813] If elderly user Tanaka finds a cooking recipe on a web page but finds the text too small to read, this system can help. Tanaka accesses the system's page using his smartphone browser and enters a URL such as "https: / / example.com / recipes / spaghetti." The server retrieves the page at this URL, analyzes the HTML source, and extracts text elements such as "How to make spaghetti," "Step 1: Boil the pasta," and "Step 2: Make the sauce." These text elements are sent to a speech synthesis API, which obtains the audio data and then compiles it into a single audio file. By playing this audio file on his smartphone, Tanaka can listen to the recipe aloud.

[0814] In this way, by converting web pages rich in text and image information into audio format, even users who have difficulty using the Internet can easily access information, contributing to bridging the digital divide.

[0815] The processing flow will be explained below.

[0816] Step 1:

[0817] The user accesses the system interface using a terminal, enters the URL of the web page they want to listen to, requests speech conversion, and clicks the submit button.

[0818] Step 2:

[0819] The device sends the entered URL information to the server, requesting the server to retrieve the web page via an HTTP request.

[0820] Step 3:

[0821] The server retrieves the specified web page based on the URL received from the user. The server sends an HTTP request to retrieve the HTML source code of the web page.

[0822] Step 4:

[0823] The HTML source code retrieved by the server is analyzed. An HTML parser is used to analyze the structure of the web page and generate a DOM (Document Object Model).

[0824] Step 5:

[0825] The server extracts the main text elements from the parsed DOM. It identifies and extracts elements such as headings (h1, h2, h3...), paragraphs (p), links (a), lists (ul, ol), etc.

[0826] Step 6:

[0827] The server converts the extracted text elements into a format for sending to the speech synthesis API. The text elements are formatted into a format that the API can process.

[0828] Step 7:

[0829] The server sends each text element to the speech synthesis API and receives the corresponding audio data. For example, the server sends the text "Step 1: Boil the pasta" to the API and receives the audio data in response.

[0830] Step 8:

[0831] The server combines each acquired audio data into a format that can be played back continuously. The acquired individual audio data is concatenated in order to generate a single continuous audio file.

[0832] Step 9:

[0833] The server saves the generated audio file on the server. It generates an access URL for the audio file and prepares it so that it can be provided to the user.

[0834] Step 10:

[0835] The server returns the access URL for the audio file to the user, and sends a response to the device containing a link for the user to access the file.

[0836] Step 11:

[0837] The user accesses the provided URL on their device and plays the audio website. The user can then stream the audio file on their smartphone or PC and listen to the contents of the web page.

[0838] Through this series of processes, text-based web pages are converted into audio-based pages and presented to the user in an easy-to-use format.

[0839] Example 1

[0840] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0841] There is a need to make web page information easily accessible to the elderly and visually impaired. However, many current web pages rely on visual information provision, which causes access barriers for these users. It is particularly difficult to obtain information when the text is small and difficult to read, or when the layout is visually complex. Therefore, a system that provides web page content using audio is needed.

[0842] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0843] In this invention, the server includes means for accepting a URL specified by a user, means for retrieving the web page of the specified URL, means for analyzing the markup language source of the retrieved web page and extracting key text elements, means for transmitting the extracted text elements to a speech synthesis service and retrieving audio data, means for combining the retrieved audio data into an audio file that can be played continuously, and means for providing the generated audio file to the user. This allows even the elderly and visually impaired to hear the contents of the web page aloud, making it easier to access information.

[0844] "User" refers to an individual or organization that uses the System.

[0845] "URL" stands for Uniform Resource Locator and is an address that specifies the location of a web resource.

[0846] A "web page" is a document in HTML format that is published on the Internet and refers to the content that is displayed using a browser.

[0847] An "HTTP request" is a communication protocol that allows a client (user's browser) to request a specific resource from a server.

[0848] "HTML source" refers to text data that represents the structure and content of a web page written in a markup language.

[0849] An "HTML parser" is a software tool for analyzing HTML source and generating a DOM (Document Object Model).

[0850] "DOM" is an abbreviation for Document Object Model, and is a model that represents the structure of a web page in a tree format.

[0851] "Text elements" refer to text data such as headings, paragraphs, links, and lists that make up the content of a web page.

[0852] "Speech synthesis service" refers to APIs and software for converting input text data into voice data.

[0853] "Audio Data" refers to digital data in audio format generated by a speech synthesis service.

[0854] An "audio file" is a digital file for storing audio data, saved in a format that allows for continuous playback.

[0855] "Streaming playback" is a method of playing audio files in real time, where playback begins before the file is completely downloaded.

[0856] The system of the present invention allows elderly people, visually impaired people, etc. to use voice to access the contents of web pages. A specific embodiment of this system will be described below.

[0857] A user accesses the system using a device (PC, smartphone, tablet, etc.). First, the user opens the device's web browser and accesses the system's web page. Next, the user enters the URL of the web page they want to play by voice and presses the "Submit" button. This information is sent to the server.

[0858] The server receives a request sent by a user and retrieves the web page at the specified URL using an HTTP request. It then analyzes the retrieved HTML source and generates a Document Object Model (DOM) using an HTML parser (e.g., BeautifulSoup or jsoup). The server then extracts key text elements such as headings, paragraphs, links, and lists from the generated DOM.

[0859] The server then sends the extracted text elements to a speech synthesis service (e.g., Google Text-to-Speech API or Amazon Polly). The service generates the corresponding audio data and returns it to the server. The server then compiles the audio data into a continuous, playable audio file and saves it as a single audio file. Finally, the server provides the saved audio file at a specific URL, which users can access to stream the audio file.

[0860] As a concrete example, consider the case where an elderly person named Tanaka browses a recipe website on his smartphone. Tanaka accesses the system's page using his smartphone's browser, enters the URL "https: / / example.com / recipes / spaghetti" and submits it. The server retrieves the web page at that URL, analyzes the HTML source, and extracts text elements such as headings and paragraphs. These text elements are then sent to a speech synthesis service, which obtains the audio data and compiles it into a single audio file. Tanaka can then stream this audio file and listen to the recipe aloud.

[0861] Examples of prompts that give specific instructions to a generative AI model include:

[0862] 1. "Design a system that uses speech synthesis technology to provide audible content from web pages for the elderly."

[0863] 2. "Explain how to parse the contents of a web page retrieved from a specific URL and convert only the key text elements into audio data."

[0864] 3. "What are the steps in the process of converting text extracted from an HTML source into audio format using the Google Text-to-Speech API?"

[0865] As described above, this system allows elderly people and people with visual impairments to easily access web page content by voice. This system lowers the barriers to using the Internet and makes it easier to access information, thereby contributing to bridging the digital divide.

[0866] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0867] Step 1:

[0868] A user accesses the system using a terminal and enters the URL of the web page they want to play aloud. As input, the user opens a web browser, enters the URL in the designated input field, and presses the submit button. As output, the URL is sent to the server.

[0869] Specific behavior:

[0870] The user opens a browser on their PC or smartphone.

[0871] Enter "https: / / example.com" in the URL input field.

[0872] Click the "Send" button.

[0873] Step 2:

[0874] The server receives a URL request sent by a user and retrieves the web page at the specified URL. As input, it receives the URL sent by the user. As output, it retrieves the HTML source of the web page at the specified URL.

[0875] Specific behavior:

[0876] The server sends an HTTP GET request based on the URL received from the user.

[0877] The server receives the HTML source as a response.

[0878] Step 3:

[0879] The server parses the HTML source and generates a DOM (Document Object Model). It uses the HTML source as input and generates a DOM tree as output.

[0880] Specific behavior:

[0881] The server generates a DOM from the HTML source using an HTML parser (e.g., BeautifulSoup).

[0882] Step 4:

[0883] The server extracts the main text elements (headings, paragraphs, links, lists, etc.) from the generated DOM. It uses the generated DOM as input. It gets a list of extracted text elements as output.

[0884] Specific behavior:

[0885] The server extracts elements such as h1 tags, p tags, a tags, and ul tags from the DOM.

[0886] The extracted elements are compiled into a text list.

[0887] Step 5:

[0888] The server sends the extracted text elements to a speech synthesis service to obtain speech data. As input, it uses the extracted text elements. As output, it obtains the corresponding speech data.

[0889] Specific behavior:

[0890] The server sends the text data to a speech synthesis service (e.g., Google Text-to-Speech API).

[0891] The server receives the audio data from the service.

[0892] Step 6:

[0893] The server compiles the acquired audio data into a continuously playable audio file and saves the audio file. As input, it uses the acquired audio data. As output, it generates and saves a continuously playable audio file.

[0894] Specific behavior:

[0895] The server combines the audio data corresponding to each text element into a single audio file.

[0896] Save the merged audio file (e.g. audio / output.mp3).

[0897] Step 7:

[0898] The server provides the stored audio file at a specific URL so that it can be streamed when accessed by the user. As input, it uses the stored audio file. As output, it generates and provides a streaming playback URL that the user can access.

[0899] Specific behavior:

[0900] The server generates a URL to access the audio file (e.g., https: / / example.com / audio / output.mp3).

[0901] When a user accesses the URL, the server streams the audio file.

[0902] By following the above processing steps in order, this system is able to provide the contents of web pages to elderly people and people with visual impairments by voice.

[0903] (Application example 1)

[0904] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0905] The purpose of this invention is to enable elderly people and people with visual impairments to easily obtain product information in physical stores. In particular, the objective is to improve information accessibility by providing a means for visually impaired people and elderly people to listen to product descriptions and guidance by audio. Furthermore, conventional methods mainly use text information, which creates a visual barrier, and there is a need to eliminate this.

[0906] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0907] In this invention, the server includes means for accepting a URL specified by a user, means for retrieving the web page of the specified URL, means for analyzing the HTML source of the retrieved web page and extracting key text elements, means for transmitting the extracted text elements to a speech synthesis API to retrieve voice data, means for combining the retrieved voice data into a continuously playable voice file, means for providing the generated voice file to the user, and application means installed on smart glasses for providing product information in a physical store in voice format, thereby enabling visually impaired people and the elderly to easily obtain product information in a physical store by voice.

[0908] "Means for accepting a URL specified by a user" refers to a means for accepting a URL entered by a user to obtain information on a specific web page, and for the entire system to start processing based on that URL.

[0909] "Means for obtaining a web page at a specified URL" refers to means for accessing a URL entered by a user and obtaining the HTML source of the corresponding web page.

[0910] "Means for analyzing the HTML source of the acquired web page and extracting major text elements" refers to means for analyzing the acquired HTML source and extracting major text elements such as headings, paragraphs, links, and lists from it.

[0911] "Means for sending the extracted text elements to a speech synthesis API and obtaining speech data" refers to means for sending the extracted text elements to a speech synthesis API to convert them into speech, and obtaining the resulting speech data.

[0912] The "means for combining acquired audio data into a continuously reproducible audio file" refers to a means for combining acquired audio data into one continuous audio file.

[0913] The "means for providing the generated audio file to the user" refers to a means for providing the generated audio file in a form that the user can access.

[0914] "Application means installed on smart glasses for providing product information in audio format in physical stores" refers to a means for installing and executing an application on smart glasses for providing product information and guidance in audio format in physical stores.

[0915] The present invention provides a system that enables visually impaired people and elderly people to easily obtain product information by voice in a brick-and-mortar store, and provides voice guidance using smart glasses. The following describes an embodiment of the present invention.

[0916] System Configuration

[0917] The system includes a server, smart glasses, and a network communication means.

[0918] server

[0919] The server has the following functions:

[0920] 1. A means of accepting a URL specified by the user

[0921] The server receives the URL of the product information page sent from the smart glasses.

[0922] 2. How to get a web page from a specified URL

[0923] The server sends an HTTP request to the specified URL and retrieves the HTML source of the corresponding web page.

[0924] 3. A method for analyzing the HTML source of a retrieved web page and extracting key text elements

[0925] An HTML parser (e.g., BeautifulSoup) is used to parse the HTML source of the web page and extract key text elements such as headings, paragraphs, links, and lists.

[0926] 4. A method for sending the extracted text elements to a speech synthesis API and obtaining the speech data

[0927] The extracted text elements are sent to the Google Text-to-Speech API to obtain the audio data.

[0928] 5. A means of combining the captured audio data into a continuously playable audio file

[0929] The individual audio data is combined into one continuous audio file.

[0930] 6. How to provide the generated audio file to the user

[0931] The completed audio file is sent to the smart glasses so that the user can listen to it.

[0932] Smart Glasses

[0933] The smart glasses have the following features:

[0934] 1. How to enter a specified URL

[0935] The URL for product information is obtained by reading the product tag or QR code.

[0936] 2. How to send a URL to the server

[0937] Send the obtained URL to the server.

[0938] 3. A means to receive and play audio files from the server

[0939] The audio file received from the server is played to the user.

[0940] Hardware and software used

[0941] Smart glasses (e.g. Google Glass)

[0942] Server (e.g. Google Cloud Platform)

[0943] BeautifulSoup for HTML parsing

[0944] Google Text-to-Speech API for speech synthesis

[0945] Specific examples

[0946] For example, if a visually impaired customer is looking for a television in a physical store, they can use this system to obtain product information by voice. When the customer approaches a television wearing smart glasses and scans the QR code, the URL corresponding to that product is sent to the server. The server then analyzes the webpage based on the URL and extracts key text information. This information is converted into audio data using a speech synthesis API and sent back to the smart glasses. The smart glasses then play the received audio file, allowing the customer to listen to the television's product information aloud.

[0947] Prompt Sentence Examples

[0948] Get the page content of "https: / / example.com / store-product-page", extract headings, paragraphs, lists, etc., and convert them into Japanese audio files. The generated audio files are intended to enable elderly and visually impaired people to easily listen to product information in physical stores.

[0949] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0950] Step 1:

[0951] When a user searches for a product in a physical store, they use the smart glasses to read the QR code on the product, which retrieves the URL of the product information page, and the smart glasses then send the URL to the server.

[0952] Input: URL of product information read from QR code

[0953] Output: URL sent to the server

[0954] Specific operation: The smart glasses' camera is used to scan the QR code and the read URL is sent to the server via the network.

[0955] Step 2:

[0956] Based on the URL received from the smart glasses, the server sends an HTTP request to retrieve the HTML source of the specified web page.

[0957] Input: URL sent from smart glasses

[0958] Output: HTML source of the retrieved web page

[0959] What happens: The server makes an HTTP GET request to the URL to retrieve the HTML source of the web page.

[0960] Step 3:

[0961] The server parses the retrieved HTML source and extracts key text elements such as headings, paragraphs, links, and lists.

[0962] Input: HTML source of the web page

[0963] Output: Extracted main text elements

[0964] Specific behavior: Analyzes HTML source using an HTML parser such as BeautifulSoup and extracts the main text elements. Extracts text data for each element such as headings, paragraphs, and lists.

[0965] Step 4:

[0966] The server sends the extracted text elements to the Google Text-to-Speech API, which converts them into audio data.

[0967] Input: Extracted main text elements

[0968] Output: Audio data

[0969] Specific operation: Sends text data to the Google Text-to-Speech API and receives audio data as a response. Audio data is generated for each text element.

[0970] Step 5:

[0971] The server combines the acquired audio data into a continuously playable audio file.

[0972] Input: Audio data

[0973] Output: A combined audio file that can be played continuously

[0974] What it does: Concatenates multiple audio data files into a single audio file, allowing users to listen to information seamlessly.

[0975] Step 6:

[0976] The server sends the generated audio file to the smart glasses and provides it to the user.

[0977] Input: Concatenated audio files

[0978] Output: Audio file sent to smart glasses

[0979] Specific operation: The server sends the audio file to the smart glasses via the network, and the smart glasses receive and play it.

[0980] Specific examples

[0981] If a visually impaired customer is looking for a television in a physical store, they can put on their smart glasses, approach a television product, and scan the QR code. The URL of this QR code is sent to a server, which then retrieves the HTML source of the webpage from the URL. The server analyzes the HTML source and extracts headings, paragraphs, lists, etc., then converts this into audio data and concatenates it into a playable audio file. Finally, this audio file is sent to the smart glasses, allowing the customer to listen to detailed information about the television.

[0982] Prompt Sentence Examples

[0983] Get the page content of "https: / / example.com / store-product-page", extract headings, paragraphs, lists, etc., and convert them into Japanese audio files. The generated audio files are intended to enable elderly and visually impaired people to easily listen to product information in physical stores.

[0984] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0985] The system of the present invention provides easy-to-use voice-based web page access for elderly people, visually impaired people, and others. By recognizing the user's emotions and reflecting them in the voice output, the system provides a more natural and easy-to-listen voice. A specific embodiment of this system is described below.

[0986] System Overview

[0987] When a user accesses the system and wants to listen to the contents of a web page aloud, they first use a device (PC, smartphone, tablet, etc.). The user opens a web browser, accesses the system, and enters the URL of the web page they want to play aloud. This information is sent to the server, which retrieves the web page at the specified URL, converts it into audio format, and provides it to the user. The system also uses an emotion engine to recognize the user's emotions and apply them to the audio output.

[0988] Program processing

[0989] Page retrieval and analysis

[0990] The server receives a request sent by a user and retrieves the web page at the specified URL. Typically, the HTTP request is used to retrieve the HTML source of the web page. This HTML source becomes the data used to analyze the content of the web page.

[0991] Text Extraction and Structure Recognition

[0992] The server analyzes the retrieved HTML source and extracts key text elements such as headings, paragraphs, links, and lists. It uses an HTML parser to generate a Document Object Model (DOM) and extracts the necessary text elements based on this. These extracted text elements are important for speech synthesis.

[0993] Preparing for speech synthesis

[0994] The server sends the extracted text elements to a speech synthesis API (such as Google Text-to-Speech API or Amazon Polly), which retrieves the corresponding audio data for each text element and combines them into a format that can be played back continuously. At this point, an emotion engine is used to simultaneously process information for analyzing the user's emotions.

[0995] Emotion Recognition and Voice Adjustment

[0996] The emotion engine receives the user's input voice, text, and non-verbal input information (facial expressions, gestures, etc.). This allows it to analyze the user's emotions and apply emotional expressions to the acquired voice data. For example, if the user is expressing a happy emotion, the voice data generated by the speech synthesis API will reflect joy and happiness.

[0997] Creation and provision of voice homepages

[0998] Once the final audio file is generated, the server stores it for delivery to the user. When the user accesses a specific URL, the server streams the audio file and delivers it to the user.

[0999] Specific examples

[1000] For example, if an elderly person named Tanaka finds a recipe on a web page but finds the text too small to read, this system can help. Tanaka accesses the system's page using a smartphone browser and enters a URL such as "https: / / example.com / recipes / spaghetti." The server retrieves the page and analyzes the HTML source to extract text elements such as "How to make spaghetti," "Step 1: Boil the pasta," and "Step 2: Make the sauce." These text elements are then sent to a speech synthesis API, which acquires the audio data and compiles it into a single audio file. At the same time, the emotion engine recognizes Tanaka's emotions from her input voice and facial expressions and adjusts the audio data based on those emotions. By playing this audio file on her smartphone, Tanaka can hear the recipe aloud, with the audio appropriately tailored to her current emotions.

[1001] In this way, web pages rich in text and images can be converted into audio format, and the user's emotions can be taken into consideration, making it easier for users who have difficulty using the Internet to access information. Furthermore, the more user-friendly audio output further promotes understanding and use of information.

[1002] The processing flow will be explained below.

[1003] Step 1:

[1004] The user accesses the system interface using a terminal, enters the URL of the web page they want to listen to, requests speech conversion, and clicks the submit button.

[1005] Step 2:

[1006] The device sends the entered URL information to the server, requesting the server to retrieve the web page via an HTTP request.

[1007] Step 3:

[1008] The server retrieves the specified web page based on the URL received from the user. The server sends an HTTP request to retrieve the HTML source code of the web page.

[1009] Step 4:

[1010] The HTML source code retrieved by the server is analyzed. An HTML parser is used to analyze the structure of the web page and generate a DOM (Document Object Model).

[1011] Step 5:

[1012] The server extracts the main text elements from the parsed DOM. It identifies and extracts elements such as headings (h1, h2, h3...), paragraphs (p), links (a), lists (ul, ol), etc.

[1013] Step 6:

[1014] The server converts the extracted text elements into a format for sending to the speech synthesis API. The text elements are formatted into a format that the API can process.

[1015] Step 7:

[1016] The server sends each text element to the speech synthesis API and receives the corresponding audio data. For example, the server sends the text "Step 1: Boil the pasta" to the API and receives the audio data in response.

[1017] Step 8:

[1018] The server activates the emotion engine based on the voice data obtained from the speech synthesis API. The emotion engine receives the user's input voice, text, and non-verbal input information (facial expressions, gestures, etc.).

[1019] Step 9:

[1020] The emotion engine analyzes the user's emotions and adjusts the voice data based on the analysis, for example, adjusting the tone and intonation of the voice to reflect emotions of joy or sadness.

[1021] Step 10:

[1022] The server combines the emotion-adjusted audio data into a continuous, playable format. Each adjusted audio data is concatenated in order to generate a single continuous audio file.

[1023] Step 11:

[1024] The server saves the generated audio file on the server. It generates an access URL for the audio file and prepares it so that it can be provided to the user.

[1025] Step 12:

[1026] The server returns the access URL for the audio file to the user, and sends a response to the device containing a link for the user to access the file.

[1027] Step 13:

[1028] The user accesses the provided URL on their device and plays the audio website. The user can then stream the audio file on their smartphone or PC and listen to the contents of the web page.

[1029] Through this series of processes, text-based web pages are converted into voice-based text, and the voice is adjusted to match the user's emotions, providing information in a more natural and easy-to-listen voice.

[1030] Example 2

[1031] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1032] For the elderly and visually impaired, reading text information on the Internet can be difficult, often resulting in visual stress. Furthermore, conventional text-to-speech systems struggle to provide natural-sounding speech that reflects the user's emotions. This can lead to a mechanical and unfamiliar experience. Furthermore, there are only a limited number of systems that can provide speech that reflects the user's emotions, limiting their range of use.

[1033] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1034] In this invention, the server includes means for accepting a URL specified by a user, means for retrieving the web page at the specified URL, means for analyzing the HTML source of the retrieved web page and extracting key text elements, means for transmitting the extracted text elements to a speech synthesis API and retrieving voice data, means for recognizing the user's emotions, means for applying emotional expressions to the retrieved voice data, means for combining the retrieved voice data into a continuously playable voice file, and means for providing the generated voice file to the user. This enables natural voice reading that takes the user's emotions into consideration. This allows a wide range of users, including the elderly and visually impaired, to receive information more easily.

[1035] "User" refers to a person who wishes to use the system to have a web page read aloud.

[1036] "URL" refers to a Uniform Resource Locator that indicates the address of a web page.

[1037] "Means" refers to a device or program provided to realize a specific function.

[1038] "Web page" refers to a document in HTML format that is published on the Internet.

[1039] "HTML source" refers to the code written in HyperText Markup Language that makes up the content of a Web page.

[1040] "Text elements" refers to textual information such as headings, paragraphs, links, and lists in HTML source.

[1041] "Speech synthesis API" refers to an application programming interface for converting text data into speech data.

[1042] "Audio data" refers to text information that has been converted into audio format.

[1043] "Means for recognizing emotions" refers to sensing technologies and algorithms for analyzing a user's emotional state.

[1044] "Means for applying emotional expressions" refers to technology that adjusts the intonation and tone of voice data based on the analyzed emotions.

[1045] "Audio file" refers to a file that contains audio data in a format that can be played continuously.

[1046] "Means of providing" refers to the technology and infrastructure required to transmit or stream the generated audio file to the user.

[1047] The system of the present invention provides voice-based web page access that is easy to use for elderly people, visually impaired people, etc. This system recognizes the user's emotions and reflects them in the voice output, providing a more natural and easy-to-listen voice.

[1048] Hardware and Software Configuration

[1049] server

[1050] The server has the following features:

[1051] Accepts a URL specified by the user.

[1052] The web page at the specified URL is retrieved via an HTTP request.

[1053] The HTML source of the retrieved web page is analyzed to extract the main text elements. For the specific analysis, an HTML parser such as BeautifulSoup (a Python library) is used.

[1054] The extracted text elements are sent to a speech synthesis API such as Google Text-to-Speech API or Amazon Polly to obtain audio data.

[1055] An emotion engine to recognize user emotions (e.g., Microsoft Cognitive Services' Emotion API).

[1056] Emotional expressions are applied to the audio data, and the adjusted audio data is compiled into an audio file that can be played continuously.

[1057] The generated audio file is saved for provision to the user, and a dedicated URL is generated.

[1058] Terminal

[1059] Users use devices such as PCs, smartphones, and tablets. These devices have a web browser installed. Users access the system through the browser and enter a URL.

[1060] Specific examples

[1061] For example, Mr. Tanaka, an elderly person, accesses the system using a web browser and enters a URL such as "https: / / example.com / recipes / spaghetti."

[1062] 1. The user (Mr. Tanaka) opens a browser on his / her PC, smartphone, or tablet and accesses the system's web page.

[1063] 2. Tanaka enters the URL of the web page she wants to play aloud (for example, "https: / / example.com / recipes / spaghetti").

[1064] 3. The server retrieves the web page at the specified URL through an HTTP request and parses the HTML source.

[1065] 4. The server extracts key text elements such as headings, paragraphs, links, and lists from the parsed HTML source.

[1066] 5. The extracted text elements are sent to a speech synthesis API, such as Google Text-to-Speech API or Amazon Polly, to obtain the corresponding audio data.

[1067] 6. The emotion engine analyzes Tanaka's emotions. For example, if Tanaka says "I'm looking forward to it" in the input voice, the voice is analyzed and the emotion is identified.

[1068] 7. Tanaka's emotions are reflected in the acquired voice data, and adjusted voice data is generated.

[1069] 8. The server compiles the generated audio data into a format that can be played continuously, and generates a single audio file.

[1070] 9. The final audio file is saved to the server and a unique URL (e.g., "https: / / example.com / voice / spaghetti") is generated.

[1071] 10. By accessing the dedicated URL, Tanaka can stream the audio file and hear the contents of the web page in audio, which appropriately reflects Tanaka's emotions.

[1072] Prompt Sentence Examples

[1073] Below are some example prompts to be input to the generative AI model:

[1074] "Play a web page specified by the user's URL (e.g., https: / / example.com / recipes / spaghetti) aloud, providing audio that reflects the user's emotions."

[1075] This system makes it possible to provide speech synthesis that takes into consideration the emotions of users, including the elderly and visually impaired, even on web pages with a wealth of text and image information. This makes it easier for users who have difficulty using the Internet to access information, and promotes understanding of the information through friendly speech output.

[1076] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1077] Step 1:

[1078] The user uses a terminal to open a web browser. They access a web page on the system and enter the URL of the web page they want to play aloud (e.g., "https: / / example.com / recipes / spaghetti"). The entered URL is sent to the server as a request. Input: The URL entered by the user. Output: The URL request to the server.

[1079] Step 2:

[1080] The terminal sends an HTTP request containing the URL information entered by the user to the server. The server receives the request and retrieves the web page at the specified URL. In fact, it downloads the HTML source of the web page using an HTTP GET request. Input: URL request. Output: Retrieved HTML source.

[1081] Step 3:

[1082] The server parses the retrieved HTML source. It uses an HTML parser (e.g., BeautifulSoup) to convert the HTML source into a Document Object Model (DOM) and extracts major text elements such as headings, paragraphs, links, and lists. Input: HTML source. Output: Extracted text elements.

[1083] Step 4:

[1084] The server sends the extracted text elements to a speech synthesis API (for example, Google Text-to-Speech API or Amazon Polly) to obtain audio data. Specifically, each text element is sent individually to the API to obtain audio data for each. Input: Text elements. Output: Audio data.

[1085] Step 5:

[1086] The server uses an emotion engine to analyze the user's emotions. This emotion information is generated based on the user's input voice, text, and non-verbal information (facial expressions, gestures, etc.). Input: Information about the user's emotions. Output: Analyzed emotion data.

[1087] Step 6:

[1088] The server applies emotional expressions to the acquired voice data. Based on the analyzed emotional data, it adjusts the intonation and tone of the voice data. This generates a voice that reflects the user's emotions. Input: Voice data, emotional data. Output: Adjusted voice data.

[1089] Step 7:

[1090] The server combines the adjusted audio data into a single audio file. It concatenates the audio data for multiple text elements into a single file to create a continuously playable audio file. Input: Adjusted audio data. Output: Audio file.

[1091] Step 8:

[1092] The server saves the generated audio file and generates a URL that users can access. Users can stream the audio file by accessing the URL. Input: Audio file. Output: Accessible URL.

[1093] This specific process allows users to listen to web pages in a natural voice that reflects their emotions, providing an environment where even the elderly and visually impaired can easily access information.

[1094] (Application example 2)

[1095] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1096] Users with visual or physical disabilities often have difficulty accessing and understanding the content of web pages. Furthermore, conventional text-to-speech systems do not provide natural-sounding speech output that reflects the user's emotions, making it difficult for them to understand the audio information. This makes it particularly difficult for users with visual or physical disabilities to use online shopping sites.

[1097] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1098] In this invention, the server includes means for accepting a URL specified by a user, means for retrieving the web page of the specified URL, means for analyzing the HTML source of the retrieved web page and extracting key text elements, means for transmitting the extracted text elements to a speech synthesis API and retrieving voice data, means for combining the retrieved voice data into a continuously playable voice file, means for recognizing the user's emotion and adjusting the voice output based on the emotion, and means for providing the generated voice file to the user, thereby enabling users with visual or physical disabilities to understand the content of the web page through natural voice that corresponds to their emotion.

[1099] The "specified URL" is the address of the web page that the user wants to access.

[1100] "Getting a web page" means downloading the HTML source from the specified URL.

[1101] "HTML source" is data written in a markup language that describes the structure and content of a web page.

[1102] A "text element" is an element in the HTML source that contains text information, such as a heading, paragraph, link, or list.

[1103] A "speech synthesis API" is an application program interface used to convert text data into speech data.

[1104] "Audio data" means playable sound data generated using a speech synthesis API.

[1105] An "audio file" is audio data combined into a format that can be played back continuously.

[1106] "Emotion recognition" refers to analyzing a user's current emotional state from their voice or other input.

[1107] "Adjusting the audio output" refers to changing the tone and rate of the generated audio based on the perceived user emotion.

[1108] This invention is a system that enables users with visual or physical disabilities to access web pages by voice. The system retrieves a specified URL, analyzes its HTML source to extract important text elements, and generates voice data using a speech synthesis API. It also recognizes the user's emotions and adjusts the generated voice output to provide it to the user.

[1109] Overall system configuration

[1110] The system mainly consists of the following steps:

[1111] 1. Accepts a URL specified by the user.

[1112] 2. The server retrieves the web page at the specified URL.

[1113] 3. Analyze the obtained HTML source and extract the main text elements.

[1114] 4. The extracted text elements are sent to the speech synthesis API to obtain the speech data.

[1115] 5. Combine the audio data into a continuously playable audio file.

[1116] 6. Recognize user emotions and adjust voice output based on those emotions.

[1117] 7. Provide the final audio file to the user.

[1118] Hardware and software used

[1119] Hardware: The system can be accessed from devices such as PCs, smartphones, and tablets.

[1120] Software: Use "Google Text-to-Speech API" or "Amazon Polly" for speech synthesis. Use an HTML parser such as "BeautifulSoup" for HTML analysis. Use the "emotion_recognition" module for emotion recognition.

[1121] How it works

[1122] 1. URL acceptance

[1123] Users access the system from a device such as a PC or smartphone and enter the specified URL, which is then sent to the server.

[1124] 2. Fetching a Web Page

[1125] The server sends an HTTP request to the specified URL and retrieves the HTML source of the page, using the "requests" library.

[1126] 3. HTML source analysis

[1127] The server parses the HTML source and extracts the main text elements (headings, paragraphs, links, lists, etc.). This analysis is done using "BeautifulSoup."

[1128] 4. Generating Audio Data

[1129] The extracted text elements are sent to a speech synthesis API such as the Google Text-to-Speech API to generate audio data.

[1130] 5. Generate audio files

[1131] The acquired audio data is combined into a format that can be played continuously, and an audio file is generated.

[1132] 6. Emotion recognition and voice output adjustment

[1133] It analyzes emotions from the user's voice input and other non-verbal information and adjusts the generated voice data accordingly, providing a more natural and understandable voice output.

[1134] 7. Providing audio files

[1135] The final generated audio file is provided in a user-accessible format.

[1136] Specific examples

[1137] For example, if a visually impaired user wants to search for a new product on an online shopping site, they can use this system to easily obtain information. When the user enters the URL "https: / / example.com / product / smartphone" into the system, the system analyzes the page and reads out information such as "latest smartphone," "price: 50,000 yen," and "equipped with a high-quality camera." At the same time, it analyzes the user's voice input to recognize emotions and provides information in a tone that corresponds to that emotion.

[1138] Prompt Sentence Examples

[1139] Example prompt:

[1140] A user wants to check product details on an online shopping website. The URL is "https: / / example.com / product / smartphone." The user utters "Tell me about my new smartphone." The system retrieves the webpage, analyzes key text elements, and provides an emotionally-aware speech readout.

[1141] Expected results:

[1142] Information such as "This is the latest smartphone. It costs 50,000 yen and has a high-quality camera" is read out to the user in a tone that reflects their emotions.

[1143] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1144] Step 1:

[1145] Enter the URL specified by the user.

[1146] A user accesses the system using a device such as a PC or smartphone and enters a specified URL. The user's input information is sent to the server. In this step, the input is the "specified URL" and the output is sending the URL to the server. Specifically, the user enters the URL into the input form from the device's browser and clicks the submit button.

[1147] Step 2:

[1148] The server retrieves the web page at the specified URL.

[1149] The server sends an HTTP request to retrieve the HTML source of the web page based on the URL received from the user. The input is the "specified URL" and the output is the "HTML source of the web page." This process uses the "requests" library, specifically requests.get() to send the HTTP request.

[1150] Step 3:

[1151] The obtained HTML source is analyzed and the main text elements are extracted.

[1152] The server parses the retrieved HTML source using the "BeautifulSoup" library and extracts key text elements such as headings, paragraphs, links, and lists. The input is the "HTML source of the web page" and the output is the "extracted text elements." Specifically, it parses the HTML source using BeautifulSoup and extracts the required text elements using the find_all() method.

[1153] Step 4:

[1154] The extracted text elements are sent to a speech synthesis API to obtain speech data.

[1155] The server sends the extracted text elements to a speech synthesis API such as "Google Text-to-Speech API" or "Amazon Polly" to obtain the audio data. The input is the extracted text elements, and the output is audio data. Specifically, the text data is sent to the speech synthesis API endpoint, and audio data is received as a response.

[1156] Step 5:

[1157] The acquired audio data is combined into a continuously playable audio file.

[1158] The server combines the acquired audio data into a format that can be played continuously to generate an audio file. The input is "audio data" and the output is an "audio file." Specifically, the audio data is combined in order and saved as a single audio file.

[1159] Step 6:

[1160] To recognize a user's emotions and adjust audio output based on the emotions.

[1161] The server analyzes the user's voice input and non-verbal information to recognize the user's emotions. It uses the "emotion_recognition" module to perform emotion analysis and adjusts the tone and speed of the voice data based on the results. The input is an audio file and emotional information, and the output is an adjusted audio file. Specifically, it applies an emotion recognition algorithm and changes the parameters of the speech synthesis API according to the user's emotions.

[1162] Step 7:

[1163] The final audio file is provided to the user.

[1164] The server then provides the final audio file to the user in streaming format. The input is the "adjusted audio file" and the output is the "audio provided to the user." Specifically, the audio file is sent as an HTTP response and is played back in real time on the device.

[1165] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1166] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1167] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1168] [Fourth embodiment]

[1169] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1170] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1171] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1172] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1173] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1174] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1175] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1176] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1177] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1178] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1179] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1180] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1181] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1182] The system of the present invention provides voice-based web page access that is easy to use for the elderly, the visually impaired, etc. A specific embodiment of this system will be described below.

[1183] System Overview

[1184] When a user accesses the system and wants to listen to the contents of a web page by audio, they first use a device (PC, smartphone, tablet, etc.). The user opens a web browser, accesses the system, and enters the URL of the web page they want to play by audio. This information is sent to the server, which retrieves the web page at the specified URL, converts it into audio format, and provides it to the user.

[1185] Program processing

[1186] Page retrieval and analysis

[1187] The server receives a request sent by a user and retrieves the web page at the specified URL. Typically, the HTTP request is used to retrieve the HTML source of the web page. This HTML source becomes the data used to analyze the content of the web page.

[1188] Text Extraction and Structure Recognition

[1189] The server analyzes the retrieved HTML source and extracts key text elements such as headings, paragraphs, links, and lists. It uses an HTML parser to generate a Document Object Model (DOM) and extracts the necessary text elements based on this. These extracted text elements are important for speech synthesis.

[1190] Preparing for speech synthesis

[1191] The server sends the extracted text elements to a speech synthesis API (e.g., Google Text-to-Speech API or Amazon Polly), which retrieves the corresponding audio data for each text element and combines them into a format that can be played back continuously.

[1192] Creation and provision of voice homepages

[1193] Once the final audio file is generated, the server stores it for delivery to the user. When the user accesses a specific URL, the server streams the audio file and delivers it to the user.

[1194] Specific examples

[1195] If elderly user Tanaka finds a cooking recipe on a web page but finds the text too small to read, this system can help. Tanaka accesses the system's page using his smartphone browser and enters a URL such as "https: / / example.com / recipes / spaghetti." The server retrieves the page at this URL, analyzes the HTML source, and extracts text elements such as "How to make spaghetti," "Step 1: Boil the pasta," and "Step 2: Make the sauce." These text elements are sent to a speech synthesis API, which obtains the audio data and then compiles it into a single audio file. By playing this audio file on his smartphone, Tanaka can listen to the recipe aloud.

[1196] In this way, by converting web pages rich in text and image information into audio format, even users who have difficulty using the Internet can easily access information, contributing to bridging the digital divide.

[1197] The processing flow will be explained below.

[1198] Step 1:

[1199] The user accesses the system interface using a terminal, enters the URL of the web page they want to listen to, requests speech conversion, and clicks the submit button.

[1200] Step 2:

[1201] The device sends the entered URL information to the server, requesting the server to retrieve the web page via an HTTP request.

[1202] Step 3:

[1203] The server retrieves the specified web page based on the URL received from the user. The server sends an HTTP request to retrieve the HTML source code of the web page.

[1204] Step 4:

[1205] The HTML source code retrieved by the server is analyzed. An HTML parser is used to analyze the structure of the web page and generate a DOM (Document Object Model).

[1206] Step 5:

[1207] The server extracts the main text elements from the parsed DOM. It identifies and extracts elements such as headings (h1, h2, h3...), paragraphs (p), links (a), lists (ul, ol), etc.

[1208] Step 6:

[1209] The server converts the extracted text elements into a format for sending to the speech synthesis API. The text elements are formatted into a format that the API can process.

[1210] Step 7:

[1211] The server sends each text element to the speech synthesis API and receives the corresponding audio data. For example, the server sends the text "Step 1: Boil the pasta" to the API and receives the audio data in response.

[1212] Step 8:

[1213] The server combines each acquired audio data into a format that can be played back continuously. The acquired individual audio data is concatenated in order to generate a single continuous audio file.

[1214] Step 9:

[1215] The server saves the generated audio file on the server. It generates an access URL for the audio file and prepares it so that it can be provided to the user.

[1216] Step 10:

[1217] The server returns the access URL for the audio file to the user, and sends a response to the device containing a link for the user to access the file.

[1218] Step 11:

[1219] The user accesses the provided URL on their device and plays the audio website. The user can then stream the audio file on their smartphone or PC and listen to the contents of the web page.

[1220] Through this series of processes, text-based web pages are converted into audio-based pages and presented to the user in an easy-to-use format.

[1221] Example 1

[1222] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1223] There is a need to make web page information easily accessible to the elderly and visually impaired. However, many current web pages rely on visual information provision, which causes access barriers for these users. It is particularly difficult to obtain information when the text is small and difficult to read, or when the layout is visually complex. Therefore, a system that provides web page content using audio is needed.

[1224] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1225] In this invention, the server includes means for accepting a URL specified by a user, means for retrieving the web page of the specified URL, means for analyzing the markup language source of the retrieved web page and extracting key text elements, means for transmitting the extracted text elements to a speech synthesis service and retrieving audio data, means for combining the retrieved audio data into an audio file that can be played continuously, and means for providing the generated audio file to the user. This allows even the elderly and visually impaired to hear the contents of the web page aloud, making it easier to access information.

[1226] "User" refers to an individual or organization that uses the System.

[1227] "URL" stands for Uniform Resource Locator and is an address that specifies the location of a web resource.

[1228] A "web page" is a document in HTML format that is published on the Internet and refers to the content that is displayed using a browser.

[1229] An "HTTP request" is a communication protocol that allows a client (user's browser) to request a specific resource from a server.

[1230] "HTML source" refers to text data that represents the structure and content of a web page written in a markup language.

[1231] An "HTML parser" is a software tool for analyzing HTML source and generating a DOM (Document Object Model).

[1232] "DOM" is an abbreviation for Document Object Model, and is a model that represents the structure of a web page in a tree format.

[1233] "Text elements" refer to text data such as headings, paragraphs, links, and lists that make up the content of a web page.

[1234] "Speech synthesis service" refers to APIs and software for converting input text data into voice data.

[1235] "Audio Data" refers to digital data in audio format generated by a speech synthesis service.

[1236] An "audio file" is a digital file for storing audio data, saved in a format that allows for continuous playback.

[1237] "Streaming playback" is a method of playing audio files in real time, where playback begins before the file is completely downloaded.

[1238] The system of the present invention allows elderly people, visually impaired people, etc. to use voice to access the contents of web pages. A specific embodiment of this system will be described below.

[1239] A user accesses the system using a device (PC, smartphone, tablet, etc.). First, the user opens the device's web browser and accesses the system's web page. Next, the user enters the URL of the web page they want to play by voice and presses the "Submit" button. This information is sent to the server.

[1240] The server receives a request sent by a user and retrieves the web page at the specified URL using an HTTP request. It then analyzes the retrieved HTML source and generates a Document Object Model (DOM) using an HTML parser (e.g., BeautifulSoup or jsoup). The server then extracts key text elements such as headings, paragraphs, links, and lists from the generated DOM.

[1241] The server then sends the extracted text elements to a speech synthesis service (e.g., Google Text-to-Speech API or Amazon Polly). The service generates the corresponding audio data and returns it to the server. The server then compiles the audio data into a continuous, playable audio file and saves it as a single audio file. Finally, the server provides the saved audio file at a specific URL, which users can access to stream the audio file.

[1242] As a concrete example, consider the case where an elderly person named Tanaka browses a recipe website on his smartphone. Tanaka accesses the system's page using his smartphone's browser, enters the URL "https: / / example.com / recipes / spaghetti" and submits it. The server retrieves the web page at that URL, analyzes the HTML source, and extracts text elements such as headings and paragraphs. These text elements are then sent to a speech synthesis service, which obtains the audio data and compiles it into a single audio file. Tanaka can then stream this audio file and listen to the recipe aloud.

[1243] Examples of prompts that give specific instructions to a generative AI model include:

[1244] 1. "Design a system that uses speech synthesis technology to provide audible content from web pages for the elderly."

[1245] 2. "Explain how to parse the contents of a web page retrieved from a specific URL and convert only the key text elements into audio data."

[1246] 3. "What are the steps in the process of converting text extracted from an HTML source into audio format using the Google Text-to-Speech API?"

[1247] As described above, this system allows elderly people and people with visual impairments to easily access web page content by voice. This system lowers the barriers to using the Internet and makes it easier to access information, thereby contributing to bridging the digital divide.

[1248] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1249] Step 1:

[1250] A user accesses the system using a terminal and enters the URL of the web page they want to play aloud. As input, the user opens a web browser, enters the URL in the designated input field, and presses the submit button. As output, the URL is sent to the server.

[1251] Specific behavior:

[1252] The user opens a browser on their PC or smartphone.

[1253] Enter "https: / / example.com" in the URL input field.

[1254] Click the "Send" button.

[1255] Step 2:

[1256] The server receives a URL request sent by a user and retrieves the web page at the specified URL. As input, it receives the URL sent by the user. As output, it retrieves the HTML source of the web page at the specified URL.

[1257] Specific behavior:

[1258] The server sends an HTTP GET request based on the URL received from the user.

[1259] The server receives the HTML source as a response.

[1260] Step 3:

[1261] The server parses the HTML source and generates a DOM (Document Object Model). It uses the HTML source as input and generates a DOM tree as output.

[1262] Specific behavior:

[1263] The server generates a DOM from the HTML source using an HTML parser (e.g., BeautifulSoup).

[1264] Step 4:

[1265] The server extracts the main text elements (headings, paragraphs, links, lists, etc.) from the generated DOM. It uses the generated DOM as input. It gets a list of extracted text elements as output.

[1266] Specific behavior:

[1267] The server extracts elements such as h1 tags, p tags, a tags, and ul tags from the DOM.

[1268] The extracted elements are compiled into a text list.

[1269] Step 5:

[1270] The server sends the extracted text elements to a speech synthesis service to obtain speech data. As input, it uses the extracted text elements. As output, it obtains the corresponding speech data.

[1271] Specific behavior:

[1272] The server sends the text data to a speech synthesis service (e.g., Google Text-to-Speech API).

[1273] The server receives the audio data from the service.

[1274] Step 6:

[1275] The server compiles the acquired audio data into a continuously playable audio file and saves the audio file. As input, it uses the acquired audio data. As output, it generates and saves a continuously playable audio file.

[1276] Specific behavior:

[1277] The server combines the audio data corresponding to each text element into a single audio file.

[1278] Save the merged audio file (e.g. audio / output.mp3).

[1279] Step 7:

[1280] The server provides the stored audio file at a specific URL so that it can be streamed when accessed by the user. As input, it uses the stored audio file. As output, it generates and provides a streaming playback URL that the user can access.

[1281] Specific behavior:

[1282] The server generates a URL to access the audio file (e.g., https: / / example.com / audio / output.mp3).

[1283] When a user accesses the URL, the server streams the audio file.

[1284] By following the above processing steps in order, this system is able to provide the contents of web pages to elderly people and people with visual impairments by voice.

[1285] (Application example 1)

[1286] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1287] The purpose of this invention is to enable elderly people and people with visual impairments to easily obtain product information in physical stores. In particular, the objective is to improve information accessibility by providing a means for visually impaired people and elderly people to listen to product descriptions and guidance by audio. Furthermore, conventional methods mainly use text information, which creates a visual barrier, and there is a need to eliminate this.

[1288] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1289] In this invention, the server includes means for accepting a URL specified by a user, means for retrieving the web page of the specified URL, means for analyzing the HTML source of the retrieved web page and extracting key text elements, means for transmitting the extracted text elements to a speech synthesis API to retrieve voice data, means for combining the retrieved voice data into a continuously playable voice file, means for providing the generated voice file to the user, and application means installed on smart glasses for providing product information in a physical store in voice format, thereby enabling visually impaired people and the elderly to easily obtain product information in a physical store by voice.

[1290] "Means for accepting a URL specified by a user" refers to a means for accepting a URL entered by a user to obtain information on a specific web page, and for the entire system to start processing based on that URL.

[1291] "Means for obtaining a web page at a specified URL" refers to means for accessing a URL entered by a user and obtaining the HTML source of the corresponding web page.

[1292] "Means for analyzing the HTML source of the acquired web page and extracting major text elements" refers to means for analyzing the acquired HTML source and extracting major text elements such as headings, paragraphs, links, and lists from it.

[1293] "Means for sending the extracted text elements to a speech synthesis API and obtaining speech data" refers to means for sending the extracted text elements to a speech synthesis API to convert them into speech, and obtaining the resulting speech data.

[1294] The "means for combining acquired audio data into a continuously reproducible audio file" refers to a means for combining acquired audio data into one continuous audio file.

[1295] The "means for providing the generated audio file to the user" refers to a means for providing the generated audio file in a form that the user can access.

[1296] "Application means installed on smart glasses for providing product information in audio format in physical stores" refers to a means for installing and executing an application on smart glasses for providing product information and guidance in audio format in physical stores.

[1297] The present invention provides a system that enables visually impaired people and elderly people to easily obtain product information by voice in a brick-and-mortar store, and provides voice guidance using smart glasses. The following describes an embodiment of the present invention.

[1298] System Configuration

[1299] The system includes a server, smart glasses, and a network communication means.

[1300] server

[1301] The server has the following functions:

[1302] 1. A means of accepting a URL specified by the user

[1303] The server receives the URL of the product information page sent from the smart glasses.

[1304] 2. How to get a web page from a specified URL

[1305] The server sends an HTTP request to the specified URL and retrieves the HTML source of the corresponding web page.

[1306] 3. A method for analyzing the HTML source of a retrieved web page and extracting key text elements

[1307] An HTML parser (e.g., BeautifulSoup) is used to parse the HTML source of the web page and extract key text elements such as headings, paragraphs, links, and lists.

[1308] 4. A method for sending the extracted text elements to a speech synthesis API and obtaining the speech data

[1309] The extracted text elements are sent to the Google Text-to-Speech API to obtain the audio data.

[1310] 5. A means of combining the captured audio data into a continuously playable audio file

[1311] The individual audio data is combined into one continuous audio file.

[1312] 6. How to provide the generated audio file to the user

[1313] The completed audio file is sent to the smart glasses so that the user can listen to it.

[1314] Smart Glasses

[1315] The smart glasses have the following features:

[1316] 1. How to enter a specified URL

[1317] The URL for product information is obtained by reading the product tag or QR code.

[1318] 2. How to send a URL to the server

[1319] Send the obtained URL to the server.

[1320] 3. A means to receive and play audio files from the server

[1321] The audio file received from the server is played to the user.

[1322] Hardware and software used

[1323] Smart glasses (e.g. Google Glass)

[1324] Server (e.g. Google Cloud Platform)

[1325] BeautifulSoup for HTML parsing

[1326] Google Text-to-Speech API for speech synthesis

[1327] Specific examples

[1328] For example, if a visually impaired customer is looking for a television in a physical store, they can use this system to obtain product information by voice. When the customer approaches a television wearing smart glasses and scans the QR code, the URL corresponding to that product is sent to the server. The server then analyzes the webpage based on the URL and extracts key text information. This information is converted into audio data using a speech synthesis API and sent back to the smart glasses. The smart glasses then play the received audio file, allowing the customer to listen to the television's product information aloud.

[1329] Prompt Sentence Examples

[1330] Get the page content of "https: / / example.com / store-product-page", extract headings, paragraphs, lists, etc., and convert them into Japanese audio files. The generated audio files are intended to enable elderly and visually impaired people to easily listen to product information in physical stores.

[1331] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1332] Step 1:

[1333] When a user searches for a product in a physical store, they use the smart glasses to read the QR code on the product, which retrieves the URL of the product information page, and the smart glasses then send the URL to the server.

[1334] Input: URL of product information read from QR code

[1335] Output: URL sent to the server

[1336] Specific operation: The smart glasses' camera is used to scan the QR code and the read URL is sent to the server via the network.

[1337] Step 2:

[1338] Based on the URL received from the smart glasses, the server sends an HTTP request to retrieve the HTML source of the specified web page.

[1339] Input: URL sent from smart glasses

[1340] Output: HTML source of the retrieved web page

[1341] What happens: The server makes an HTTP GET request to the URL to retrieve the HTML source of the web page.

[1342] Step 3:

[1343] The server parses the retrieved HTML source and extracts key text elements such as headings, paragraphs, links, and lists.

[1344] Input: HTML source of the web page

[1345] Output: Extracted main text elements

[1346] Specific behavior: Analyzes HTML source using an HTML parser such as BeautifulSoup and extracts the main text elements. Extracts text data for each element such as headings, paragraphs, and lists.

[1347] Step 4:

[1348] The server sends the extracted text elements to the Google Text-to-Speech API, which converts them into audio data.

[1349] Input: Extracted main text elements

[1350] Output: Audio data

[1351] Specific operation: Sends text data to the Google Text-to-Speech API and receives audio data as a response. Audio data is generated for each text element.

[1352] Step 5:

[1353] The server combines the acquired audio data into a continuously playable audio file.

[1354] Input: Audio data

[1355] Output: A combined audio file that can be played continuously

[1356] What it does: Concatenates multiple audio data files into a single audio file, allowing users to listen to information seamlessly.

[1357] Step 6:

[1358] The server sends the generated audio file to the smart glasses and provides it to the user.

[1359] Input: Concatenated audio files

[1360] Output: Audio file sent to smart glasses

[1361] Specific operation: The server sends the audio file to the smart glasses via the network, and the smart glasses receive and play it.

[1362] Specific examples

[1363] If a visually impaired customer is looking for a television in a physical store, they can put on their smart glasses, approach a television product, and scan the QR code. The URL of this QR code is sent to a server, which then retrieves the HTML source of the webpage from the URL. The server analyzes the HTML source and extracts headings, paragraphs, lists, etc., then converts this into audio data and concatenates it into a playable audio file. Finally, this audio file is sent to the smart glasses, allowing the customer to listen to detailed information about the television.

[1364] Prompt Sentence Examples

[1365] Get the page content of "https: / / example.com / store-product-page", extract headings, paragraphs, lists, etc., and convert them into Japanese audio files. The generated audio files are intended to enable elderly and visually impaired people to easily listen to product information in physical stores.

[1366] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1367] The system of the present invention provides easy-to-use voice-based web page access for elderly people, visually impaired people, and others. By recognizing the user's emotions and reflecting them in the voice output, the system provides a more natural and easy-to-listen voice. A specific embodiment of this system is described below.

[1368] System Overview

[1369] When a user accesses the system and wants to listen to the contents of a web page aloud, they first use a device (PC, smartphone, tablet, etc.). The user opens a web browser, accesses the system, and enters the URL of the web page they want to play aloud. This information is sent to the server, which retrieves the web page at the specified URL, converts it into audio format, and provides it to the user. The system also uses an emotion engine to recognize the user's emotions and apply them to the audio output.

[1370] Program processing

[1371] Page retrieval and analysis

[1372] The server receives a request sent by a user and retrieves the web page at the specified URL. Typically, the HTTP request is used to retrieve the HTML source of the web page. This HTML source becomes the data used to analyze the content of the web page.

[1373] Text Extraction and Structure Recognition

[1374] The server analyzes the retrieved HTML source and extracts key text elements such as headings, paragraphs, links, and lists. It uses an HTML parser to generate a Document Object Model (DOM) and extracts the necessary text elements based on this. These extracted text elements are important for speech synthesis.

[1375] Preparing for speech synthesis

[1376] The server sends the extracted text elements to a speech synthesis API (such as Google Text-to-Speech API or Amazon Polly), which retrieves the corresponding audio data for each text element and combines them into a format that can be played back continuously. At this point, an emotion engine is used to simultaneously process information for analyzing the user's emotions.

[1377] Emotion Recognition and Voice Adjustment

[1378] The emotion engine receives the user's input voice, text, and non-verbal input information (facial expressions, gestures, etc.). This allows it to analyze the user's emotions and apply emotional expressions to the acquired voice data. For example, if the user is expressing a happy emotion, the voice data generated by the speech synthesis API will reflect joy and happiness.

[1379] Creation and provision of voice homepages

[1380] Once the final audio file is generated, the server stores it for delivery to the user. When the user accesses a specific URL, the server streams the audio file and delivers it to the user.

[1381] Specific examples

[1382] For example, if an elderly person named Tanaka finds a recipe on a web page but finds the text too small to read, this system can help. Tanaka accesses the system's page using a smartphone browser and enters a URL such as "https: / / example.com / recipes / spaghetti." The server retrieves the page and analyzes the HTML source to extract text elements such as "How to make spaghetti," "Step 1: Boil the pasta," and "Step 2: Make the sauce." These text elements are then sent to a speech synthesis API, which acquires the audio data and compiles it into a single audio file. At the same time, the emotion engine recognizes Tanaka's emotions from her input voice and facial expressions and adjusts the audio data based on those emotions. By playing this audio file on her smartphone, Tanaka can hear the recipe aloud, with the audio appropriately tailored to her current emotions.

[1383] In this way, web pages rich in text and images can be converted into audio format, and the user's emotions can be taken into consideration, making it easier for users who have difficulty using the Internet to access information. Furthermore, the more user-friendly audio output further promotes understanding and use of information.

[1384] The processing flow will be explained below.

[1385] Step 1:

[1386] The user accesses the system interface using a terminal, enters the URL of the web page they want to listen to, requests speech conversion, and clicks the submit button.

[1387] Step 2:

[1388] The device sends the entered URL information to the server, requesting the server to retrieve the web page via an HTTP request.

[1389] Step 3:

[1390] The server retrieves the specified web page based on the URL received from the user. The server sends an HTTP request to retrieve the HTML source code of the web page.

[1391] Step 4:

[1392] The HTML source code retrieved by the server is analyzed. An HTML parser is used to analyze the structure of the web page and generate a DOM (Document Object Model).

[1393] Step 5:

[1394] The server extracts the main text elements from the parsed DOM. It identifies and extracts elements such as headings (h1, h2, h3...), paragraphs (p), links (a), lists (ul, ol), etc.

[1395] Step 6:

[1396] The server converts the extracted text elements into a format for sending to the speech synthesis API. The text elements are formatted into a format that the API can process.

[1397] Step 7:

[1398] The server sends each text element to the speech synthesis API and receives the corresponding audio data. For example, the server sends the text "Step 1: Boil the pasta" to the API and receives the audio data in response.

[1399] Step 8:

[1400] The server activates the emotion engine based on the voice data obtained from the speech synthesis API. The emotion engine receives the user's input voice, text, and non-verbal input information (facial expressions, gestures, etc.).

[1401] Step 9:

[1402] The emotion engine analyzes the user's emotions and adjusts the voice data based on the analysis, for example, adjusting the tone and intonation of the voice to reflect emotions of joy or sadness.

[1403] Step 10:

[1404] The server combines the emotion-adjusted audio data into a continuous, playable format. Each adjusted audio data is concatenated in order to generate a single continuous audio file.

[1405] Step 11:

[1406] The server saves the generated audio file on the server. It generates an access URL for the audio file and prepares it so that it can be provided to the user.

[1407] Step 12:

[1408] The server returns the access URL for the audio file to the user, and sends a response to the device containing a link for the user to access the file.

[1409] Step 13:

[1410] The user accesses the provided URL on their device and plays the audio website. The user can then stream the audio file on their smartphone or PC and listen to the contents of the web page.

[1411] Through this series of processes, text-based web pages are converted into voice-based text, and the voice is adjusted to match the user's emotions, providing information in a more natural and easy-to-listen voice.

[1412] Example 2

[1413] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1414] For the elderly and visually impaired, reading text information on the Internet can be difficult, often resulting in visual stress. Furthermore, conventional text-to-speech systems struggle to provide natural-sounding speech that reflects the user's emotions. This can lead to a mechanical and unfamiliar experience. Furthermore, there are only a limited number of systems that can provide speech that reflects the user's emotions, limiting their range of use.

[1415] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1416] In this invention, the server includes means for accepting a URL specified by a user, means for retrieving the web page at the specified URL, means for analyzing the HTML source of the retrieved web page and extracting key text elements, means for transmitting the extracted text elements to a speech synthesis API and retrieving voice data, means for recognizing the user's emotions, means for applying emotional expressions to the retrieved voice data, means for combining the retrieved voice data into a continuously playable voice file, and means for providing the generated voice file to the user. This enables natural voice reading that takes the user's emotions into consideration. This allows a wide range of users, including the elderly and visually impaired, to receive information more easily.

[1417] "User" refers to a person who wishes to use the system to have a web page read aloud.

[1418] "URL" refers to a Uniform Resource Locator that indicates the address of a web page.

[1419] "Means" refers to a device or program provided to realize a specific function.

[1420] "Web page" refers to a document in HTML format that is published on the Internet.

[1421] "HTML source" refers to the code written in HyperText Markup Language that makes up the content of a Web page.

[1422] "Text elements" refers to textual information such as headings, paragraphs, links, and lists in HTML source.

[1423] "Speech synthesis API" refers to an application programming interface for converting text data into speech data.

[1424] "Audio data" refers to text information that has been converted into audio format.

[1425] "Means for recognizing emotions" refers to sensing technologies and algorithms for analyzing a user's emotional state.

[1426] "Means for applying emotional expressions" refers to technology that adjusts the intonation and tone of voice data based on the analyzed emotions.

[1427] "Audio file" refers to a file that contains audio data in a format that can be played continuously.

[1428] "Means of providing" refers to the technology and infrastructure required to transmit or stream the generated audio file to the user.

[1429] The system of the present invention provides voice-based web page access that is easy to use for elderly people, visually impaired people, etc. This system recognizes the user's emotions and reflects them in the voice output, providing a more natural and easy-to-listen voice.

[1430] Hardware and Software Configuration

[1431] server

[1432] The server has the following features:

[1433] Accepts a URL specified by the user.

[1434] The web page at the specified URL is retrieved via an HTTP request.

[1435] The HTML source of the retrieved web page is analyzed to extract the main text elements. For the specific analysis, an HTML parser such as BeautifulSoup (a Python library) is used.

[1436] The extracted text elements are sent to a speech synthesis API such as Google Text-to-Speech API or Amazon Polly to obtain audio data.

[1437] An emotion engine to recognize user emotions (e.g., Microsoft Cognitive Services' Emotion API).

[1438] Emotional expressions are applied to the audio data, and the adjusted audio data is compiled into an audio file that can be played continuously.

[1439] The generated audio file is saved for provision to the user, and a dedicated URL is generated.

[1440] Terminal

[1441] Users use devices such as PCs, smartphones, and tablets. These devices have a web browser installed. Users access the system through the browser and enter a URL.

[1442] Specific examples

[1443] For example, Mr. Tanaka, an elderly person, accesses the system using a web browser and enters a URL such as "https: / / example.com / recipes / spaghetti."

[1444] 1. The user (Mr. Tanaka) opens a browser on his / her PC, smartphone, or tablet and accesses the system's web page.

[1445] 2. Tanaka enters the URL of the web page she wants to play aloud (for example, "https: / / example.com / recipes / spaghetti").

[1446] 3. The server retrieves the web page at the specified URL through an HTTP request and parses the HTML source.

[1447] 4. The server extracts key text elements such as headings, paragraphs, links, and lists from the parsed HTML source.

[1448] 5. The extracted text elements are sent to a speech synthesis API, such as Google Text-to-Speech API or Amazon Polly, to obtain the corresponding audio data.

[1449] 6. The emotion engine analyzes Tanaka's emotions. For example, if Tanaka says "I'm looking forward to it" in the input voice, the voice is analyzed and the emotion is identified.

[1450] 7. Tanaka's emotions are reflected in the acquired voice data, and adjusted voice data is generated.

[1451] 8. The server compiles the generated audio data into a format that can be played continuously, and generates a single audio file.

[1452] 9. The final audio file is saved to the server and a unique URL (e.g., "https: / / example.com / voice / spaghetti") is generated.

[1453] 10. By accessing the dedicated URL, Tanaka can stream the audio file and hear the contents of the web page in audio, which appropriately reflects Tanaka's emotions.

[1454] Prompt Sentence Examples

[1455] Below are some example prompts to be input to the generative AI model:

[1456] "Play a web page specified by the user's URL (e.g., https: / / example.com / recipes / spaghetti) aloud, providing audio that reflects the user's emotions."

[1457] This system makes it possible to provide speech synthesis that takes into consideration the emotions of users, including the elderly and visually impaired, even on web pages with a wealth of text and image information. This makes it easier for users who have difficulty using the Internet to access information, and promotes understanding of the information through friendly speech output.

[1458] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1459] Step 1:

[1460] The user uses a terminal to open a web browser. They access a web page on the system and enter the URL of the web page they want to play aloud (e.g., "https: / / example.com / recipes / spaghetti"). The entered URL is sent to the server as a request. Input: The URL entered by the user. Output: The URL request to the server.

[1461] Step 2:

[1462] The terminal sends an HTTP request containing the URL information entered by the user to the server. The server receives the request and retrieves the web page at the specified URL. In fact, it downloads the HTML source of the web page using an HTTP GET request. Input: URL request. Output: Retrieved HTML source.

[1463] Step 3:

[1464] The server parses the retrieved HTML source. It uses an HTML parser (e.g., BeautifulSoup) to convert the HTML source into a Document Object Model (DOM) and extracts major text elements such as headings, paragraphs, links, and lists. Input: HTML source. Output: Extracted text elements.

[1465] Step 4:

[1466] The server sends the extracted text elements to a speech synthesis API (for example, Google Text-to-Speech API or Amazon Polly) to obtain audio data. Specifically, each text element is sent individually to the API to obtain audio data for each. Input: Text elements. Output: Audio data.

[1467] Step 5:

[1468] The server uses an emotion engine to analyze the user's emotions. This emotion information is generated based on the user's input voice, text, and non-verbal information (facial expressions, gestures, etc.). Input: Information about the user's emotions. Output: Analyzed emotion data.

[1469] Step 6:

[1470] The server applies emotional expressions to the acquired voice data. Based on the analyzed emotional data, it adjusts the intonation and tone of the voice data. This generates a voice that reflects the user's emotions. Input: Voice data, emotional data. Output: Adjusted voice data.

[1471] Step 7:

[1472] The server combines the adjusted audio data into a single audio file. It concatenates the audio data for multiple text elements into a single file to create a continuously playable audio file. Input: Adjusted audio data. Output: Audio file.

[1473] Step 8:

[1474] The server saves the generated audio file and generates a URL that users can access. Users can stream the audio file by accessing the URL. Input: Audio file. Output: Accessible URL.

[1475] This specific process allows users to listen to web pages in a natural voice that reflects their emotions, providing an environment where even the elderly and visually impaired can easily access information.

[1476] (Application example 2)

[1477] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1478] Users with visual or physical disabilities often have difficulty accessing and understanding the content of web pages. Furthermore, conventional text-to-speech systems do not provide natural-sounding speech output that reflects the user's emotions, making it difficult for them to understand the audio information. This makes it particularly difficult for users with visual or physical disabilities to use online shopping sites.

[1479] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1480] In this invention, the server includes means for accepting a URL specified by a user, means for retrieving the web page of the specified URL, means for analyzing the HTML source of the retrieved web page and extracting key text elements, means for transmitting the extracted text elements to a speech synthesis API and retrieving voice data, means for combining the retrieved voice data into a continuously playable voice file, means for recognizing the user's emotion and adjusting the voice output based on the emotion, and means for providing the generated voice file to the user, thereby enabling users with visual or physical disabilities to understand the content of the web page through natural voice that corresponds to their emotion.

[1481] The "specified URL" is the address of the web page that the user wants to access.

[1482] "Getting a web page" means downloading the HTML source from the specified URL.

[1483] "HTML source" is data written in a markup language that describes the structure and content of a web page.

[1484] A "text element" is an element in the HTML source that contains text information, such as a heading, paragraph, link, or list.

[1485] A "speech synthesis API" is an application program interface used to convert text data into speech data.

[1486] "Audio data" means playable sound data generated using a speech synthesis API.

[1487] An "audio file" is audio data combined into a format that can be played back continuously.

[1488] "Emotion recognition" refers to analyzing a user's current emotional state from their voice or other input.

[1489] "Adjusting the audio output" refers to changing the tone and rate of the generated audio based on the perceived user emotion.

[1490] This invention is a system that enables users with visual or physical disabilities to access web pages by voice. The system retrieves a specified URL, analyzes its HTML source to extract important text elements, and generates voice data using a speech synthesis API. It also recognizes the user's emotions and adjusts the generated voice output to provide it to the user.

[1491] Overall system configuration

[1492] The system mainly consists of the following steps:

[1493] 1. Accepts a URL specified by the user.

[1494] 2. The server retrieves the web page at the specified URL.

[1495] 3. Analyze the obtained HTML source and extract the main text elements.

[1496] 4. The extracted text elements are sent to the speech synthesis API to obtain the speech data.

[1497] 5. Combine the audio data into a continuously playable audio file.

[1498] 6. Recognize user emotions and adjust voice output based on those emotions.

[1499] 7. Provide the final audio file to the user.

[1500] Hardware and software used

[1501] Hardware: The system can be accessed from devices such as PCs, smartphones, and tablets.

[1502] Software: Use "Google Text-to-Speech API" or "Amazon Polly" for speech synthesis. Use an HTML parser such as "BeautifulSoup" for HTML analysis. Use the "emotion_recognition" module for emotion recognition.

[1503] How it works

[1504] 1. URL acceptance

[1505] Users access the system from a device such as a PC or smartphone and enter the specified URL, which is then sent to the server.

[1506] 2. Fetching a Web Page

[1507] The server sends an HTTP request to the specified URL and retrieves the HTML source of the page, using the "requests" library.

[1508] 3. HTML source analysis

[1509] The server parses the HTML source and extracts the main text elements (headings, paragraphs, links, lists, etc.). This analysis is done using "BeautifulSoup."

[1510] 4. Generating Audio Data

[1511] The extracted text elements are sent to a speech synthesis API such as the Google Text-to-Speech API to generate audio data.

[1512] 5. Generate audio files

[1513] The acquired audio data is combined into a format that can be played continuously, and an audio file is generated.

[1514] 6. Emotion recognition and voice output adjustment

[1515] It analyzes emotions from the user's voice input and other non-verbal information and adjusts the generated voice data accordingly, providing a more natural and understandable voice output.

[1516] 7. Providing audio files

[1517] The final generated audio file is provided in a user-accessible format.

[1518] Specific examples

[1519] For example, if a visually impaired user wants to search for a new product on an online shopping site, they can use this system to easily obtain information. When the user enters the URL "https: / / example.com / product / smartphone" into the system, the system analyzes the page and reads out information such as "latest smartphone," "price: 50,000 yen," and "equipped with a high-quality camera." At the same time, it analyzes the user's voice input to recognize emotions and provides information in a tone that corresponds to that emotion.

[1520] Prompt Sentence Examples

[1521] Example prompt:

[1522] A user wants to check product details on an online shopping website. The URL is "https: / / example.com / product / smartphone." The user utters "Tell me about my new smartphone." The system retrieves the webpage, analyzes key text elements, and provides an emotionally-aware speech readout.

[1523] Expected results:

[1524] Information such as "This is the latest smartphone. It costs 50,000 yen and has a high-quality camera" is read out to the user in a tone that reflects their emotions.

[1525] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1526] Step 1:

[1527] Enter the URL specified by the user.

[1528] A user accesses the system using a device such as a PC or smartphone and enters a specified URL. The user's input information is sent to the server. In this step, the input is the "specified URL" and the output is sending the URL to the server. Specifically, the user enters the URL into the input form from the device's browser and clicks the submit button.

[1529] Step 2:

[1530] The server retrieves the web page at the specified URL.

[1531] The server sends an HTTP request to retrieve the HTML source of the web page based on the URL received from the user. The input is the "specified URL" and the output is the "HTML source of the web page." This process uses the "requests" library, specifically requests.get() to send the HTTP request.

[1532] Step 3:

[1533] The obtained HTML source is analyzed and the main text elements are extracted.

[1534] The server parses the retrieved HTML source using the "BeautifulSoup" library and extracts key text elements such as headings, paragraphs, links, and lists. The input is the "HTML source of the web page" and the output is the "extracted text elements." Specifically, it parses the HTML source using BeautifulSoup and extracts the required text elements using the find_all() method.

[1535] Step 4:

[1536] The extracted text elements are sent to a speech synthesis API to obtain speech data.

[1537] The server sends the extracted text elements to a speech synthesis API such as "Google Text-to-Speech API" or "Amazon Polly" to obtain the audio data. The input is the extracted text elements, and the output is audio data. Specifically, the text data is sent to the speech synthesis API endpoint, and audio data is received as a response.

[1538] Step 5:

[1539] The acquired audio data is combined into a continuously playable audio file.

[1540] The server combines the acquired audio data into a format that can be played continuously to generate an audio file. The input is "audio data" and the output is an "audio file." Specifically, the audio data is combined in order and saved as a single audio file.

[1541] Step 6:

[1542] To recognize a user's emotions and adjust audio output based on the emotions.

[1543] The server analyzes the user's voice input and non-verbal information to recognize the user's emotions. It uses the "emotion_recognition" module to perform emotion analysis and adjusts the tone and speed of the voice data based on the results. The input is an audio file and emotional information, and the output is an adjusted audio file. Specifically, it applies an emotion recognition algorithm and changes the parameters of the speech synthesis API according to the user's emotions.

[1544] Step 7:

[1545] The final audio file is provided to the user.

[1546] The server then provides the final audio file to the user in streaming format. The input is the "adjusted audio file" and the output is the "audio provided to the user." Specifically, the audio file is sent as an HTTP response and is played back in real time on the device.

[1547] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1548] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1549] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1550] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1551] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1552] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1553] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1554] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1555] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1556] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1557] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1558] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1559] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1560] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1561] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1562] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1563] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1564] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1565] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1566] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1567] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1568] The following is further disclosed regarding the above embodiment.

[1569] (Claim 1)

[1570] a means for accepting a URL specified by a user;

[1571] A means for retrieving a web page at a specified URL;

[1572] A means for analyzing the HTML source of the retrieved web page and extracting key text elements;

[1573] A means for transmitting the extracted text elements to a speech synthesis API and obtaining speech data;

[1574] means for combining the acquired audio data into a continuously playable audio file;

[1575] a means for providing the generated audio file to a user;

[1576] A system including:

[1577] (Claim 2)

[1578] 10. The system of claim 1, further comprising means for extracting headings, paragraphs, links, and lists from the parsed HTML source.

[1579] (Claim 3)

[1580] 2. The system according to claim 1, further comprising means for sorting the extracted text elements based on their content or importance and arranging them in an appropriate order.

[1581] "Example 1"

[1582] (Claim 1)

[1583] a means for accepting a URL specified by a user;

[1584] A means for retrieving a web page at a specified URL;

[1585] means for parsing the markup language source of the retrieved web page to extract key text elements;

[1586] means for transmitting the extracted text elements to a speech synthesis service to obtain speech data;

[1587] means for combining the acquired audio data into a continuously playable audio file;

[1588] a means for providing the generated audio file to a user;

[1589] A system including:

[1590] (Claim 2)

[1591] 10. The system of claim 1, further comprising means for extracting headings, paragraphs, links, and lists from the parsed markup language source.

[1592] (Claim 3)

[1593] 2. The system according to claim 1, further comprising means for sorting the extracted text elements based on their content or importance and arranging them in an appropriate order.

[1594] "Application Example 1"

[1595] (Claim 1)

[1596] a means for accepting a URL specified by a user;

[1597] A means for retrieving a web page at a specified URL;

[1598] A means for analyzing the HTML source of the retrieved web page and extracting key text elements;

[1599] A means for transmitting the extracted text elements to a speech synthesis API and obtaining speech data;

[1600] means for combining the acquired audio data into a continuously playable audio file;

[1601] a means for providing the generated audio file to a user;

[1602] an application means installed on the smart glasses for providing product guidance in a physical store in audio format;

[1603] A system including:

[1604] (Claim 2)

[1605] 10. The system of claim 1, further comprising means for extracting headings, paragraphs, links, and lists from the parsed HTML source.

[1606] (Claim 3)

[1607] 2. The system according to claim 1, further comprising means for sorting the extracted text elements based on their content or importance and arranging them in an appropriate order.

[1608] "Example 2: Combining Emotion Engines"

[1609] (Claim 1)

[1610] a means for accepting a URL specified by a user;

[1611] A means for retrieving a web page at a specified URL;

[1612] A means for analyzing the HTML source of the retrieved web page and extracting key text elements;

[1613] A means for transmitting the extracted text elements to a speech synthesis API and obtaining speech data;

[1614] a means for recognizing a user's emotion;

[1615] means for applying emotional expressions to the acquired audio data;

[1616] means for combining the acquired audio data into a continuously playable audio file;

[1617] a means for providing the generated audio file to a user;

[1618] A system including:

[1619] (Claim 2)

[1620] 10. The system of claim 1, further comprising means for extracting headings, paragraphs, links, and lists from the parsed HTML source.

[1621] (Claim 3)

[1622] 2. The system according to claim 1, further comprising means for sorting the extracted text elements based on their content or importance and arranging them in an appropriate order.

[1623] "Application example 2 when combining emotion engines"

[1624] (Claim 1)

[1625] a means for accepting a URL specified by a user;

[1626] A means for retrieving a web page at a specified URL;

[1627] A means for analyzing the HTML source of the retrieved web page and extracting key text elements;

[1628] A means for transmitting the extracted text elements to a speech synthesis API and obtaining speech data;

[1629] means for combining the acquired audio data into a continuously playable audio file;

[1630] means for recognizing a user's emotion and adjusting the audio output based on the emotion;

[1631] a means for providing the generated audio file to a user;

[1632] A system including:

[1633] (Claim 2)

[1634] 10. The system of claim 1, further comprising means for extracting headings, paragraphs, links, and lists from the parsed HTML source.

[1635] (Claim 3)

[1636] 2. The system according to claim 1, further comprising means for sorting the extracted text elements based on their content or importance and arranging them in an appropriate order. [Explanation of symbols]

[1637] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for accepting a URL specified by a user; A means for retrieving a web page at a specified URL; A means for analyzing the HTML source of the retrieved web page and extracting key text elements; A means for transmitting the extracted text elements to a speech synthesis API and obtaining speech data; means for combining the acquired audio data into a continuously playable audio file; a means for providing the generated audio file to a user; A system including:

2. 2. The system of claim 1, further comprising means for extracting headings, paragraphs, links, and lists from the parsed HTML source.

3. 2. The system according to claim 1, further comprising means for sorting the extracted text elements based on their contents or importance, and arranging them in an appropriate order.

Images