System

Abstract

A system is provided.SOLUTION: A system comprising: means for acquiring a text from a web page; means for estimating a reference source of a pronoun based on a context and inserting the reference source instead of the pronoun; means for replacing a homophone with a synonym in accordance with the context; means for adjusting an order of a verb, an object, and a subject; means for recursively repeating the above to increase a degree of the text; and means for storing and outputting the converted text.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional screen readers are unable to properly process pronouns and homonyms contained in text, often producing speech output that is difficult for users to understand. This makes it difficult for users to accurately understand the meaning of the text, resulting in the problem of taking a long time to acquire and understand the information. The present invention aims to solve these problems. [Means for solving the problem]

[0005] In this invention, a system is constructed that provides clearer and easier-to-understand speech output by using a means for retrieving text from a web page, a means for inferring the source of a pronoun based on the context and inserting it in place of the pronoun, a means for replacing homonyms with synonyms appropriate to the context, a means for adjusting the order of verbs, objects, and subjects, and a means for recursively repeating the above to improve the quality of the text, thereby enabling users to quickly and accurately understand information when listening to text through a screen reader.

[0006] A "pronoun" is a word used to refer to a particular noun or noun phrase, the source of which must be identified based on context.

[0007] "Homonyms" are words that are pronounced the same but have different meanings, and their correct interpretation must be based on context.

[0008] "Synonyms" refer to different words that are close or identical in meaning and are used as substitutes in appropriate contexts.

[0009] "Context" refers to the context of words and phrases in a sentence or conversation, and is an essential element for accurately understanding meaning and information.

[0010] "Recursive" refers to the repeated execution of a process or calculation, using the results of each operation as input for the next.

[0011] "Level of refinement" indicates the accuracy and naturalness of the text, and refers to a state in which it has been adjusted to be easy for users to understand.

[0012] A "screen reader" is a type of software that allows blind users to hear text on their computer aloud.

[0013] "Structural analysis" refers to the process of analyzing the internal structure of a sentence or text, and is a necessary tool for interpreting grammar and meaning. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] The system of the present invention operates by allowing a server to take text from a web page and convert it into a user-friendly audio format. This process is comprised of multiple steps, each of which performs a specific operation.

[0036] About program processing

[0037] 1. Obtaining text (server)

[0038] The server first retrieves the text from a particular web page using an HTTP request, downloading the HTML data for the page, and then uses an HTML parser to extract the text data.

[0039] 2. Preprocessing (Server)

[0040] The extracted text data is pre-processed using an NLP (Natural Language Processing) library. Specifically, the text is divided into sentences and words, and syntactic analysis is performed to analyze dependencies. This allows for a detailed understanding of how the text is structured.

[0041] 3. Pronoun Resolution (Server)

[0042] The server then detects pronouns in the sentence and infers what each pronoun refers to, using syntactic and contextual analysis, and inserts the inferred reference in place of the pronoun, making the sentence clearer.

[0043] 4. Homophone Processing (Server)

[0044] The server then detects words containing homonyms, which are then replaced with appropriate synonyms based on the context, reducing the chance of misinterpretation.

[0045] 5. Adjusting the text structure (server)

[0046] The server then parses each sentence and adjusts the order of verbs, objects, and subjects to make the sentence more meaningful.

[0047] 6. Recursive Improvement (Server)

[0048] These processes are repeated recursively to refine the text, with each cycle adjusting the text until it feels more natural and understandable.

[0049] 7. Generating and outputting the final converted text (server)

[0050] The final converted text is stored in a storage device and prepared in a suitable format. The server sends this data to the terminal.

[0051] 8. Audio playback (device)

[0052] The device passes the received text data to the screen reader and plays it aloud to the user, allowing the user to receive text information audibly.

[0053] Specific examples

[0054] Original text:

[0055] Whether the plan will be successful or not depends on his efforts.

[0056] After conversion:

[0057] 1. The server retrieves the text from the web page.

[0058] 2. Perform syntactic analysis and understand the context.

[0059] 3. Assume that "he" refers to "Mr. Tanaka" (hypothetical) and replace the pronoun.

[0060] 4. Ensure there are no homonyms.

[0061] 5. Adjust the order of verbs, objects, and subjects while maintaining the sentence structure.

[0062] 6. Continue processing recursively until the sentence is determined to be clear.

[0063] 7. Finally, say, "Whether the plan will be successful or not depends on Tanaka's efforts."

[0064] 8. Send the converted text to the device and play it with a screen reader.

[0065] This allows the user to clearly hear and understand that "Whether the plan will be successful or not depends on Tanaka's efforts."

[0066] The processing flow will be explained below.

[0067] Step 1:

[0068] The server retrieves text data from a specific web page by sending an HTTP request and downloading the HTML data of the target web page.

[0069] Step 2:

[0070] The server parses the HTML data and extracts the text data, using a library such as BeautifulSoup.

[0071] Step 3:

[0072] The server splits the extracted text into sentences and words, and then uses an NLP (Natural Language Processing) library to tokenize it.

[0073] Step 4:

[0074] The server parses the sentence, creating a dependency tree of the sentence and analyzing the internal structure of the sentence.

[0075] Step 5:

[0076] The server detects pronouns and infers the pronoun's source based on the context. It uses syntactic and contextual analysis algorithms to identify the pronoun's source.

[0077] Step 6:

[0078] The server will insert the inferred reference in place of the pronoun, making the sentence clearer.

[0079] Step 7:

[0080] The server detects homonyms using a word list containing homonyms and contextual analysis.

[0081] Step 8:

[0082] The server replaces homonyms with appropriate synonyms based on the context, and selects synonyms by referencing dictionaries such as WordNet.

[0083] Step 9:

[0084] The server reconstructs each sentence, examining the order of verb, object, and subject, and rearranging them to make them easier for the listener to understand.

[0085] Step 10:

[0086] The server repeats the above process recursively, refining the sentence so that each cycle further improves the results.

[0087] Step 11:

[0088] The server saves the final converted text to a text file and outputs the saved data in an appropriate format.

[0089] Step 12:

[0090] The server sends the converted text data to the terminal, using an appropriate format such as JSON.

[0091] Step 13:

[0092] The device passes the received text data to the screen reader, which converts it into speech and prepares to play it back.

[0093] Step 14:

[0094] The user listens to the audio played by the screen reader, audibly receiving and understanding the improved text.

[0095] The above is a specific processing flow in the system of the present invention.

[0096] Example 1

[0097] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0098] Conventional technologies for converting text obtained from web pages into speech format have had many problems. In particular, it has been difficult to understand the source of pronouns, properly replace homonyms, and optimize sentence structure, making it difficult for users to accurately grasp the information. Furthermore, there has been a lack of technology for automating these processes recursively. The present invention solves these problems and provides a system that allows users to more easily obtain accurate information.

[0099] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0100] In this invention, the server includes means for acquiring text data from a web page, means for preprocessing the acquired text data using a natural language processing library, means for inferring the source of a pronoun based on the context and inserting it in place of the pronoun, means for replacing homonyms with appropriate words that match the context, means for adjusting the order of verbs, objects, and subjects based on syntactic analysis, means for recursively repeating the above to improve the quality of the text data, means for storing and outputting the converted text data, and means for playing back the converted sentence data aloud. This allows the user to obtain natural and easy-to-understand audio information.

[0101] A "web page" is a document in which information published on the Internet is written in HTML format.

[0102] "Text data" refers to data that is stored or transmitted electronically as character information.

[0103] "Natural language processing library" is a general term for software tools that allow computers to process and analyze the language that humans use on a daily basis.

[0104] "Preprocessing" refers to the preprocessing of raw data to convert it into a format that can be processed by an algorithm.

[0105] A pronoun is a word used in place of a noun to refer to a person, thing, place, etc.

[0106] A "reference" is a specific noun or situation to which a pronoun refers.

[0107] "Context" refers to the entire situation that determines the meaning of words, such as the context and background information of a sentence or utterance.

[0108] "Homonyms" are words that are pronounced the same but have different meanings.

[0109] An "appropriate word" is one chosen to fit the context and convey the correct meaning.

[0110] "Syntax analysis" is the process of analyzing the grammatical structure of a sentence and clarifying the relationships between each element.

[0111] A "verb" is a part of speech that primarily expresses an action or state.

[0112] An "object" is a noun or noun phrase that is the target of an action.

[0113] A "subject" is a noun or noun phrase that indicates the agent of an action or state.

[0114] "Recursive" is an adjective that refers to the repeated application of the same process.

[0115] "Level of proficiency" is a measure of maturity and completeness of writing, technique, etc.

[0116] "Memory" refers to the long-term storage of data.

[0117] "Output" refers to providing processed data to the outside.

[0118] "Playback by voice" refers to converting text data into machine voice and reading it aloud.

[0119] The system of the present invention operates such that a server takes text data from a web page and converts it into an audio format that is easy for a user to understand.

[0120] 1. Obtaining text (server)

[0121] The server first retrieves text data from a specific web page. To do this, it uses an HTTP request to download the HTML data of the web page. The specific software used is the requests library, and the BeautifulSoup library is used to extract text from the retrieved HTML data.

[0122] 2. Preprocessing (Server)

[0123] The server preprocesses the acquired text data using a natural language processing library, such as spaCy, to split the text into sentences and words and perform syntactic analysis, which reveals the relationships between each part of the text.

[0124] 3. Pronoun Resolution (Server)

[0125] The server detects pronouns in a sentence and infers what each pronoun refers to. Using syntactic and contextual analysis, it inserts the inferred referent in place of the pronoun. As a concrete example, we use the neuralcoref library.

[0126] 4. Homophone Processing (Server)

[0127] The server detects homonyms in a sentence and replaces them with the appropriate word depending on the context, for example, by selecting the appropriate word from a list of homonyms based on the context, reducing the chance of misunderstanding.

[0128] 5. Adjusting the text structure (server)

[0129] Based on the syntactic analysis, the server adjusts the order of the subject, verb, and object, which makes the text more meaningful.

[0130] 6. Recursive Improvement (Server)

[0131] The process is repeated recursively to refine the text data, with each processing cycle adjusting the sentences until they become more natural and understandable.

[0132] 7. Generating and outputting the final converted text (server)

[0133] The server finally stores the converted text data in a storage device, prepares it in an appropriate format, and sends the converted text data to the terminal.

[0134] 8. Audio playback (device)

[0135] The device passes the text data received from the server to the screen reader and plays it aloud to the user. For example, by using the pyttsx3 library, the user can receive text information aurally.

[0136] Specific examples

[0137] Original: Whether the plan succeeds or not depends on his efforts.

[0138] The system retrieves a sentence from a web page and uses an NLP library to parse it and understand the context. It infers that the pronoun "he" refers to "Tanaka-san," and replaces the pronoun. It then checks for the absence of homonyms and adjusts the order of the verb, object, and subject while maintaining the sentence structure. It continues the recursive process, eventually arriving at the sentence, "Whether the plan will be successful or not depends on Tanaka-san's efforts." The converted sentence is sent to the device and played back by a screen reader. The user can clearly hear and understand that "Whether the plan will be successful or not depends on Tanaka-san's efforts."

[0139] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0140] Step 1:

[0141] The server retrieves text data from a specific web page. Specifically, it sends an HTTP request to download the HTML data of the web page. The input is the URL of the web page, and the output is the retrieved HTML data. The server then extracts the text data from the retrieved HTML data using the BeautifulSoup library.

[0142] Step 2:

[0143] The server preprocesses the retrieved text data using a natural language processing library. Specifically, it uses the spaCy library to split the text into sentences and words and perform syntactic analysis. The input is the extracted text data, and the output is a collection of split and parsed sentences.

[0144] Step 3:

[0145] The server detects pronouns in a sentence and infers what each pronoun refers to. The technology used utilizes the neuralcoref library. The input is a collection of parsed sentences, and the output is a sentence in which pronouns have been replaced with their referents. Syntactic and contextual analysis are used for inference.

[0146] Step 4:

[0147] The server detects homonyms and replaces them with the appropriate word depending on the context. It selects the appropriate word from a list of homonyms based on the context. The input is a sentence with pronouns resolved, and the output is a sentence with homonyms replaced with the appropriate word.

[0148] Step 5:

[0149] The server adjusts the order of subjects, verbs, and objects based on syntactic analysis. It reconstructs the relationships between elements in the text and places them in a natural grammatical order. The input is a sentence with homonyms processed, and the output is the adjusted sentence.

[0150] Step 6:

[0151] The server repeats these processes recursively to refine the text data, making adjustments with each processing cycle until the text becomes more natural and easier to understand. The input is the text that has undergone this series of processes, and the output is the final refined text.

[0152] Step 7:

[0153] The server stores the final converted text in a storage device and prepares it in an appropriate format. The data stored in the storage device is sent to the user's terminal. The input is the final edited text, and the output is the data stored and sent to the terminal.

[0154] Step 8:

[0155] The device passes the text data received from the server to a screen reader library (e.g., pyttsx3) and plays it aloud to the user. The input at this time is the received text data, and the output is audio playback for the user. This allows the user to receive text information aurally.

[0156] (Application example 1)

[0157] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0158] In conventional virtual stores, users had few easy ways to obtain product information, and had to rely solely on visual access. As a result, it was not possible to provide appropriate services for visually impaired people or users who have difficulty grasping a large amount of information at once. Furthermore, users had limited ways to quickly and easily access information that interested them, resulting in an incomplete in-store experience.

[0159] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0160] In this invention, the server includes means for retrieving sentences from web pages, means for inferring the source of a pronoun based on the context and inserting it in place of the pronoun, means for replacing homonyms with synonyms that fit the context, means for adjusting the order of verbs, objects, and subjects, means for recursively repeating the above to improve the quality of the sentences, means for storing and outputting the converted sentences, means for retrieving product information in a virtual space and converting the sentences into voice, and means for providing real-time voice guidance of the retrieved product information. This enables real-time voice guidance of product information in a virtual store, making it possible to provide an environment in which all users, including visually impaired people, can easily access information.

[0161] A "web page" is a document in HTML format that can be viewed on the Internet.

[0162] The "means for obtaining text" is a means for extracting text data from a specific web page.

[0163] "Methods of inferring the pronoun's referent based on the context and inserting it in place of the pronoun" refers to methods of identifying the object referred to by a pronoun in a sentence and inserting a specific noun in its place.

[0164] "Means for replacing homonyms with context-appropriate synonyms" refers to means for converting homonyms into synonyms with accurate meanings based on the context.

[0165] "Means for adjusting the order of verbs, objects, and subjects" are means for rearranging the order of verbs, objects, and subjects so that the meaning of a sentence becomes more comprehensible.

[0166] "Means for recursively refining sentences" refers to repeating the process multiple times until the sentences become natural and easy to understand.

[0167] The "means for storing and outputting converted text" refers to a means for saving converted text and outputting it as needed.

[0168] "Means for acquiring product information in a virtual space and converting text into audio" refers to means for acquiring text information about products displayed in a virtual space and converting it into audio.

[0169] The "means for providing voice guidance of acquired product information in real time" is a means for instantly providing information about the product being viewed by the user by voice.

[0170] This invention relates to a system that provides audio guidance of text retrieved from a web page within a virtual store. This system consists of a server and terminals and is designed to enable users to quickly and easily obtain product information within the virtual space.

[0171] System Overview

[0172] The server performs a series of processes to retrieve text from a web page and convert it into a format that is easy for the user to understand. The server includes the following means:

[0173] 1. How to get text from a web page

[0174] 2. A method of guessing the pronoun's source and inserting it in place of the pronoun

[0175] 3. A method for replacing homonyms with context-appropriate synonyms

[0176] 4. A means of adjusting the order of verbs, objects, and subjects

[0177] 5. Repeat the above steps recursively to improve your writing

[0178] 6. A means of storing and outputting the converted text

[0179] 7. A means of obtaining product information in virtual space and converting text into speech

[0180] 8. A means of providing real-time voice guidance of acquired product information

[0181] Program processing

[0182] The server downloads HTML data from a web page using requests and BeautifulSoup, extracts the text, and then uses natural language processing libraries such as spacy to perform syntactic and contextual analysis of the text, resolving pronouns, processing homonyms, and adjusting sentence structure. The final adjusted text is then saved and sent to the device in the appropriate format.

[0183] The device converts the received text into speech using speech synthesis software such as pyttsx3 and provides real-time guidance to the user, allowing the user to easily understand product information by listening to it in the virtual space.

[0184] Hardware and software used

[0185] Server: The requests library for sending HTTP requests, BeautifulSoup for HTML parsing, and spacy for natural language processing.

[0186] Terminal: pyttsx3 for speech synthesis

[0187] Specific examples

[0188] For example, suppose a user is viewing a product page in a virtual store, and the product description is as follows:

[0189] "This product is made from high-quality leather. The most popular color is red."

[0190] In this case, the system behaves as follows:

[0191] The server retrieves the above sentence from the web page and converts it into an intelligible format through natural language processing. The device then converts the sentence into speech, guiding the user in real time: "This product is made of high-quality leather. The most popular color is red."

[0192] Prompt Sentence Examples

[0193] "When a customer is browsing products in a virtual store, we want them to be provided with a voice guide that retrieves product descriptions from a product database. For example, if a customer using smart glasses is viewing a product page for a red leather product, we want to be provided with a real-time guide that says, 'This product is made with high-quality leather. Red is a particularly popular color.'"

[0194] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0195] Step 1:

[0196] The server uses requests to send an HTTP request from the specified web page URL and download the HTML data. The input is the URL and the output is the raw data in HTML format, which is then passed to the next processing step.

[0197] Step 2:

[0198] The server uses the BeautifulSoup library to parse the HTML data and extract the necessary text information. Extracts text from tags, etc. The input is HTML data, and the output is extracted text data.

[0199] Step 3:

[0200] The server uses the spacy library to preprocess the extracted text data for natural language processing. It splits the text into sentences and words and performs syntactic analysis. The input is the text data, and the output is a parsed syntax tree.

[0201] Step 4:

[0202] The server detects pronouns in text data, infers the referent to which the pronoun refers based on the context, and inserts the inferred referent in place of the pronoun. The input is a syntax tree and pronoun data, and the output is the text data with the pronoun replaced.

[0203] Step 5:

[0204] The server detects homonyms and replaces them with appropriate synonyms based on the context. It uses a homonym dictionary to select words that fit the context. The input is text data with pronouns replaced, and the output is text data with homonyms appropriately replaced.

[0205] Step 6:

[0206] The server adjusts the order of verbs, objects, and subjects in the text, making the sentences more natural and easier to understand. The input is text data with homonyms replaced, and the output is text data with adjusted syntax.

[0207] Step 7:

[0208] The server repeats these processes recursively, adjusting each cycle until the sentence becomes more natural and easier to understand. The input is the text data obtained in the previous step, and the output is the optimized text data.

[0209] Step 8:

[0210] The server stores the final converted text data in a storage device and sends it to the terminal in an appropriate format. The input is optimized text data, and the output is data ready to be sent to the terminal.

[0211] Step 9:

[0212] The terminal uses the pyttsx3 library to convert the received text data into speech and provide real-time guidance to the user. The input is the text data sent from the server, and the output is the speech guidance.

[0213] Examples of specific examples and prompts

[0214] Example prompt sentence:

[0215] "When a customer is browsing products in a virtual store, we want them to be provided with a voice guide that retrieves product descriptions from a product database. For example, if a customer using smart glasses is viewing a product page for a red leather product, we want to be provided with a real-time guide that says, 'This product is made with high-quality leather. Red is a particularly popular color.'"

[0216] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0217] The system of the present invention is configured to retrieve text from a web page and convert it into a speech format that is easy for users to understand. Furthermore, by combining it with an emotion engine, the system can adjust the text according to the user's emotions, providing a better user experience.

[0218] About program processing

[0219] Text acquisition and initial analysis (server)

[0220] The server retrieves text data from a specific web page via an HTTP request. It then uses an HTML parser to extract the text data and preprocesses it with an NLP (Natural Language Processing) library. This splits the text into sentences and words, and analyzes dependencies through syntactic analysis.

[0221] Pronoun Resolution (Server)

[0222] The server detects pronouns in a sentence and uses syntactic and contextual analysis algorithms to infer their referents, which are then inserted in place of the pronouns to improve the clarity of the sentence.

[0223] Homophone Processing (Server)

[0224] The server then detects homonyms and replaces them with appropriate synonyms based on the context, referencing a thesaurus such as WordNet, resulting in a sentence with less room for misinterpretation.

[0225] Text structure adjustment (server)

[0226] The server parses each sentence and reframes the order of verb, object, and subject to make it easier for the listener to understand.

[0227] Using the Emotion Engine (Server)

[0228] The server uses an emotion engine to recognize the user's emotions. Specifically, it analyzes the user's voice input and facial expression data to identify the user's current emotional state, and adjusts the text accordingly.

[0229] Recursive Improvements (Server)

[0230] The above processing steps are repeated recursively to refine the sentence. By incorporating an emotion engine into this process, optimal sentences that reflect the user's emotions are generated.

[0231] Generation and output of final converted text (server)

[0232] The completed text data is stored in a storage device and prepared in an appropriate format. The server sends this data to the terminal.

[0233] Playback via audio (device)

[0234] The device passes the received text data to the screen reader, which converts it into speech and plays it back, allowing the user to receive it aurally and understand the information effectively.

[0235] Specific examples

[0236] Original text:

[0237] Whether the plan will be successful or not depends on his efforts.

[0238] Converted example:

[0239] 1. The server retrieves the text from the web page.

[0240] 2. Perform syntactic analysis and understand the context.

[0241] 3. Assuming that "he" is "Tanaka-san" (hypothesis), replace the pronoun.

[0242] 4. Ensure there are no homonyms.

[0243] 5. Adjust the sentence structure and optimize the order of verbs, objects, and subjects.

[0244] 6. The emotion engine recognizes that the user is tired and adjusts the expression to be concise and reassuring.

[0245] 7. Continue recursively to optimize the text.

[0246] 8. Finally, say, "Whether the plan will be successful or not depends on Tanaka's efforts."

[0247] 9. Send the converted text to the device and play it with a screen reader.

[0248] By listening to this optimized voice, users can quickly and accurately understand information. In particular, by utilizing the emotion engine, sentences are provided that take into consideration the user's emotional state, resulting in a better user experience.

[0249] The processing flow will be explained below.

[0250] Step 1:

[0251] The server retrieves text data from a specific web page by issuing an HTTP request and downloading the HTML content of the web page.

[0252] Step 2:

[0253] The server parses the HTML data and uses an HTML parser library such as BeautifulSoup to extract the required text data.

[0254] Step 3:

[0255] The server splits the extracted text data into sentences, and uses an NLP library to tokenize the sentences and identify their components.

[0256] Step 4:

[0257] The server detects pronouns in the sentence using grammatical analysis to identify their positions.

[0258] Step 5:

[0259] The server infers the pronoun's referent, using syntactic and contextual analysis algorithms to identify the specific noun or noun phrase to which the pronoun refers.

[0260] Step 6:

[0261] The server inserts the inferred reference in place of the pronoun, and clarifies the sentence by removing the pronoun and replacing it with a concrete noun.

[0262] Step 7:

[0263] The server detects homonyms, consults a homonym dictionary, and lists target words in the sentence.

[0264] Step 8:

[0265] The server replaces homonyms with appropriate synonyms depending on the context. It analyzes the context and selects the synonym with the most appropriate meaning.

[0266] Step 9:

[0267] The server analyzes the order of verbs, objects, and subjects in each sentence and rearranges them in an order that is easy for the listener to understand. It also uses syntactic analysis to understand the structure of the sentence and rearranges the order as necessary.

[0268] Step 10:

[0269] The server uses an emotion engine to recognize the user's emotions. It collects the user's voice input and facial expression data and uses an emotion analysis model to identify the user's emotional state.

[0270] Step 11:

[0271] The server adjusts the sentences based on the user's emotions, for example, if the user is tired, it adapts the sentences to a concise and reassuring format.

[0272] Step 12:

[0273] The server recursively repeats each of the above processing steps to refine the sentence, repeating the process until the sentence is optimized.

[0274] Step 13:

[0275] The server then stores the converted text data in a storage device and converts it into an appropriate format for output, such as a text file or JSON format.

[0276] Step 14:

[0277] The server sends the converted text data to the terminal, which then sends the data to the terminal via the network.

[0278] Step 15:

[0279] The device passes the received text data to the screen reader, which converts the text into speech and prepares it for playback.

[0280] Step 16:

[0281] The user listens to the audio played by the screen reader, receives the improved text aurally, and can understand the information quickly and accurately.

[0282] Example 2

[0283] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0284] Conventional text conversion systems often use text extracted from web pages as is, rarely converting it into a format that is easy for users to understand. Furthermore, pronoun references can be ambiguous, and interpretations of homonyms can be misleading. Furthermore, the text structure can be complex, and the system does not respond to the user's emotional state, resulting in a poor user experience. Therefore, a text conversion system that takes into account the clarity of the entire text and the user's emotions is needed.

[0285] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0286] In this invention, the server includes means for retrieving sentences from web pages, means for inferring the source of a pronoun based on the context and inserting it in place of the pronoun, means for replacing homonyms with synonyms that match the context, means for analyzing the user's emotional state and adjusting the sentences according to the emotion, means for adjusting the order of verbs, objects, and subjects, means for recursively repeating the above to improve the quality of the sentences, and means for storing and outputting the converted sentences. This makes it possible to convert sentences into ones that are easy for the user to understand, and provides sentences that correspond to the user's emotional state, thereby realizing a better user experience.

[0287] A "web page" is a type of document published on the Internet and constructed using HTML and other web technologies.

[0288] A "sentence" is text data that combines characters and symbols to form meaning.

[0289] A "pronoun" is an element of language used to refer to a particular noun or noun phrase, such as "he," "it," or "I."

[0290] "Context" refers to the context or situation in a sentence or speech that allows us to understand the meaning and intention of words.

[0291] "Homonyms" are words that are pronounced the same but have different meanings.

[0292] "Synonyms" are words that have similar meanings or concepts, but are technically different.

[0293] "Emotional state" refers to the user's current mood or psychological state, including happiness, sadness, anger, fatigue, etc.

[0294] "Syntax analysis" is a natural language processing technique that analyzes the grammatical structure of sentences and phrases and clarifies the relationships between each element.

[0295] "Recursive" refers to the process in which a process or operation is repeatedly performed based on its results.

[0296] "Degree of sophistication" refers to the degree of completion and high quality of a piece of writing or a product.

[0297] The present invention is a system for acquiring text from a web page and converting it into a speech format that is easy for users to understand. The configuration and specific processing steps of the system are described below.

[0298] System configuration

[0299] This system mainly consists of a server and a terminal. The server acquires, analyzes, and converts text, and the terminal plays the converted text in audio format.

[0300] Specific examples of hardware and software used

[0301] The server is built on a common web server platform and requires the following software:

[0302] HTML parsers (e.g. BeautifulSoup, lxml)

[0303] Natural language processing libraries (e.g. spaCy, NLTK)

[0304] Sentiment analysis SDK (e.g. Microsoft Azure's Face API, Google Cloud Speech-to-Text)

[0305] Database (e.g. MySQL, PostgreSQL)

[0306] A terminal is a device capable of playing audio and uses the following software:

[0307] Screen reader APIs (e.g., TalkBack on Android, VoiceOver on iOS)

[0308] Specific examples

[0309] Processing flow

[0310] 1. Text Acquisition and Initial Analysis

[0311] The server sends an HTTP request to the URL provided by the user to retrieve the HTML source code of the web page.

[0312] The obtained HTML source code is analyzed using an HTML parser (e.g., BeautifulSoup) to extract key text data.

[0313] The extracted text data is preprocessed using a natural language processing library (e.g., spaCy), where it is split into sentences and words and parsed syntactically.

[0314] 2. Pronoun Resolution

[0315] The server uses syntactic and contextual analysis algorithms to identify the source of a pronoun in a sentence. For example, it infers that the pronoun "he" refers to "Tanaka-san" and replaces the pronoun with a hypothetical real name.

[0316] 3. Homophone Processing

[0317] The server refers to a thesaurus such as WordNet and replaces detected homonyms with appropriate words depending on the context. For example, it determines whether "kami" means "god" or "paper" depending on the context.

[0318] 4. Adjusting sentence structure

[0319] The server reorders the sentence based on the results of the parsing. For example, it reorders the verb, object, and subject, adjusting the sentence "Whether the plan will be successful or not depends on his efforts" to "Whether the plan will be successful or not depends on Tanaka's efforts."

[0320] 5. Use of Emotion Engine

[0321] The server performs sentiment analysis to identify the user's emotional state, and adjusts the text to be more concise and friendly depending on the user's emotional state (e.g., tired).

[0322] 6. Recursive Improvement

[0323] The server recursively executes the above processing steps to refine the sentence.

[0324] 7. Generating and outputting the final converted text

[0325] The server stores the finalized text in a database and sends it to the device.

[0326] 8. Audio playback

[0327] The device uses a screen reader API (e.g., TalkBack) to convert the received text data into speech and play it back to the user.

[0328] Prompt Sentence Examples

[0329] Here are some examples of prompts that a generative AI model might use:

[0330] "Outline a system that takes specific text from a web page and converts it into a clear, understandable audio format."

[0331] This concludes the description of the preferred embodiment of the present invention. The system allows for the capture of text from a web page and converting it into an easily understandable speech format that is tailored to the user's emotional state, thereby improving the user experience and effectively conveying information.

[0332] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0333] Step 1:

[0334] The server sends an HTTP request to the URL of a web page provided by the user. The input is the URL provided by the user, and the output is the HTML source code of the web page. Specifically, the server makes an HTTP GET request and receives HTML data in response. This HTML data is used in the next analysis step.

[0335] Step 2:

[0336] The server parses the retrieved HTML source code using an HTML parser (e.g., BeautifulSoup, lxml) to extract text data. The input is the HTML source code, and the output is the extracted text data. Specifically, the HTML parser parses the document tree to identify and extract useful text content. This data is pure text with unnecessary HTML tags removed.

[0337] Step 3:

[0338] The server preprocesses the extracted text data using a natural language processing library (e.g., spaCy, NLTK). The input is the extracted text data, and the output is tokenized and parsed data. Specifically, the text is divided into sentences and words, and parsed to identify the relationships between grammatical elements (nouns, verbs, adjectives, etc.). The results of this analysis make it easier to manipulate the text in subsequent processing steps.

[0339] Step 4:

[0340] The server infers the pronoun's referent based on the context and inserts it in place of the pronoun. The input is parsed data, and the output is text with pronouns resolved. Specifically, it uses syntactic analysis and core reference analysis to identify the entity to which the pronoun refers, and then substitutes the identified entity name for the pronoun. This process improves the clarity of the text.

[0341] Step 5:

[0342] The server replaces homonyms with synonyms that fit the context. The input is text with pronouns resolved, and the output is text with homonyms appropriately replaced. Specifically, it references a thesaurus such as WordNet and performs context analysis to select the appropriate word. For example, it determines whether the word "kami" means "god" or "paper" depending on the context, and replaces it with the appropriate word.

[0343] Step 6:

[0344] The server reconstructs the order of verbs, objects, and subjects based on the results of the syntactic analysis. The input is text with homonyms replaced, and the output is a syntactically optimized sentence. Specifically, it uses phrase structure information from an NLP library to reorder the words into subject-verb-object. This process makes the sentence easier for the listener to understand.

[0345] Step 7:

[0346] The server uses an emotion engine to analyze and adjust the user's emotional state. The input is the user's voice input and facial expression data, and the output is text that has been adjusted based on the user's emotional information. Specifically, it uses a voice recognition API and facial expression recognition SDK to analyze the user's current emotional state and adjusts the text based on that information. For example, if the user is tired, the text is changed to a simpler, more friendly expression.

[0347] Step 8:

[0348] The server recursively executes each of the above processing steps to refine the sentence. The input is the sentence refined by the emotion engine, and the output is a further optimized sentence. Specifically, it performs syntactic analysis and sentiment analysis again to evaluate whether additional improvements are possible. If necessary, it performs pronoun resolution, homonym processing, and structural adjustment again.

[0349] Step 9:

[0350] The server saves the final adjusted text in a storage device and sends it to the terminal. The input is the final adjusted text, and the output is the saved data and the text sent to the terminal. Specifically, the text is saved in a database and data is sent to the terminal using a REST API or WebSocket.

[0351] Step 10:

[0352] The device passes the received text data to a screen reader API (e.g. TalkBack, VoiceOver), which converts it into speech and plays it back. The input is text data received from the server, and the output is speech data. Specifically, the screen reader API analyzes the received text and converts it into speech using a speech synthesis engine. The user listens to this speech to understand the information.

[0353] Through the above processing steps, the present system provides information in a voice format that is easy for users to understand and takes into consideration their emotions.

[0354] (Application example 2)

[0355] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0356] There is a huge amount of information on the Internet, and it is not easy to quickly and accurately understand the necessary information. Furthermore, while it is important to properly understand product details, especially in virtual stores, there are limitations to understanding information from text alone. Furthermore, there is a lack of information provided that reflects the user's emotional state, making it difficult to provide an effective user experience. To solve these issues, a system is needed that adjusts text content based on emotions and delivers it in audio format.

[0357] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0358] In this invention, the server includes means for retrieving sentences from web pages, means for inferring the source of a pronoun based on the context and inserting it in place of the pronoun, means for replacing homonyms with synonyms that fit the context, means for adjusting the order of verbs, objects, and subjects, means for recursively repeating the above to improve the quality of the sentences, means for storing and outputting the converted sentences, means for recognizing the user's emotions and adjusting the sentences based on the emotions, and means for outputting the sentences to the user in audio format. This allows the user to accurately understand detailed information about products in a virtual store and to receive appropriate information according to their emotional state.

[0359] A "means of extracting text from a web page" is a technique or method for collecting text data from a specific website on the Internet.

[0360] "Means of inferring the source of a pronoun based on the context and inserting it in place of the pronoun" refers to techniques or methods for identifying the specific noun referred to by a pronoun used in a sentence based on context analysis and replacing the pronoun with the specific noun.

[0361] A "method of replacing homonyms with context-appropriate synonyms" is a technique or method that uses context analysis to replace homonyms in a sentence with words that have the appropriate meaning.

[0362] "Methods for adjusting the order of verbs, objects, and subjects" are techniques or methods for reanalyzing sentence structure and rearranging the order of verbs, objects, and subjects to make them easier for listeners or readers to understand.

[0363] "Means for improving the quality of writing by recursively repeating" refers to techniques and methods for improving the quality of writing by repeating each of the above processes multiple times.

[0364] "Means for storing and outputting converted text" refers to the technology or method for storing text data after processing and providing it to an external party as needed.

[0365] "Means for recognizing a user's emotions and adjusting text based on those emotions" refers to technologies and methods for analyzing voice input and facial expression data to identify a user's emotional state and adjusting text in accordance with that state.

[0366] "Means for outputting text to a user in audio format" refers to techniques or methods for converting text data into audio and providing it to a user.

[0367] The present invention is a system for taking text from a web page and converting it into a user-friendly speech format. The system is implemented through the following steps:

[0368] Hardware and software used

[0369] The server uses the following key software libraries and technologies:

[0370] Requests library: An HTTP client for retrieving HTML data from web pages.

[0371] BeautifulSoup Library: A library for parsing HTML data and extracting text data.

[0372] NLTK (Natural Language Processing Library): A library for dividing text into sentences and words and performing syntactic and contextual analysis.

[0373] WordNet (Thesaurus): A dictionary database for replacing homonyms with context-appropriate synonyms.

[0374] Emotion Recognition Library: A library for collecting and analyzing a user's voice and facial expression data to identify their emotional state.

[0375] Pyttsx3 Library: A speech synthesis library for converting text data into speech and playing it back to the user.

[0376] System configuration

[0377] The system is configured as follows:

[0378] 1. Text Acquisition and Initial Analysis: The server acquires text data from a web page via an HTTP request, then extracts the text data using an HTML parser and preprocesses it using a natural language processing library.

[0379] 2. Pronoun resolution: The server detects pronouns in the sentence and uses context and syntactic analysis algorithms to infer their origin and insert them in place of the pronoun.

[0380] 3. Homophone Processing: The server then detects homonyms and replaces them with appropriate synonyms that fit the context by consulting a thesaurus such as WordNet.

[0381] 4. Sentence structure adjustment: The server parses each sentence and reframes the order of verb, object, and subject to make it easier for the listener to understand.

[0382] 5. Using the emotion engine: The server uses the emotion engine to recognize the user's emotions. Specifically, it analyzes the user's voice input and facial expression data to identify the user's current emotional state. Based on this, it adjusts the sentence.

[0383] 6. Audio playback: The user device passes the received text data to the screen reader, which converts it into audio and plays it back. The user receives this audio through their ears and can effectively understand the information.

[0384] Specific examples

[0385] For example, product explanations in a virtual store are carried out as follows.

[0386] Example

[0387] 1. When a user looks at a particular product, the smart glasses or head-mounted display retrieves detailed information about that product from a web page.

[0388] 2. The server analyzes the text data and processes pronouns and homonyms.

[0389] 3. Recognize the user's emotional state and adjust the sentences based on the emotion.

[0390] 4. The optimized text data is synthesized into speech and played back to the user.

[0391] Prompt Sentence Examples

[0392] Take a sentence from a web page, format it to be user-friendly, and then use an emotion engine to adapt it to the user's emotions. For example, if the user is tired, use a concise and relaxing phrase. Optimize the following sentence for speech: 'This product is made of high-quality materials and is very durable, but I'm a little worried because he's never used it before.'

[0393] This allows the user to effectively understand the information in the virtual store and receive appropriate information according to their emotions.

[0394] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0395] Step 1:

[0396] The server retrieves text data from a web page. The server first sends an HTTP request to a specific website and receives HTML formatted data. To parse the received HTML data, the BeautifulSoup library is used to extract the required text data. The input of this process is the website URL, and the output is the extracted text data.

[0397] Step 2:

[0398] The server performs an initial analysis of the extracted text data using a natural language processing library (such as NLTK). First, the text is divided into sentences and words. Next, syntactic analysis is performed to analyze the dependencies between sentences. The input of this process is the extracted text data, and the output is the divided sentences and words, as well as information about their dependencies.

[0399] Step 3:

[0400] The server detects pronouns in a sentence and infers their origin based on the context. It uses syntactic and contextual analysis algorithms to determine what the pronoun specifically refers to and inserts the actual noun in its place. The input to this process is the parsed result, and the output is the text with the pronoun replaced.

[0401] Step 4:

[0402] The server detects homonyms and replaces them with appropriate synonyms based on the context, referencing a thesaurus such as WordNet. It checks the meaning of words in a sentence based on the context and converts them to appropriate synonyms. The input to this process is the text with pronouns replaced, and the output is the text with homonyms replaced with appropriate synonyms.

[0403] Step 5:

[0404] The server parses each sentence and reconstructs the order of verbs, objects, and subjects, adjusting the order of the sentences to make them easier for the listener to understand. The input to this process is the text with homonyms replaced, and the output is the reconstructed text.

[0405] Step 6:

[0406] The server uses an emotion engine to recognize the user's emotions. It collects voice input and facial expression data from the user's device and analyzes them using an emotion recognition model. Based on the acquired emotional state information, the system adjusts the text to reflect that emotion. The input for this process is the user's voice and facial expression data, and the output is text adjusted based on the user's emotional state.

[0407] Step 7:

[0408] The server then converts the converted and adjusted text data into audio format using a speech synthesis library (such as Pyttsx3). The generated audio file is then sent to the user's device, which then plays the audio using a screen reader. The input of this process is the adjusted text data, and the output is the generated audio file.

[0409] Step 8:

[0410] Users can effectively understand information by hearing the audio played from the device. In the context of a virtual store, users can accurately understand product details and make appropriate purchasing decisions.

[0411] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0412] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0413] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0414] [Second embodiment]

[0415] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0416] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0417] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0418] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0419] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0420] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0421] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0422] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0423] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0424] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0425] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0426] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0427] The system of the present invention operates by allowing a server to take text from a web page and convert it into a user-friendly audio format. This process is comprised of multiple steps, each of which performs a specific operation.

[0428] About program processing

[0429] 1. Obtaining text (server)

[0430] The server first retrieves the text from a particular web page using an HTTP request, downloading the HTML data for the page, and then uses an HTML parser to extract the text data.

[0431] 2. Preprocessing (Server)

[0432] The extracted text data is pre-processed using an NLP (Natural Language Processing) library. Specifically, the text is divided into sentences and words, and syntactic analysis is performed to analyze dependencies. This allows for a detailed understanding of how the text is structured.

[0433] 3. Pronoun Resolution (Server)

[0434] The server then detects pronouns in the sentence and infers what each pronoun refers to, using syntactic and contextual analysis, and inserts the inferred reference in place of the pronoun, making the sentence clearer.

[0435] 4. Homophone Processing (Server)

[0436] The server then detects words containing homonyms, which are then replaced with appropriate synonyms based on the context, reducing the chance of misinterpretation.

[0437] 5. Adjusting the text structure (server)

[0438] The server then parses each sentence and adjusts the order of verbs, objects, and subjects to make the sentence more meaningful.

[0439] 6. Recursive Improvement (Server)

[0440] These processes are repeated recursively to refine the text, with each cycle adjusting the text until it feels more natural and understandable.

[0441] 7. Generating and outputting the final converted text (server)

[0442] The final converted text is stored in a storage device and prepared in a suitable format. The server sends this data to the terminal.

[0443] 8. Audio playback (device)

[0444] The device passes the received text data to the screen reader and plays it aloud to the user, allowing the user to receive text information audibly.

[0445] Specific examples

[0446] Original text:

[0447] Whether the plan will be successful or not depends on his efforts.

[0448] After conversion:

[0449] 1. The server retrieves the text from the web page.

[0450] 2. Perform syntactic analysis and understand the context.

[0451] 3. Assume that "he" refers to "Mr. Tanaka" (hypothetical) and replace the pronoun.

[0452] 4. Ensure there are no homonyms.

[0453] 5. Adjust the order of verbs, objects, and subjects while maintaining the sentence structure.

[0454] 6. Continue processing recursively until the sentence is determined to be clear.

[0455] 7. Finally, say, "Whether the plan will be successful or not depends on Tanaka's efforts."

[0456] 8. Send the converted text to the device and play it with a screen reader.

[0457] This allows the user to clearly hear and understand that "Whether the plan will be successful or not depends on Tanaka's efforts."

[0458] The processing flow will be explained below.

[0459] Step 1:

[0460] The server retrieves text data from a specific web page by sending an HTTP request and downloading the HTML data of the target web page.

[0461] Step 2:

[0462] The server parses the HTML data and extracts the text data, using a library such as BeautifulSoup.

[0463] Step 3:

[0464] The server splits the extracted text into sentences and words, and then uses an NLP (Natural Language Processing) library to tokenize it.

[0465] Step 4:

[0466] The server parses the sentence, creating a dependency tree of the sentence and analyzing the internal structure of the sentence.

[0467] Step 5:

[0468] The server detects pronouns and infers the pronoun's source based on the context. It uses syntactic and contextual analysis algorithms to identify the pronoun's source.

[0469] Step 6:

[0470] The server will insert the inferred reference in place of the pronoun, making the sentence clearer.

[0471] Step 7:

[0472] The server detects homonyms using a word list containing homonyms and contextual analysis.

[0473] Step 8:

[0474] The server replaces homonyms with appropriate synonyms based on the context, and selects synonyms by referencing dictionaries such as WordNet.

[0475] Step 9:

[0476] The server reconstructs each sentence, examining the order of verb, object, and subject, and rearranging them to make them easier for the listener to understand.

[0477] Step 10:

[0478] The server repeats the above process recursively, refining the sentence so that each cycle further improves the results.

[0479] Step 11:

[0480] The server saves the final converted text to a text file and outputs the saved data in an appropriate format.

[0481] Step 12:

[0482] The server sends the converted text data to the terminal, using an appropriate format such as JSON.

[0483] Step 13:

[0484] The device passes the received text data to the screen reader, which converts it into speech and prepares to play it back.

[0485] Step 14:

[0486] The user listens to the audio played by the screen reader, audibly receiving and understanding the improved text.

[0487] The above is a specific processing flow in the system of the present invention.

[0488] Example 1

[0489] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0490] Conventional technologies for converting text obtained from web pages into speech format have had many problems. In particular, it has been difficult to understand the source of pronouns, properly replace homonyms, and optimize sentence structure, making it difficult for users to accurately grasp the information. Furthermore, there has been a lack of technology for automating these processes recursively. The present invention solves these problems and provides a system that allows users to more easily obtain accurate information.

[0491] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0492] In this invention, the server includes means for acquiring text data from a web page, means for preprocessing the acquired text data using a natural language processing library, means for inferring the source of a pronoun based on the context and inserting it in place of the pronoun, means for replacing homonyms with appropriate words that match the context, means for adjusting the order of verbs, objects, and subjects based on syntactic analysis, means for recursively repeating the above to improve the quality of the text data, means for storing and outputting the converted text data, and means for playing back the converted sentence data aloud. This allows the user to obtain natural and easy-to-understand audio information.

[0493] A "web page" is a document in which information published on the Internet is written in HTML format.

[0494] "Text data" refers to data that is stored or transmitted electronically as character information.

[0495] "Natural language processing library" is a general term for software tools that allow computers to process and analyze the language that humans use on a daily basis.

[0496] "Preprocessing" refers to the preprocessing of raw data to convert it into a format that can be processed by an algorithm.

[0497] A pronoun is a word used in place of a noun to refer to a person, thing, place, etc.

[0498] A "reference" is a specific noun or situation to which a pronoun refers.

[0499] "Context" refers to the entire situation that determines the meaning of words, such as the context and background information of a sentence or utterance.

[0500] "Homonyms" are words that are pronounced the same but have different meanings.

[0501] An "appropriate word" is one chosen to fit the context and convey the correct meaning.

[0502] "Syntax analysis" is the process of analyzing the grammatical structure of a sentence and clarifying the relationships between each element.

[0503] A "verb" is a part of speech that primarily expresses an action or state.

[0504] An "object" is a noun or noun phrase that is the target of an action.

[0505] A "subject" is a noun or noun phrase that indicates the agent of an action or state.

[0506] "Recursive" is an adjective that refers to the repeated application of the same process.

[0507] "Level of proficiency" is a measure of maturity and completeness of writing, technique, etc.

[0508] "Memory" refers to the long-term storage of data.

[0509] "Output" refers to providing processed data to the outside.

[0510] "Playback by voice" refers to converting text data into machine voice and reading it aloud.

[0511] The system of the present invention operates such that a server takes text data from a web page and converts it into an audio format that is easy for a user to understand.

[0512] 1. Obtaining text (server)

[0513] The server first retrieves text data from a specific web page. To do this, it uses an HTTP request to download the HTML data of the web page. The specific software used is the requests library, and the BeautifulSoup library is used to extract text from the retrieved HTML data.

[0514] 2. Preprocessing (Server)

[0515] The server preprocesses the acquired text data using a natural language processing library, such as spaCy, to split the text into sentences and words and perform syntactic analysis, which reveals the relationships between each part of the text.

[0516] 3. Pronoun Resolution (Server)

[0517] The server detects pronouns in a sentence and infers what each pronoun refers to. Using syntactic and contextual analysis, it inserts the inferred referent in place of the pronoun. As a concrete example, we use the neuralcoref library.

[0518] 4. Homophone Processing (Server)

[0519] The server detects homonyms in a sentence and replaces them with the appropriate word depending on the context, for example, by selecting the appropriate word from a list of homonyms based on the context, reducing the chance of misunderstanding.

[0520] 5. Adjusting the text structure (server)

[0521] Based on the syntactic analysis, the server adjusts the order of the subject, verb, and object, which makes the text more meaningful.

[0522] 6. Recursive Improvement (Server)

[0523] The process is repeated recursively to refine the text data, with each processing cycle adjusting the sentences until they become more natural and understandable.

[0524] 7. Generating and outputting the final converted text (server)

[0525] The server finally stores the converted text data in a storage device, prepares it in an appropriate format, and sends the converted text data to the terminal.

[0526] 8. Audio playback (device)

[0527] The device passes the text data received from the server to the screen reader and plays it aloud to the user. For example, by using the pyttsx3 library, the user can receive text information aurally.

[0528] Specific examples

[0529] Original: Whether the plan succeeds or not depends on his efforts.

[0530] The system retrieves a sentence from a web page and uses an NLP library to parse it and understand the context. It infers that the pronoun "he" refers to "Tanaka-san," and replaces the pronoun. It then checks for the absence of homonyms and adjusts the order of the verb, object, and subject while maintaining the sentence structure. It continues the recursive process, eventually arriving at the sentence, "Whether the plan will be successful or not depends on Tanaka-san's efforts." The converted sentence is sent to the device and played back by a screen reader. The user can clearly hear and understand that it is "Whether the plan will be successful or not depends on Tanaka-san's efforts."

[0531] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0532] Step 1:

[0533] The server retrieves text data from a specific web page. Specifically, it sends an HTTP request to download the HTML data of the web page. The input is the URL of the web page, and the output is the retrieved HTML data. The server then extracts the text data from the retrieved HTML data using the BeautifulSoup library.

[0534] Step 2:

[0535] The server preprocesses the retrieved text data using a natural language processing library. Specifically, it uses the spaCy library to split the text into sentences and words and perform syntactic analysis. The input is the extracted text data, and the output is a collection of split and parsed sentences.

[0536] Step 3:

[0537] The server detects pronouns in a sentence and infers what each pronoun refers to. The technology used utilizes the neuralcoref library. The input is a collection of parsed sentences, and the output is a sentence in which pronouns have been replaced with their referents. Syntactic and contextual analysis are used for inference.

[0538] Step 4:

[0539] The server detects homonyms and replaces them with the appropriate word depending on the context. It selects the appropriate word from a list of homonyms based on the context. The input is a sentence with pronouns resolved, and the output is a sentence with homonyms replaced with the appropriate word.

[0540] Step 5:

[0541] The server adjusts the order of subjects, verbs, and objects based on syntactic analysis. It reconstructs the relationships between elements in the text and places them in a natural grammatical order. The input is a sentence with homonyms processed, and the output is the adjusted sentence.

[0542] Step 6:

[0543] The server repeats these processes recursively to refine the text data, making adjustments with each processing cycle until the text becomes more natural and easier to understand. The input is the text that has undergone this series of processes, and the output is the final refined text.

[0544] Step 7:

[0545] The server stores the final converted text in a storage device and prepares it in an appropriate format. The data stored in the storage device is sent to the user's terminal. The input is the final edited text, and the output is the data stored and sent to the terminal.

[0546] Step 8:

[0547] The device passes the text data received from the server to a screen reader library (e.g., pyttsx3) and plays it aloud to the user. The input at this time is the received text data, and the output is audio playback for the user. This allows the user to receive text information aurally.

[0548] (Application example 1)

[0549] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0550] In conventional virtual stores, users had few easy ways to obtain product information, and had to rely solely on visual access. As a result, it was not possible to provide appropriate services for visually impaired people or users who have difficulty grasping a large amount of information at once. Furthermore, users had limited ways to quickly and easily access information that interested them, resulting in an incomplete in-store experience.

[0551] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0552] In this invention, the server includes means for retrieving sentences from web pages, means for inferring the source of a pronoun based on the context and inserting it in place of the pronoun, means for replacing homonyms with synonyms that fit the context, means for adjusting the order of verbs, objects, and subjects, means for recursively repeating the above to improve the quality of the sentences, means for storing and outputting the converted sentences, means for retrieving product information in a virtual space and converting the sentences into voice, and means for providing real-time voice guidance of the retrieved product information. This enables real-time voice guidance of product information in a virtual store, making it possible to provide an environment in which all users, including visually impaired people, can easily access information.

[0553] A "web page" is a document in HTML format that can be viewed on the Internet.

[0554] The "means for obtaining text" is a means for extracting text data from a specific web page.

[0555] "Methods of inferring the pronoun's referent based on the context and inserting it in place of the pronoun" refers to methods of identifying the object referred to by a pronoun in a sentence and inserting a specific noun in its place.

[0556] "Means for replacing homonyms with context-appropriate synonyms" refers to means for converting homonyms into synonyms with accurate meanings based on the context.

[0557] "Means for adjusting the order of verbs, objects, and subjects" are means for rearranging the order of verbs, objects, and subjects so that the meaning of a sentence becomes more comprehensible.

[0558] "Means for recursively refining sentences" refers to repeating the process multiple times until the sentences become natural and easy to understand.

[0559] The "means for storing and outputting converted text" refers to a means for saving converted text and outputting it as needed.

[0560] "Means for acquiring product information in a virtual space and converting text into audio" refers to means for acquiring text information about products displayed in a virtual space and converting it into audio.

[0561] The "means for providing voice guidance of acquired product information in real time" is a means for instantly providing information about the product being viewed by the user by voice.

[0562] This invention relates to a system that provides audio guidance of text retrieved from a web page within a virtual store. This system consists of a server and terminals and is designed to enable users to quickly and easily obtain product information within the virtual space.

[0563] System Overview

[0564] The server performs a series of processes to retrieve text from a web page and convert it into a format that is easy for the user to understand. The server includes the following means:

[0565] 1. How to get text from a web page

[0566] 2. A method of guessing the pronoun's source and inserting it in place of the pronoun

[0567] 3. A method for replacing homonyms with context-appropriate synonyms

[0568] 4. A means of adjusting the order of verbs, objects, and subjects

[0569] 5. Repeat the above steps recursively to improve your writing

[0570] 6. A means of storing and outputting the converted text

[0571] 7. A means of obtaining product information in virtual space and converting text into speech

[0572] 8. A means of providing real-time voice guidance of acquired product information

[0573] Program processing

[0574] The server downloads HTML data from a web page using requests and BeautifulSoup, extracts the text, and then uses natural language processing libraries such as spacy to perform syntactic and contextual analysis of the text, resolving pronouns, processing homonyms, and adjusting sentence structure. The final adjusted text is then saved and sent to the device in the appropriate format.

[0575] The device converts the received text into speech using speech synthesis software such as pyttsx3 and provides real-time guidance to the user, allowing the user to easily understand product information by listening to it in the virtual space.

[0576] Hardware and software used

[0577] Server: The requests library for sending HTTP requests, BeautifulSoup for HTML parsing, and spacy for natural language processing.

[0578] Terminal: pyttsx3 for speech synthesis

[0579] Specific examples

[0580] For example, suppose a user is viewing a product page in a virtual store, and the product description is as follows:

[0581] "This product is made from high-quality leather. The most popular color is red."

[0582] In this case, the system behaves as follows:

[0583] The server retrieves the above sentence from the web page and converts it into an intelligible format through natural language processing. The device then converts the sentence into speech, guiding the user in real time: "This product is made of high-quality leather. The most popular color is red."

[0584] Prompt Sentence Examples

[0585] "When a customer is browsing products in a virtual store, we want them to be provided with a voice guide that retrieves product descriptions from a product database. For example, if a customer using smart glasses is viewing a product page for a red leather product, we want to be provided with a real-time guide that says, 'This product is made with high-quality leather. Red is a particularly popular color.'"

[0586] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0587] Step 1:

[0588] The server uses requests to send an HTTP request from the specified web page URL and download the HTML data. The input is the URL and the output is the raw data in HTML format, which is then passed to the next processing step.

[0589] Step 2:

[0590] The server uses the BeautifulSoup library to parse the HTML data and extract the necessary text information.< / url:> Extracts text from tags, etc. The input is HTML data, and the output is extracted text data.

[0591] Step 3:

[0592] The server uses the spacy library to preprocess the extracted text data for natural language processing. It splits the text into sentences and words and performs syntactic analysis. The input is the text data, and the output is a parsed syntax tree.

[0593] Step 4:

[0594] The server detects pronouns in text data, infers the referent to which the pronoun refers based on the context, and inserts the inferred referent in place of the pronoun. The input is a syntax tree and pronoun data, and the output is the text data with the pronoun replaced.

[0595] Step 5:

[0596] The server detects homonyms and replaces them with appropriate synonyms based on the context. It uses a homonym dictionary to select words that fit the context. The input is text data with pronouns replaced, and the output is text data with homonyms appropriately replaced.

[0597] Step 6:

[0598] The server adjusts the order of verbs, objects, and subjects in the text, making the sentences more natural and easier to understand. The input is text data with homonyms replaced, and the output is text data with adjusted syntax.

[0599] Step 7:

[0600] The server repeats these processes recursively, adjusting each cycle until the sentence becomes more natural and easier to understand. The input is the text data obtained in the previous step, and the output is the optimized text data.

[0601] Step 8:

[0602] The server stores the final converted text data in a storage device and sends it to the terminal in an appropriate format. The input is optimized text data, and the output is data ready to be sent to the terminal.

[0603] Step 9:

[0604] The terminal uses the pyttsx3 library to convert the received text data into speech and provide real-time guidance to the user. The input is the text data sent from the server, and the output is the speech guidance.

[0605] Examples of specific examples and prompts

[0606] Example prompt sentence:

[0607] "When a customer is browsing products in a virtual store, we want them to be provided with a voice guide that retrieves product descriptions from a product database. For example, if a customer using smart glasses is viewing a product page for a red leather product, we want to be provided with a real-time guide that says, 'This product is made with high-quality leather. Red is a particularly popular color.'"

[0608] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0609] The system of the present invention is configured to retrieve text from a web page and convert it into a speech format that is easy for users to understand. Furthermore, by combining it with an emotion engine, the system can adjust the text according to the user's emotions, providing a better user experience.

[0610] About program processing

[0611] Text acquisition and initial analysis (server)

[0612] The server retrieves text data from a specific web page via an HTTP request. It then uses an HTML parser to extract the text data and preprocesses it with an NLP (Natural Language Processing) library. This splits the text into sentences and words, and analyzes dependencies through syntactic analysis.

[0613] Pronoun Resolution (Server)

[0614] The server detects pronouns in a sentence and uses syntactic and contextual analysis algorithms to infer their referents, which are then inserted in place of the pronouns to improve the clarity of the sentence.

[0615] Homophone Processing (Server)

[0616] The server then detects homonyms and replaces them with appropriate synonyms based on the context, referencing a thesaurus such as WordNet, resulting in a sentence with less room for misinterpretation.

[0617] Text structure adjustment (server)

[0618] The server parses each sentence and reframes the order of verb, object, and subject to make it easier for the listener to understand.

[0619] Using the Emotion Engine (Server)

[0620] The server uses an emotion engine to recognize the user's emotions. Specifically, it analyzes the user's voice input and facial expression data to identify the user's current emotional state, and adjusts the text accordingly.

[0621] Recursive Improvements (Server)

[0622] The above processing steps are repeated recursively to refine the sentence. By incorporating an emotion engine into this process, optimal sentences that reflect the user's emotions are generated.

[0623] Generation and output of final converted text (server)

[0624] The completed text data is stored in a storage device and prepared in an appropriate format. The server sends this data to the terminal.

[0625] Playback via audio (device)

[0626] The device passes the received text data to the screen reader, which converts it into speech and plays it back, allowing the user to receive it aurally and understand the information effectively.

[0627] Specific examples

[0628] Original text:

[0629] Whether the plan will be successful or not depends on his efforts.

[0630] Converted example:

[0631] 1. The server retrieves the text from the web page.

[0632] 2. Perform syntactic analysis and understand the context.

[0633] 3. Assuming that "he" is "Tanaka-san" (hypothesis), replace the pronoun.

[0634] 4. Ensure there are no homonyms.

[0635] 5. Adjust the sentence structure and optimize the order of verbs, objects, and subjects.

[0636] 6. The emotion engine recognizes that the user is tired and adjusts the expression to be concise and reassuring.

[0637] 7. Continue recursively to optimize the text.

[0638] 8. Finally, say, "Whether the plan will be successful or not depends on Tanaka's efforts."

[0639] 9. Send the converted text to the device and play it with a screen reader.

[0640] By listening to this optimized voice, users can quickly and accurately understand information. In particular, by utilizing the emotion engine, sentences are provided that take into consideration the user's emotional state, resulting in a better user experience.

[0641] The processing flow will be explained below.

[0642] Step 1:

[0643] The server retrieves text data from a specific web page by issuing an HTTP request and downloading the HTML content of the web page.

[0644] Step 2:

[0645] The server parses the HTML data and uses an HTML parser library such as BeautifulSoup to extract the required text data.

[0646] Step 3:

[0647] The server splits the extracted text data into sentences, and uses an NLP library to tokenize the sentences and identify their components.

[0648] Step 4:

[0649] The server detects pronouns in the sentence using grammatical analysis to identify their positions.

[0650] Step 5:

[0651] The server infers the pronoun's referent, using syntactic and contextual analysis algorithms to identify the specific noun or noun phrase to which the pronoun refers.

[0652] Step 6:

[0653] The server inserts the inferred reference in place of the pronoun, and clarifies the sentence by removing the pronoun and replacing it with a concrete noun.

[0654] Step 7:

[0655] The server detects homonyms, consults a homonym dictionary, and lists target words in the sentence.

[0656] Step 8:

[0657] The server replaces homonyms with appropriate synonyms depending on the context. It analyzes the context and selects the synonym with the most appropriate meaning.

[0658] Step 9:

[0659] The server analyzes the order of verbs, objects, and subjects in each sentence and rearranges them in an order that is easy for the listener to understand. It also uses syntactic analysis to understand the structure of the sentence and rearranges the order as necessary.

[0660] Step 10:

[0661] The server uses an emotion engine to recognize the user's emotions. It collects the user's voice input and facial expression data and uses an emotion analysis model to identify the user's emotional state.

[0662] Step 11:

[0663] The server adjusts the sentences based on the user's emotions, for example, if the user is tired, it adapts the sentences to a concise and reassuring format.

[0664] Step 12:

[0665] The server recursively repeats each of the above processing steps to refine the sentence, repeating the process until the sentence is optimized.

[0666] Step 13:

[0667] The server then stores the converted text data in a storage device and converts it into an appropriate format for output, such as a text file or JSON format.

[0668] Step 14:

[0669] The server sends the converted text data to the terminal, which then sends the data to the terminal via the network.

[0670] Step 15:

[0671] The device passes the received text data to the screen reader, which converts the text into speech and prepares it for playback.

[0672] Step 16:

[0673] The user listens to the audio played by the screen reader, receives the improved text aurally, and can understand the information quickly and accurately.

[0674] Example 2

[0675] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0676] Conventional text conversion systems often use text extracted from web pages as is, rarely converting it into a format that is easy for users to understand. Furthermore, pronoun references can be ambiguous, and interpretations of homonyms can be misleading. Furthermore, the text structure can be complex, and the system does not respond to the user's emotional state, resulting in a poor user experience. Therefore, a text conversion system that takes into account the clarity of the entire text and the user's emotions is needed.

[0677] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0678] In this invention, the server includes means for retrieving sentences from web pages, means for inferring the source of a pronoun based on the context and inserting it in place of the pronoun, means for replacing homonyms with synonyms that match the context, means for analyzing the user's emotional state and adjusting the sentences according to the emotion, means for adjusting the order of verbs, objects, and subjects, means for recursively repeating the above to improve the quality of the sentences, and means for storing and outputting the converted sentences. This makes it possible to convert sentences into ones that are easy for the user to understand, and provides sentences that correspond to the user's emotional state, thereby realizing a better user experience.

[0679] A "web page" is a type of document published on the Internet and constructed using HTML and other web technologies.

[0680] A "sentence" is text data that combines characters and symbols to form meaning.

[0681] A "pronoun" is an element of language used to refer to a particular noun or noun phrase, such as "he," "it," or "I."

[0682] "Context" refers to the context or situation in a sentence or speech that allows us to understand the meaning and intention of words.

[0683] "Homonyms" are words that are pronounced the same but have different meanings.

[0684] "Synonyms" are words that have similar meanings or concepts, but are technically different.

[0685] "Emotional state" refers to the user's current mood or psychological state, including happiness, sadness, anger, fatigue, etc.

[0686] "Syntax analysis" is a natural language processing technique that analyzes the grammatical structure of sentences and phrases and clarifies the relationships between each element.

[0687] "Recursive" refers to the process in which a process or operation is repeatedly performed based on its results.

[0688] "Degree of sophistication" refers to the degree of completion and high quality of a piece of writing or a product.

[0689] The present invention is a system for acquiring text from a web page and converting it into a speech format that is easy for users to understand. The configuration and specific processing steps of the system are described below.

[0690] System configuration

[0691] This system mainly consists of a server and a terminal. The server acquires, analyzes, and converts text, and the terminal plays the converted text in audio format.

[0692] Specific examples of hardware and software used

[0693] The server is built on a common web server platform and requires the following software:

[0694] HTML parsers (e.g. BeautifulSoup, lxml)

[0695] Natural language processing libraries (e.g. spaCy, NLTK)

[0696] Sentiment analysis SDK (e.g. Microsoft Azure's Face API, Google Cloud Speech-to-Text)

[0697] Database (e.g. MySQL, PostgreSQL)

[0698] A terminal is a device capable of playing audio and uses the following software:

[0699] Screen reader APIs (e.g., TalkBack on Android, VoiceOver on iOS)

[0700] Specific examples

[0701] Processing flow

[0702] 1. Text Acquisition and Initial Analysis

[0703] The server sends an HTTP request to the URL provided by the user to retrieve the HTML source code of the web page.

[0704] The obtained HTML source code is analyzed using an HTML parser (e.g., BeautifulSoup) to extract key text data.

[0705] The extracted text data is preprocessed using a natural language processing library (e.g., spaCy), where it is split into sentences and words and parsed syntactically.

[0706] 2. Pronoun Resolution

[0707] The server uses syntactic and contextual analysis algorithms to identify the source of a pronoun in a sentence. For example, it infers that the pronoun "he" refers to "Tanaka-san" and replaces the pronoun with a hypothetical real name.

[0708] 3. Homophone Processing

[0709] The server refers to a thesaurus such as WordNet and replaces detected homonyms with appropriate words depending on the context. For example, it determines whether "kami" means "god" or "paper" depending on the context.

[0710] 4. Adjusting sentence structure

[0711] The server reorders the sentence based on the results of the parsing. For example, it reorders the verb, object, and subject, adjusting the sentence "Whether the plan will be successful or not depends on his efforts" to "Whether the plan will be successful or not depends on Tanaka's efforts."

[0712] 5. Use of Emotion Engine

[0713] The server performs sentiment analysis to identify the user's emotional state, and adjusts the text to be more concise and friendly depending on the user's emotional state (e.g., tired).

[0714] 6. Recursive Improvement

[0715] The server recursively executes the above processing steps to refine the sentence.

[0716] 7. Generating and outputting the final converted text

[0717] The server stores the finalized text in a database and sends it to the device.

[0718] 8. Audio playback

[0719] The device uses a screen reader API (e.g., TalkBack) to convert the received text data into speech and play it back to the user.

[0720] Prompt Sentence Examples

[0721] Here are some examples of prompts that a generative AI model might use:

[0722] "Outline a system that takes specific text from a web page and converts it into a clear, understandable audio format."

[0723] This concludes the description of the preferred embodiment of the present invention. The system allows for the capture of text from a web page and converting it into an easily understandable speech format that is tailored to the user's emotional state, thereby improving the user experience and effectively conveying information.

[0724] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0725] Step 1:

[0726] The server sends an HTTP request to the URL of a web page provided by the user. The input is the URL provided by the user, and the output is the HTML source code of the web page. Specifically, the server makes an HTTP GET request and receives HTML data in response. This HTML data is used in the next analysis step.

[0727] Step 2:

[0728] The server parses the retrieved HTML source code using an HTML parser (e.g., BeautifulSoup, lxml) to extract text data. The input is the HTML source code, and the output is the extracted text data. Specifically, the HTML parser parses the document tree to identify and extract useful text content. This data is pure text with unnecessary HTML tags removed.

[0729] Step 3:

[0730] The server preprocesses the extracted text data using a natural language processing library (e.g., spaCy, NLTK). The input is the extracted text data, and the output is tokenized and parsed data. Specifically, the text is divided into sentences and words, and parsed to identify the relationships between grammatical elements (nouns, verbs, adjectives, etc.). The results of this analysis make it easier to manipulate the text in subsequent processing steps.

[0731] Step 4:

[0732] The server infers the pronoun's referent based on the context and inserts it in place of the pronoun. The input is parsed data, and the output is text with pronouns resolved. Specifically, it uses syntactic analysis and core reference analysis to identify the entity to which the pronoun refers, and then substitutes the identified entity name for the pronoun. This process improves the clarity of the text.

[0733] Step 5:

[0734] The server replaces homonyms with synonyms that fit the context. The input is text with pronouns resolved, and the output is text with homonyms appropriately replaced. Specifically, it references a thesaurus such as WordNet and performs context analysis to select the appropriate word. For example, it determines whether the word "kami" means "god" or "paper" depending on the context, and replaces it with the appropriate word.

[0735] Step 6:

[0736] The server reconstructs the order of verbs, objects, and subjects based on the results of the syntactic analysis. The input is text with homonyms replaced, and the output is a syntactically optimized sentence. Specifically, it uses phrase structure information from an NLP library to reorder the words into subject-verb-object. This process makes the sentence easier for the listener to understand.

[0737] Step 7:

[0738] The server uses an emotion engine to analyze and adjust the user's emotional state. The input is the user's voice input and facial expression data, and the output is text that has been adjusted based on the user's emotional information. Specifically, it uses a voice recognition API and facial expression recognition SDK to analyze the user's current emotional state and adjusts the text based on that information. For example, if the user is tired, the text is changed to a simpler, more friendly expression.

[0739] Step 8:

[0740] The server recursively executes each of the above processing steps to refine the sentence. The input is the sentence refined by the emotion engine, and the output is a further optimized sentence. Specifically, it performs syntactic analysis and sentiment analysis again to evaluate whether additional improvements are possible. If necessary, it performs pronoun resolution, homonym processing, and structural adjustment again.

[0741] Step 9:

[0742] The server saves the final adjusted text in a storage device and sends it to the terminal. The input is the final adjusted text, and the output is the saved data and the text sent to the terminal. Specifically, the text is saved in a database and data is sent to the terminal using a REST API or WebSocket.

[0743] Step 10:

[0744] The device passes the received text data to a screen reader API (e.g. TalkBack, VoiceOver), which converts it into speech and plays it back. The input is text data received from the server, and the output is speech data. Specifically, the screen reader API analyzes the received text and converts it into speech using a speech synthesis engine. The user listens to this speech to understand the information.

[0745] Through the above processing steps, the present system provides information in a voice format that is easy for users to understand and takes into consideration their emotions.

[0746] (Application example 2)

[0747] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0748] There is a huge amount of information on the Internet, and it is not easy to quickly and accurately understand the necessary information. Furthermore, while it is important to properly understand product details, especially in virtual stores, there are limitations to understanding information from text alone. Furthermore, there is a lack of information provided that reflects the user's emotional state, making it difficult to provide an effective user experience. To solve these issues, a system is needed that adjusts text content based on emotions and delivers it in audio format.

[0749] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0750] In this invention, the server includes means for retrieving sentences from web pages, means for inferring the source of a pronoun based on the context and inserting it in place of the pronoun, means for replacing homonyms with synonyms that fit the context, means for adjusting the order of verbs, objects, and subjects, means for recursively repeating the above to improve the quality of the sentences, means for storing and outputting the converted sentences, means for recognizing the user's emotions and adjusting the sentences based on the emotions, and means for outputting the sentences to the user in audio format. This allows the user to accurately understand detailed information about products in a virtual store and to receive appropriate information according to their emotional state.

[0751] A "means of extracting text from a web page" is a technique or method for collecting text data from a specific website on the Internet.

[0752] "Means of inferring the source of a pronoun based on the context and inserting it in place of the pronoun" refers to techniques or methods for identifying the specific noun referred to by a pronoun used in a sentence based on context analysis and replacing the pronoun with the specific noun.

[0753] A "method of replacing homonyms with context-appropriate synonyms" is a technique or method that uses context analysis to replace homonyms in a sentence with words that have the appropriate meaning.

[0754] "Methods for adjusting the order of verbs, objects, and subjects" are techniques or methods for reanalyzing sentence structure and rearranging the order of verbs, objects, and subjects to make them easier for listeners or readers to understand.

[0755] "Means for improving the quality of writing by recursively repeating" refers to techniques and methods for improving the quality of writing by repeating each of the above processes multiple times.

[0756] "Means for storing and outputting converted text" refers to the technology or method for storing text data after processing and providing it to an external party as needed.

[0757] "Means for recognizing a user's emotions and adjusting text based on those emotions" refers to technologies and methods for analyzing voice input and facial expression data to identify a user's emotional state and adjusting text in accordance with that state.

[0758] "Means for outputting text to a user in audio format" refers to techniques or methods for converting text data into audio and providing it to a user.

[0759] The present invention is a system for taking text from a web page and converting it into a user-friendly speech format. The system is implemented through the following steps:

[0760] Hardware and software used

[0761] The server uses the following key software libraries and technologies:

[0762] Requests library: An HTTP client for retrieving HTML data from web pages.

[0763] BeautifulSoup Library: A library for parsing HTML data and extracting text data.

[0764] NLTK (Natural Language Processing Library): A library for dividing text into sentences and words and performing syntactic and contextual analysis.

[0765] WordNet (Thesaurus): A dictionary database for replacing homonyms with context-appropriate synonyms.

[0766] Emotion Recognition Library: A library for collecting and analyzing a user's voice and facial expression data to identify their emotional state.

[0767] Pyttsx3 Library: A speech synthesis library for converting text data into speech and playing it back to the user.

[0768] System configuration

[0769] The system is configured as follows:

[0770] 1. Text Acquisition and Initial Analysis: The server acquires text data from a web page via an HTTP request, then extracts the text data using an HTML parser and preprocesses it using a natural language processing library.

[0771] 2. Pronoun resolution: The server detects pronouns in the sentence and uses context and syntactic analysis algorithms to infer their origin and insert them in place of the pronoun.

[0772] 3. Homophone Processing: The server then detects homonyms and replaces them with appropriate synonyms that fit the context by consulting a thesaurus such as WordNet.

[0773] 4. Sentence structure adjustment: The server parses each sentence and reframes the order of verb, object, and subject to make it easier for the listener to understand.

[0774] 5. Using the emotion engine: The server uses the emotion engine to recognize the user's emotions. Specifically, it analyzes the user's voice input and facial expression data to identify the user's current emotional state. Based on this, it adjusts the sentence.

[0775] 6. Audio playback: The user device passes the received text data to the screen reader, which converts it into audio and plays it back. The user receives this audio through their ears and can effectively understand the information.

[0776] Specific examples

[0777] For example, product explanations in a virtual store are carried out as follows.

[0778] Example

[0779] 1. When a user looks at a particular product, the smart glasses or head-mounted display retrieves detailed information about that product from a web page.

[0780] 2. The server analyzes the text data and processes pronouns and homonyms.

[0781] 3. Recognize the user's emotional state and adjust the sentences based on the emotion.

[0782] 4. The optimized text data is synthesized into speech and played back to the user.

[0783] Prompt Sentence Examples

[0784] Take a sentence from a web page, format it to be user-friendly, and then use an emotion engine to adapt it to the user's emotions. For example, if the user is tired, use a concise and relaxing phrase. Optimize the following sentence for speech: 'This product is made of high-quality materials and is very durable, but I'm a little worried because he's never used it before.'

[0785] This allows the user to effectively understand the information in the virtual store and receive appropriate information according to their emotions.

[0786] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0787] Step 1:

[0788] The server retrieves text data from a web page. The server first sends an HTTP request to a specific website and receives HTML formatted data. To parse the received HTML data, the BeautifulSoup library is used to extract the required text data. The input of this process is the website URL, and the output is the extracted text data.

[0789] Step 2:

[0790] The server performs an initial analysis of the extracted text data using a natural language processing library (such as NLTK). First, the text is divided into sentences and words. Next, syntactic analysis is performed to analyze the dependencies between sentences. The input of this process is the extracted text data, and the output is the divided sentences and words, as well as information about their dependencies.

[0791] Step 3:

[0792] The server detects pronouns in a sentence and infers their origin based on the context. It uses syntactic and contextual analysis algorithms to determine what the pronoun specifically refers to and inserts the actual noun in its place. The input to this process is the parsed result, and the output is the text with the pronoun replaced.

[0793] Step 4:

[0794] The server detects homonyms and replaces them with appropriate synonyms based on the context, referencing a thesaurus such as WordNet. It checks the meaning of words in a sentence based on the context and converts them to appropriate synonyms. The input to this process is the text with pronouns replaced, and the output is the text with homonyms replaced with appropriate synonyms.

[0795] Step 5:

[0796] The server parses each sentence and reconstructs the order of verbs, objects, and subjects, adjusting the order of the sentences to make them easier for the listener to understand. The input to this process is the text with homonyms replaced, and the output is the reconstructed text.

[0797] Step 6:

[0798] The server uses an emotion engine to recognize the user's emotions. It collects voice input and facial expression data from the user's device and analyzes them using an emotion recognition model. Based on the acquired emotional state information, the system adjusts the text to reflect that emotion. The input for this process is the user's voice and facial expression data, and the output is text adjusted based on the user's emotional state.

[0799] Step 7:

[0800] The server then converts the converted and adjusted text data into audio format using a speech synthesis library (such as Pyttsx3). The generated audio file is then sent to the user's device, which then plays the audio using a screen reader. The input of this process is the adjusted text data, and the output is the generated audio file.

[0801] Step 8:

[0802] Users can effectively understand information by hearing the audio played from the device. In the context of a virtual store, users can accurately understand product details and make appropriate purchasing decisions.

[0803] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0804] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0805] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0806] [Third embodiment]

[0807] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0808] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0809] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0810] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0811] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0812] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0813] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0814] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0815] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0816] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0817] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0818] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0819] The system of the present invention operates by allowing a server to take text from a web page and convert it into a user-friendly audio format. This process is comprised of multiple steps, each of which performs a specific operation.

[0820] About program processing

[0821] 1. Obtaining text (server)

[0822] The server first retrieves the text from a particular web page using an HTTP request, downloading the HTML data for the page, and then uses an HTML parser to extract the text data.

[0823] 2. Preprocessing (Server)

[0824] The extracted text data is pre-processed using an NLP (Natural Language Processing) library. Specifically, the text is divided into sentences and words, and syntactic analysis is performed to analyze dependencies. This allows for a detailed understanding of how the text is structured.

[0825] 3. Pronoun Resolution (Server)

[0826] The server then detects pronouns in the sentence and infers what each pronoun refers to, using syntactic and contextual analysis, and inserts the inferred reference in place of the pronoun, making the sentence clearer.

[0827] 4. Homophone Processing (Server)

[0828] The server then detects words containing homonyms, which are then replaced with appropriate synonyms based on the context, reducing the chance of misinterpretation.

[0829] 5. Adjusting the text structure (server)

[0830] The server then parses each sentence and adjusts the order of verbs, objects, and subjects to make the sentence more meaningful.

[0831] 6. Recursive Improvement (Server)

[0832] These processes are repeated recursively to refine the text, with each cycle adjusting the text until it feels more natural and understandable.

[0833] 7. Generating and outputting the final converted text (server)

[0834] The final converted text is stored in a storage device and prepared in a suitable format. The server sends this data to the terminal.

[0835] 8. Audio playback (device)

[0836] The device passes the received text data to the screen reader and plays it aloud to the user, allowing the user to receive text information audibly.

[0837] Specific examples

[0838] Original text:

[0839] Whether the plan will be successful or not depends on his efforts.

[0840] After conversion:

[0841] 1. The server retrieves the text from the web page.

[0842] 2. Perform syntactic analysis and understand the context.

[0843] 3. Assume that "he" refers to "Mr. Tanaka" (hypothetical) and replace the pronoun.

[0844] 4. Ensure there are no homonyms.

[0845] 5. Adjust the order of verbs, objects, and subjects while maintaining the sentence structure.

[0846] 6. Continue processing recursively until the sentence is determined to be clear.

[0847] 7. Finally, say, "Whether the plan will be successful or not depends on Tanaka's efforts."

[0848] 8. Send the converted text to the device and play it with a screen reader.

[0849] This allows the user to clearly hear and understand that "Whether the plan will be successful or not depends on Tanaka's efforts."

[0850] The processing flow will be explained below.

[0851] Step 1:

[0852] The server retrieves text data from a specific web page by sending an HTTP request and downloading the HTML data of the target web page.

[0853] Step 2:

[0854] The server parses the HTML data and extracts the text data, using a library such as BeautifulSoup.

[0855] Step 3:

[0856] The server splits the extracted text into sentences and words, and then uses an NLP (Natural Language Processing) library to tokenize it.

[0857] Step 4:

[0858] The server parses the sentence, creating a dependency tree of the sentence and analyzing the internal structure of the sentence.

[0859] Step 5:

[0860] The server detects pronouns and infers the pronoun's source based on the context. It uses syntactic and contextual analysis algorithms to identify the pronoun's source.

[0861] Step 6:

[0862] The server will insert the inferred reference in place of the pronoun, making the sentence clearer.

[0863] Step 7:

[0864] The server detects homonyms using a word list containing homonyms and contextual analysis.

[0865] Step 8:

[0866] The server replaces homonyms with appropriate synonyms based on the context, and selects synonyms by referencing dictionaries such as WordNet.

[0867] Step 9:

[0868] The server reconstructs each sentence, examining the order of verb, object, and subject, and rearranging them to make them easier for the listener to understand.

[0869] Step 10:

[0870] The server repeats the above process recursively, refining the sentence so that each cycle further improves the results.

[0871] Step 11:

[0872] The server saves the final converted text to a text file and outputs the saved data in an appropriate format.

[0873] Step 12:

[0874] The server sends the converted text data to the terminal, using an appropriate format such as JSON.

[0875] Step 13:

[0876] The device passes the received text data to the screen reader, which converts it into speech and prepares to play it back.

[0877] Step 14:

[0878] The user listens to the audio played by the screen reader, audibly receiving and understanding the improved text.

[0879] The above is a specific processing flow in the system of the present invention.

[0880] Example 1

[0881] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0882] Conventional technologies for converting text obtained from web pages into speech format have had many problems. In particular, it has been difficult to understand the source of pronouns, properly replace homonyms, and optimize sentence structure, making it difficult for users to accurately grasp the information. Furthermore, there has been a lack of technology for automating these processes recursively. The present invention solves these problems and provides a system that allows users to more easily obtain accurate information.

[0883] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0884] In this invention, the server includes means for acquiring text data from a web page, means for preprocessing the acquired text data using a natural language processing library, means for inferring the source of a pronoun based on the context and inserting it in place of the pronoun, means for replacing homonyms with appropriate words that match the context, means for adjusting the order of verbs, objects, and subjects based on syntactic analysis, means for recursively repeating the above to improve the quality of the text data, means for storing and outputting the converted text data, and means for playing back the converted sentence data aloud. This allows the user to obtain natural and easy-to-understand audio information.

[0885] A "web page" is a document in which information published on the Internet is written in HTML format.

[0886] "Text data" refers to data that is stored or transmitted electronically as character information.

[0887] "Natural language processing library" is a general term for software tools that allow computers to process and analyze the language that humans use on a daily basis.

[0888] "Preprocessing" refers to the preprocessing of raw data to convert it into a format that can be processed by an algorithm.

[0889] A pronoun is a word used in place of a noun to refer to a person, thing, place, etc.

[0890] A "reference" is a specific noun or situation to which a pronoun refers.

[0891] "Context" refers to the entire situation that determines the meaning of words, such as the context and background information of a sentence or utterance.

[0892] "Homonyms" are words that are pronounced the same but have different meanings.

[0893] An "appropriate word" is one chosen to fit the context and convey the correct meaning.

[0894] "Syntax analysis" is the process of analyzing the grammatical structure of a sentence and clarifying the relationships between each element.

[0895] A "verb" is a part of speech that primarily expresses an action or state.

[0896] An "object" is a noun or noun phrase that is the target of an action.

[0897] A "subject" is a noun or noun phrase that indicates the agent of an action or state.

[0898] "Recursive" is an adjective that refers to the repeated application of the same process.

[0899] "Level of proficiency" is a measure of maturity and completeness of writing, technique, etc.

[0900] "Memory" refers to the long-term storage of data.

[0901] "Output" refers to providing processed data to the outside.

[0902] "Playback by voice" refers to converting text data into machine voice and reading it aloud.

[0903] The system of the present invention operates such that a server takes text data from a web page and converts it into an audio format that is easy for a user to understand.

[0904] 1. Obtaining text (server)

[0905] The server first retrieves text data from a specific web page. To do this, it uses an HTTP request to download the HTML data of the web page. The specific software used is the requests library, and the BeautifulSoup library is used to extract text from the retrieved HTML data.

[0906] 2. Preprocessing (Server)

[0907] The server preprocesses the acquired text data using a natural language processing library, such as spaCy, to split the text into sentences and words and perform syntactic analysis, which reveals the relationships between each part of the text.

[0908] 3. Pronoun Resolution (Server)

[0909] The server detects pronouns in a sentence and infers what each pronoun refers to. Using syntactic and contextual analysis, it inserts the inferred referent in place of the pronoun. As a concrete example, we use the neuralcoref library.

[0910] 4. Homophone Processing (Server)

[0911] The server detects homonyms in a sentence and replaces them with the appropriate word depending on the context, for example, by selecting the appropriate word from a list of homonyms based on the context, reducing the chance of misunderstanding.

[0912] 5. Adjusting the text structure (server)

[0913] Based on the syntactic analysis, the server adjusts the order of the subject, verb, and object, which makes the text more meaningful.

[0914] 6. Recursive Improvement (Server)

[0915] The process is repeated recursively to refine the text data, with each processing cycle adjusting the sentences until they become more natural and understandable.

[0916] 7. Generating and outputting the final converted text (server)

[0917] The server finally stores the converted text data in a storage device, prepares it in an appropriate format, and sends the converted text data to the terminal.

[0918] 8. Audio playback (device)

[0919] The device passes the text data received from the server to the screen reader and plays it aloud to the user. For example, by using the pyttsx3 library, the user can receive text information aurally.

[0920] Specific examples

[0921] Original: Whether the plan succeeds or not depends on his efforts.

[0922] The system retrieves a sentence from a web page and uses an NLP library to parse it and understand the context. It infers that the pronoun "he" refers to "Tanaka-san," and replaces the pronoun. It then checks for the absence of homonyms and adjusts the order of the verb, object, and subject while maintaining the sentence structure. It continues the recursive process, eventually arriving at the sentence, "Whether the plan will be successful or not depends on Tanaka-san's efforts." The converted sentence is sent to the device and played back by a screen reader. The user can clearly hear and understand that it is "Whether the plan will be successful or not depends on Tanaka-san's efforts."

[0923] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0924] Step 1:

[0925] The server retrieves text data from a specific web page. Specifically, it sends an HTTP request to download the HTML data of the web page. The input is the URL of the web page, and the output is the retrieved HTML data. The server then extracts the text data from the retrieved HTML data using the BeautifulSoup library.

[0926] Step 2:

[0927] The server preprocesses the retrieved text data using a natural language processing library. Specifically, it uses the spaCy library to split the text into sentences and words and perform syntactic analysis. The input is the extracted text data, and the output is a collection of split and parsed sentences.

[0928] Step 3:

[0929] The server detects pronouns in a sentence and infers what each pronoun refers to. The technology used utilizes the neuralcoref library. The input is a collection of parsed sentences, and the output is a sentence in which pronouns have been replaced with their referents. Syntactic and contextual analysis are used for inference.

[0930] Step 4:

[0931] The server detects homonyms and replaces them with the appropriate word depending on the context. It selects the appropriate word from a list of homonyms based on the context. The input is a sentence with pronouns resolved, and the output is a sentence with homonyms replaced with the appropriate word.

[0932] Step 5:

[0933] The server adjusts the order of subjects, verbs, and objects based on syntactic analysis. It reconstructs the relationships between elements in the text and places them in a natural grammatical order. The input is a sentence with homonyms processed, and the output is the adjusted sentence.

[0934] Step 6:

[0935] The server repeats these processes recursively to refine the text data, making adjustments with each processing cycle until the text becomes more natural and easier to understand. The input is the text that has undergone this series of processes, and the output is the final refined text.

[0936] Step 7:

[0937] The server stores the final converted text in a storage device and prepares it in an appropriate format. The data stored in the storage device is sent to the user's terminal. The input is the final edited text, and the output is the data stored and sent to the terminal.

[0938] Step 8:

[0939] The device passes the text data received from the server to a screen reader library (e.g., pyttsx3) and plays it aloud to the user. The input at this time is the received text data, and the output is audio playback for the user. This allows the user to receive text information aurally.

[0940] (Application example 1)

[0941] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0942] In conventional virtual stores, users had few easy ways to obtain product information, and had to rely solely on visual access. As a result, it was not possible to provide appropriate services for visually impaired people or users who have difficulty grasping a large amount of information at once. Furthermore, users had limited ways to quickly and easily access information that interested them, resulting in an incomplete in-store experience.

[0943] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0944] In this invention, the server includes means for retrieving sentences from web pages, means for inferring the source of a pronoun based on the context and inserting it in place of the pronoun, means for replacing homonyms with synonyms that fit the context, means for adjusting the order of verbs, objects, and subjects, means for recursively repeating the above to improve the quality of the sentences, means for storing and outputting the converted sentences, means for retrieving product information in a virtual space and converting the sentences into voice, and means for providing real-time voice guidance of the retrieved product information. This enables real-time voice guidance of product information in a virtual store, making it possible to provide an environment in which all users, including visually impaired people, can easily access information.

[0945] A "web page" is a document in HTML format that can be viewed on the Internet.

[0946] The "means for obtaining text" is a means for extracting text data from a specific web page.

[0947] "Methods of inferring the pronoun's referent based on the context and inserting it in place of the pronoun" refers to methods of identifying the object referred to by a pronoun in a sentence and inserting a specific noun in its place.

[0948] "Means for replacing homonyms with context-appropriate synonyms" refers to means for converting homonyms into synonyms with accurate meanings based on the context.

[0949] "Means for adjusting the order of verbs, objects, and subjects" are means for rearranging the order of verbs, objects, and subjects so that the meaning of a sentence becomes more comprehensible.

[0950] "Means for recursively refining sentences" refers to repeating the process multiple times until the sentences become natural and easy to understand.

[0951] The "means for storing and outputting converted text" refers to a means for saving converted text and outputting it as needed.

[0952] "Means for acquiring product information in a virtual space and converting text into audio" refers to means for acquiring text information about products displayed in a virtual space and converting it into audio.

[0953] The "means for providing voice guidance of acquired product information in real time" is a means for instantly providing information about the product being viewed by the user by voice.

[0954] This invention relates to a system that provides audio guidance of text retrieved from a web page within a virtual store. This system consists of a server and terminals and is designed to enable users to quickly and easily obtain product information within the virtual space.

[0955] System Overview

[0956] The server performs a series of processes to retrieve text from a web page and convert it into a format that is easy for the user to understand. The server includes the following means:

[0957] 1. How to get text from a web page

[0958] 2. A method of guessing the pronoun's source and inserting it in place of the pronoun

[0959] 3. A method for replacing homonyms with context-appropriate synonyms

[0960] 4. A means of adjusting the order of verbs, objects, and subjects

[0961] 5. Repeat the above steps recursively to improve your writing

[0962] 6. A means of storing and outputting the converted text

[0963] 7. A means of obtaining product information in virtual space and converting text into speech

[0964] 8. A means of providing real-time voice guidance of acquired product information

[0965] Program processing

[0966] The server downloads HTML data from a web page using requests and BeautifulSoup, extracts the text, and then uses natural language processing libraries such as spacy to perform syntactic and contextual analysis of the text, resolving pronouns, processing homonyms, and adjusting sentence structure. The final adjusted text is then saved and sent to the device in the appropriate format.

[0967] The device converts the received text into speech using speech synthesis software such as pyttsx3 and provides real-time guidance to the user, allowing the user to easily understand product information by listening to it in the virtual space.

[0968] Hardware and software used

[0969] Server: The requests library for sending HTTP requests, BeautifulSoup for HTML parsing, and spacy for natural language processing.

[0970] Terminal: pyttsx3 for speech synthesis

[0971] Specific examples

[0972] For example, suppose a user is viewing a product page in a virtual store, and the product description is as follows:

[0973] "This product is made from high-quality leather. The most popular color is red."

[0974] In this case, the system behaves as follows:

[0975] The server retrieves the above sentence from the web page and converts it into an intelligible format through natural language processing. The device then converts the sentence into speech, guiding the user in real time: "This product is made of high-quality leather. The most popular color is red."

[0976] Prompt Sentence Examples

[0977] "When a customer is browsing products in a virtual store, we want them to be provided with a voice guide that retrieves product descriptions from a product database. For example, if a customer using smart glasses is viewing a product page for a red leather product, we want to be provided with a real-time guide that says, 'This product is made with high-quality leather. Red is a particularly popular color.'"

[0978] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0979] Step 1:

[0980] The server uses requests to send an HTTP request from the specified web page URL and download the HTML data. The input is the URL and the output is the raw data in HTML format, which is then passed to the next processing step.

[0981] Step 2:

[0982] The server uses the BeautifulSoup library to parse the HTML data and extract the necessary text information.< / url:> Extracts text from tags, etc. The input is HTML data, and the output is extracted text data.

[0983] Step 3:

[0984] The server uses the spacy library to preprocess the extracted text data for natural language processing. It splits the text into sentences and words and performs syntactic analysis. The input is the text data, and the output is a parsed syntax tree.

[0985] Step 4:

[0986] The server detects pronouns in text data, infers the referent to which the pronoun refers based on the context, and inserts the inferred referent in place of the pronoun. The input is a syntax tree and pronoun data, and the output is the text data with the pronoun replaced.

[0987] Step 5:

[0988] The server detects homonyms and replaces them with appropriate synonyms based on the context. It uses a homonym dictionary to select words that fit the context. The input is text data with pronouns replaced, and the output is text data with homonyms appropriately replaced.

[0989] Step 6:

[0990] The server adjusts the order of verbs, objects, and subjects in the text, making the sentences more natural and easier to understand. The input is text data with homonyms replaced, and the output is text data with adjusted syntax.

[0991] Step 7:

[0992] The server repeats these processes recursively, adjusting each cycle until the sentence becomes more natural and easier to understand. The input is the text data obtained in the previous step, and the output is the optimized text data.

[0993] Step 8:

[0994] The server stores the final converted text data in a storage device and sends it to the terminal in an appropriate format. The input is optimized text data, and the output is data ready to be sent to the terminal.

[0995] Step 9:

[0996] The terminal uses the pyttsx3 library to convert the received text data into speech and provide real-time guidance to the user. The input is the text data sent from the server, and the output is the speech guidance.

[0997] Examples of specific examples and prompts

[0998] Example prompt sentence:

[0999] "When a customer is browsing products in a virtual store, we want them to be provided with a voice guide that retrieves product descriptions from a product database. For example, if a customer using smart glasses is viewing a product page for a red leather product, we want to be provided with a real-time guide that says, 'This product is made with high-quality leather. Red is a particularly popular color.'"

[1000] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1001] The system of the present invention is configured to retrieve text from a web page and convert it into a speech format that is easy for users to understand. Furthermore, by combining it with an emotion engine, the system can adjust the text according to the user's emotions, providing a better user experience.

[1002] About program processing

[1003] Text acquisition and initial analysis (server)

[1004] The server retrieves text data from a specific web page via an HTTP request. It then uses an HTML parser to extract the text data and preprocesses it with an NLP (Natural Language Processing) library. This splits the text into sentences and words, and analyzes dependencies through syntactic analysis.

[1005] Pronoun Resolution (Server)

[1006] The server detects pronouns in a sentence and uses syntactic and contextual analysis algorithms to infer their referents, which are then inserted in place of the pronouns to improve the clarity of the sentence.

[1007] Homophone Processing (Server)

[1008] The server then detects homonyms and replaces them with appropriate synonyms based on the context, referencing a thesaurus such as WordNet, resulting in a sentence with less room for misinterpretation.

[1009] Text structure adjustment (server)

[1010] The server parses each sentence and reframes the order of verb, object, and subject to make it easier for the listener to understand.

[1011] Using the Emotion Engine (Server)

[1012] The server uses an emotion engine to recognize the user's emotions. Specifically, it analyzes the user's voice input and facial expression data to identify the user's current emotional state, and adjusts the text accordingly.

[1013] Recursive Improvements (Server)

[1014] The above processing steps are repeated recursively to refine the sentence. By incorporating an emotion engine into this process, optimal sentences that reflect the user's emotions are generated.

[1015] Generation and output of final converted text (server)

[1016] The completed text data is stored in a storage device and prepared in an appropriate format. The server sends this data to the terminal.

[1017] Playback via audio (device)

[1018] The device passes the received text data to the screen reader, which converts it into speech and plays it back, allowing the user to receive it aurally and understand the information effectively.

[1019] Specific examples

[1020] Original text:

[1021] Whether the plan will be successful or not depends on his efforts.

[1022] Converted example:

[1023] 1. The server retrieves the text from the web page.

[1024] 2. Perform syntactic analysis and understand the context.

[1025] 3. Assuming that "he" is "Tanaka-san" (hypothesis), replace the pronoun.

[1026] 4. Ensure there are no homonyms.

[1027] 5. Adjust the sentence structure and optimize the order of verbs, objects, and subjects.

[1028] 6. The emotion engine recognizes that the user is tired and adjusts the expression to be concise and reassuring.

[1029] 7. Continue recursively to optimize the text.

[1030] 8. Finally, say, "Whether the plan will be successful or not depends on Tanaka's efforts."

[1031] 9. Send the converted text to the device and play it with a screen reader.

[1032] By listening to this optimized voice, users can quickly and accurately understand information. In particular, by utilizing the emotion engine, sentences are provided that take into consideration the user's emotional state, resulting in a better user experience.

[1033] The processing flow will be explained below.

[1034] Step 1:

[1035] The server retrieves text data from a specific web page by issuing an HTTP request and downloading the HTML content of the web page.

[1036] Step 2:

[1037] The server parses the HTML data and uses an HTML parser library such as BeautifulSoup to extract the required text data.

[1038] Step 3:

[1039] The server splits the extracted text data into sentences, and uses an NLP library to tokenize the sentences and identify their components.

[1040] Step 4:

[1041] The server detects pronouns in the sentence using grammatical analysis to identify their positions.

[1042] Step 5:

[1043] The server infers the pronoun's referent, using syntactic and contextual analysis algorithms to identify the specific noun or noun phrase to which the pronoun refers.

[1044] Step 6:

[1045] The server inserts the inferred reference in place of the pronoun, and clarifies the sentence by removing the pronoun and replacing it with a concrete noun.

[1046] Step 7:

[1047] The server detects homonyms, consults a homonym dictionary, and lists target words in the sentence.

[1048] Step 8:

[1049] The server replaces homonyms with appropriate synonyms depending on the context. It analyzes the context and selects the synonym with the most appropriate meaning.

[1050] Step 9:

[1051] The server analyzes the order of verbs, objects, and subjects in each sentence and rearranges them in an order that is easy for the listener to understand. It also uses syntactic analysis to understand the structure of the sentence and rearranges the order as necessary.

[1052] Step 10:

[1053] The server uses an emotion engine to recognize the user's emotions. It collects the user's voice input and facial expression data and uses an emotion analysis model to identify the user's emotional state.

[1054] Step 11:

[1055] The server adjusts the sentences based on the user's emotions, for example, if the user is tired, it adapts the sentences to a concise and reassuring format.

[1056] Step 12:

[1057] The server recursively repeats each of the above processing steps to refine the sentence, repeating the process until the sentence is optimized.

[1058] Step 13:

[1059] The server then stores the converted text data in a storage device and converts it into an appropriate format for output, such as a text file or JSON format.

[1060] Step 14:

[1061] The server sends the converted text data to the terminal, which then sends the data to the terminal via the network.

[1062] Step 15:

[1063] The device passes the received text data to the screen reader, which converts the text into speech and prepares it for playback.

[1064] Step 16:

[1065] The user listens to the audio played by the screen reader, receives the improved text aurally, and can understand the information quickly and accurately.

[1066] Example 2

[1067] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1068] Conventional text conversion systems often use text extracted from web pages as is, rarely converting it into a format that is easy for users to understand. Furthermore, pronoun references can be ambiguous, and interpretations of homonyms can be misleading. Furthermore, the text structure can be complex, and the system does not respond to the user's emotional state, resulting in a poor user experience. Therefore, a text conversion system that takes into account the clarity of the entire text and the user's emotions is needed.

[1069] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1070] In this invention, the server includes means for retrieving sentences from web pages, means for inferring the source of a pronoun based on the context and inserting it in place of the pronoun, means for replacing homonyms with synonyms that match the context, means for analyzing the user's emotional state and adjusting the sentences according to the emotion, means for adjusting the order of verbs, objects, and subjects, means for recursively repeating the above to improve the quality of the sentences, and means for storing and outputting the converted sentences. This makes it possible to convert sentences into ones that are easy for the user to understand, and provides sentences that correspond to the user's emotional state, thereby realizing a better user experience.

[1071] A "web page" is a type of document published on the Internet and constructed using HTML and other web technologies.

[1072] A "sentence" is text data that combines characters and symbols to form meaning.

[1073] A "pronoun" is an element of language used to refer to a particular noun or noun phrase, such as "he," "it," or "I."

[1074] "Context" refers to the context or situation in a sentence or speech that allows us to understand the meaning and intention of words.

[1075] "Homonyms" are words that are pronounced the same but have different meanings.

[1076] "Synonyms" are words that have similar meanings or concepts, but are technically different.

[1077] "Emotional state" refers to the user's current mood or psychological state, including happiness, sadness, anger, fatigue, etc.

[1078] "Syntax analysis" is a natural language processing technique that analyzes the grammatical structure of sentences and phrases and clarifies the relationships between each element.

[1079] "Recursive" refers to the process in which a process or operation is repeatedly performed based on its results.

[1080] "Degree of sophistication" refers to the degree of completion and high quality of a piece of writing or a product.

[1081] The present invention is a system for acquiring text from a web page and converting it into a speech format that is easy for users to understand. The configuration and specific processing steps of the system are described below.

[1082] System configuration

[1083] This system mainly consists of a server and a terminal. The server acquires, analyzes, and converts text, and the terminal plays the converted text in audio format.

[1084] Specific examples of hardware and software used

[1085] The server is built on a common web server platform and requires the following software:

[1086] HTML parsers (e.g. BeautifulSoup, lxml)

[1087] Natural language processing libraries (e.g. spaCy, NLTK)

[1088] Sentiment analysis SDK (e.g. Microsoft Azure's Face API, Google Cloud Speech-to-Text)

[1089] Database (e.g. MySQL, PostgreSQL)

[1090] A terminal is a device capable of playing audio and uses the following software:

[1091] Screen reader APIs (e.g., TalkBack on Android, VoiceOver on iOS)

[1092] Specific examples

[1093] Processing flow

[1094] 1. Text Acquisition and Initial Analysis

[1095] The server sends an HTTP request to the URL provided by the user to retrieve the HTML source code of the web page.

[1096] The obtained HTML source code is analyzed using an HTML parser (e.g., BeautifulSoup) to extract key text data.

[1097] The extracted text data is preprocessed using a natural language processing library (e.g., spaCy), where it is split into sentences and words and parsed syntactically.

[1098] 2. Pronoun Resolution

[1099] The server uses syntactic and contextual analysis algorithms to identify the source of a pronoun in a sentence. For example, it infers that the pronoun "he" refers to "Tanaka-san" and replaces the pronoun with a hypothetical real name.

[1100] 3. Homophone Processing

[1101] The server refers to a thesaurus such as WordNet and replaces detected homonyms with appropriate words depending on the context. For example, it determines whether "kami" means "god" or "paper" depending on the context.

[1102] 4. Adjusting sentence structure

[1103] The server reorders the sentence based on the results of the parsing. For example, it reorders the verb, object, and subject, adjusting the sentence "Whether the plan will be successful or not depends on his efforts" to "Whether the plan will be successful or not depends on Tanaka's efforts."

[1104] 5. Use of Emotion Engine

[1105] The server performs sentiment analysis to identify the user's emotional state, and adjusts the text to be more concise and friendly depending on the user's emotional state (e.g., tired).

[1106] 6. Recursive Improvement

[1107] The server recursively executes the above processing steps to refine the sentence.

[1108] 7. Generating and outputting the final converted text

[1109] The server stores the finalized text in a database and sends it to the device.

[1110] 8. Audio playback

[1111] The device uses a screen reader API (e.g., TalkBack) to convert the received text data into speech and play it back to the user.

[1112] Prompt Sentence Examples

[1113] Here are some examples of prompts that a generative AI model might use:

[1114] "Outline a system that takes specific text from a web page and converts it into a clear, understandable audio format."

[1115] This concludes the description of the preferred embodiment of the present invention. The system allows for the capture of text from a web page and converting it into an easily understandable speech format that is tailored to the user's emotional state, thereby improving the user experience and effectively conveying information.

[1116] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1117] Step 1:

[1118] The server sends an HTTP request to the URL of a web page provided by the user. The input is the URL provided by the user, and the output is the HTML source code of the web page. Specifically, the server makes an HTTP GET request and receives HTML data in response. This HTML data is used in the next analysis step.

[1119] Step 2:

[1120] The server parses the retrieved HTML source code using an HTML parser (e.g., BeautifulSoup, lxml) to extract text data. The input is the HTML source code, and the output is the extracted text data. Specifically, the HTML parser parses the document tree to identify and extract useful text content. This data is pure text with unnecessary HTML tags removed.

[1121] Step 3:

[1122] The server preprocesses the extracted text data using a natural language processing library (e.g., spaCy, NLTK). The input is the extracted text data, and the output is tokenized and parsed data. Specifically, the text is divided into sentences and words, and parsed to identify the relationships between grammatical elements (nouns, verbs, adjectives, etc.). The results of this analysis make it easier to manipulate the text in subsequent processing steps.

[1123] Step 4:

[1124] The server infers the pronoun's referent based on the context and inserts it in place of the pronoun. The input is parsed data, and the output is text with pronouns resolved. Specifically, it uses syntactic analysis and core reference analysis to identify the entity to which the pronoun refers, and then substitutes the identified entity name for the pronoun. This process improves the clarity of the text.

[1125] Step 5:

[1126] The server replaces homonyms with synonyms that fit the context. The input is text with pronouns resolved, and the output is text with homonyms appropriately replaced. Specifically, it references a thesaurus such as WordNet and performs context analysis to select the appropriate word. For example, it determines whether the word "kami" means "god" or "paper" depending on the context, and replaces it with the appropriate word.

[1127] Step 6:

[1128] The server reconstructs the order of verbs, objects, and subjects based on the results of the syntactic analysis. The input is text with homonyms replaced, and the output is a syntactically optimized sentence. Specifically, it uses phrase structure information from an NLP library to reorder the words into subject-verb-object. This process makes the sentence easier for the listener to understand.

[1129] Step 7:

[1130] The server uses an emotion engine to analyze and adjust the user's emotional state. The input is the user's voice input and facial expression data, and the output is text that has been adjusted based on the user's emotional information. Specifically, it uses a voice recognition API and facial expression recognition SDK to analyze the user's current emotional state and adjusts the text based on that information. For example, if the user is tired, the text is changed to a simpler, more friendly expression.

[1131] Step 8:

[1132] The server recursively executes each of the above processing steps to refine the sentence. The input is the sentence refined by the emotion engine, and the output is a further optimized sentence. Specifically, it performs syntactic analysis and sentiment analysis again to evaluate whether additional improvements are possible. If necessary, it performs pronoun resolution, homonym processing, and structural adjustment again.

[1133] Step 9:

[1134] The server saves the final adjusted text in a storage device and sends it to the terminal. The input is the final adjusted text, and the output is the saved data and the text sent to the terminal. Specifically, the text is saved in a database and data is sent to the terminal using a REST API or WebSocket.

[1135] Step 10:

[1136] The device passes the received text data to a screen reader API (e.g. TalkBack, VoiceOver), which converts it into speech and plays it back. The input is text data received from the server, and the output is speech data. Specifically, the screen reader API analyzes the received text and converts it into speech using a speech synthesis engine. The user listens to this speech to understand the information.

[1137] Through the above processing steps, the present system provides information in a voice format that is easy for users to understand and takes into consideration their emotions.

[1138] (Application example 2)

[1139] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1140] There is a huge amount of information on the Internet, and it is not easy to quickly and accurately understand the necessary information. Furthermore, while it is important to properly understand product details, especially in virtual stores, there are limitations to understanding information from text alone. Furthermore, there is a lack of information provided that reflects the user's emotional state, making it difficult to provide an effective user experience. To solve these issues, a system is needed that adjusts text content based on emotions and delivers it in audio format.

[1141] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1142] In this invention, the server includes means for retrieving sentences from web pages, means for inferring the source of a pronoun based on the context and inserting it in place of the pronoun, means for replacing homonyms with synonyms that fit the context, means for adjusting the order of verbs, objects, and subjects, means for recursively repeating the above to improve the quality of the sentences, means for storing and outputting the converted sentences, means for recognizing the user's emotions and adjusting the sentences based on the emotions, and means for outputting the sentences to the user in audio format. This allows the user to accurately understand detailed information about products in a virtual store and to receive appropriate information according to their emotional state.

[1143] A "means of extracting text from a web page" is a technique or method for collecting text data from a specific website on the Internet.

[1144] "Means of inferring the source of a pronoun based on the context and inserting it in place of the pronoun" refers to techniques or methods for identifying the specific noun referred to by a pronoun used in a sentence based on context analysis and replacing the pronoun with the specific noun.

[1145] A "method of replacing homonyms with context-appropriate synonyms" is a technique or method that uses context analysis to replace homonyms in a sentence with words that have the appropriate meaning.

[1146] "Methods for adjusting the order of verbs, objects, and subjects" are techniques or methods for reanalyzing sentence structure and rearranging the order of verbs, objects, and subjects to make them easier for listeners or readers to understand.

[1147] "Means for improving the quality of writing by recursively repeating" refers to techniques and methods for improving the quality of writing by repeating each of the above processes multiple times.

[1148] "Means for storing and outputting converted text" refers to the technology or method for storing text data after processing and providing it to an external party as needed.

[1149] "Means for recognizing a user's emotions and adjusting text based on those emotions" refers to technologies and methods for analyzing voice input and facial expression data to identify a user's emotional state and adjusting text in accordance with that state.

[1150] "Means for outputting text to a user in audio format" refers to techniques or methods for converting text data into audio and providing it to a user.

[1151] The present invention is a system for taking text from a web page and converting it into a user-friendly speech format. The system is implemented through the following steps:

[1152] Hardware and software used

[1153] The server uses the following key software libraries and technologies:

[1154] Requests library: An HTTP client for retrieving HTML data from web pages.

[1155] BeautifulSoup Library: A library for parsing HTML data and extracting text data.

[1156] NLTK (Natural Language Processing Library): A library for dividing text into sentences and words and performing syntactic and contextual analysis.

[1157] WordNet (Thesaurus): A dictionary database for replacing homonyms with context-appropriate synonyms.

[1158] Emotion Recognition Library: A library for collecting and analyzing a user's voice and facial expression data to identify their emotional state.

[1159] Pyttsx3 Library: A speech synthesis library for converting text data into speech and playing it back to the user.

[1160] System configuration

[1161] The system is configured as follows:

[1162] 1. Text Acquisition and Initial Analysis: The server acquires text data from a web page via an HTTP request, then extracts the text data using an HTML parser and preprocesses it using a natural language processing library.

[1163] 2. Pronoun resolution: The server detects pronouns in the sentence and uses context and syntactic analysis algorithms to infer their origin and insert them in place of the pronoun.

[1164] 3. Homophone Processing: The server then detects homonyms and replaces them with appropriate synonyms that fit the context by consulting a thesaurus such as WordNet.

[1165] 4. Sentence structure adjustment: The server parses each sentence and reframes the order of verb, object, and subject to make it easier for the listener to understand.

[1166] 5. Using the emotion engine: The server uses the emotion engine to recognize the user's emotions. Specifically, it analyzes the user's voice input and facial expression data to identify the user's current emotional state. Based on this, it adjusts the sentence.

[1167] 6. Audio playback: The user device passes the received text data to the screen reader, which converts it into audio and plays it back. The user receives this audio through their ears and can effectively understand the information.

[1168] Specific examples

[1169] For example, product explanations in a virtual store are carried out as follows.

[1170] Example

[1171] 1. When a user looks at a particular product, the smart glasses or head-mounted display retrieves detailed information about that product from a web page.

[1172] 2. The server analyzes the text data and processes pronouns and homonyms.

[1173] 3. Recognize the user's emotional state and adjust the sentences based on the emotion.

[1174] 4. The optimized text data is synthesized into speech and played back to the user.

[1175] Prompt Sentence Examples

[1176] Take a sentence from a web page, format it to be user-friendly, and then use an emotion engine to adapt it to the user's emotions. For example, if the user is tired, use a concise and relaxing phrase. Optimize the following sentence for speech: 'This product is made of high-quality materials and is very durable, but I'm a little worried because he's never used it before.'

[1177] This allows the user to effectively understand the information in the virtual store and receive appropriate information according to their emotions.

[1178] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1179] Step 1:

[1180] The server retrieves text data from a web page. The server first sends an HTTP request to a specific website and receives HTML formatted data. To parse the received HTML data, the BeautifulSoup library is used to extract the required text data. The input of this process is the website URL, and the output is the extracted text data.

[1181] Step 2:

[1182] The server performs an initial analysis of the extracted text data using a natural language processing library (such as NLTK). First, the text is divided into sentences and words. Next, syntactic analysis is performed to analyze the dependencies between sentences. The input of this process is the extracted text data, and the output is the divided sentences and words, as well as information about their dependencies.

[1183] Step 3:

[1184] The server detects pronouns in a sentence and infers their origin based on the context. It uses syntactic and contextual analysis algorithms to determine what the pronoun specifically refers to and inserts the actual noun in its place. The input to this process is the parsed result, and the output is the text with the pronoun replaced.

[1185] Step 4:

[1186] The server detects homonyms and replaces them with appropriate synonyms based on the context, referencing a thesaurus such as WordNet. It checks the meaning of words in a sentence based on the context and converts them to appropriate synonyms. The input to this process is the text with pronouns replaced, and the output is the text with homonyms replaced with appropriate synonyms.

[1187] Step 5:

[1188] The server parses each sentence and reconstructs the order of verbs, objects, and subjects, adjusting the order of the sentences to make them easier for the listener to understand. The input to this process is the text with homonyms replaced, and the output is the reconstructed text.

[1189] Step 6:

[1190] The server uses an emotion engine to recognize the user's emotions. It collects voice input and facial expression data from the user's device and analyzes them using an emotion recognition model. Based on the acquired emotional state information, the system adjusts the text to reflect that emotion. The input for this process is the user's voice and facial expression data, and the output is text adjusted based on the user's emotional state.

[1191] Step 7:

[1192] The server then converts the converted and adjusted text data into audio format using a speech synthesis library (such as Pyttsx3). The generated audio file is then sent to the user's device, which then plays the audio using a screen reader. The input of this process is the adjusted text data, and the output is the generated audio file.

[1193] Step 8:

[1194] Users can effectively understand information by hearing the audio played from the device. In the context of a virtual store, users can accurately understand product details and make appropriate purchasing decisions.

[1195] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1196] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1197] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1198] [Fourth embodiment]

[1199] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1200] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1201] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1202] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1203] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1204] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1205] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1206] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1207] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1208] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1209] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1210] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1211] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1212] The system of the present invention operates by allowing a server to take text from a web page and convert it into a user-friendly audio format. This process is comprised of multiple steps, each of which performs a specific operation.

[1213] About program processing

[1214] 1. Obtaining text (server)

[1215] The server first retrieves the text from a particular web page using an HTTP request, downloading the HTML data for the page, and then uses an HTML parser to extract the text data.

[1216] 2. Preprocessing (Server)

[1217] The extracted text data is pre-processed using an NLP (Natural Language Processing) library. Specifically, the text is divided into sentences and words, and syntactic analysis is performed to analyze dependencies. This allows for a detailed understanding of how the text is structured.

[1218] 3. Pronoun Resolution (Server)

[1219] The server then detects pronouns in the sentence and infers what each pronoun refers to, using syntactic and contextual analysis, and inserts the inferred reference in place of the pronoun, making the sentence clearer.

[1220] 4. Homophone Processing (Server)

[1221] The server then detects words containing homonyms, which are then replaced with appropriate synonyms based on the context, reducing the chance of misinterpretation.

[1222] 5. Adjusting the text structure (server)

[1223] The server then parses each sentence and adjusts the order of verbs, objects, and subjects to make the sentence more meaningful.

[1224] 6. Recursive Improvement (Server)

[1225] These processes are repeated recursively to refine the text, with each cycle adjusting the text until it feels more natural and understandable.

[1226] 7. Generating and outputting the final converted text (server)

[1227] The final converted text is stored in a storage device and prepared in a suitable format. The server sends this data to the terminal.

[1228] 8. Audio playback (device)

[1229] The device passes the received text data to the screen reader and plays it aloud to the user, allowing the user to receive text information audibly.

[1230] Specific examples

[1231] Original text:

[1232] Whether the plan will be successful or not depends on his efforts.

[1233] After conversion:

[1234] 1. The server retrieves the text from the web page.

[1235] 2. Perform syntactic analysis and understand the context.

[1236] 3. Assume that "he" refers to "Mr. Tanaka" (hypothetical) and replace the pronoun.

[1237] 4. Ensure there are no homonyms.

[1238] 5. Adjust the order of verbs, objects, and subjects while maintaining the sentence structure.

[1239] 6. Continue processing recursively until the sentence is determined to be clear.

[1240] 7. Finally, say, "Whether the plan will be successful or not depends on Tanaka's efforts."

[1241] 8. Send the converted text to the device and play it with a screen reader.

[1242] This allows the user to clearly hear and understand that "Whether the plan will be successful or not depends on Tanaka's efforts."

[1243] The processing flow will be explained below.

[1244] Step 1:

[1245] The server retrieves text data from a specific web page by sending an HTTP request and downloading the HTML data of the target web page.

[1246] Step 2:

[1247] The server parses the HTML data and extracts the text data, using a library such as BeautifulSoup.

[1248] Step 3:

[1249] The server splits the extracted text into sentences and words, and then uses an NLP (Natural Language Processing) library to tokenize it.

[1250] Step 4:

[1251] The server parses the sentence, creating a dependency tree of the sentence and analyzing the internal structure of the sentence.

[1252] Step 5:

[1253] The server detects pronouns and infers the pronoun's source based on the context. It uses syntactic and contextual analysis algorithms to identify the pronoun's source.

[1254] Step 6:

[1255] The server will insert the inferred reference in place of the pronoun, making the sentence clearer.

[1256] Step 7:

[1257] The server detects homonyms using a word list containing homonyms and contextual analysis.

[1258] Step 8:

[1259] The server replaces homonyms with appropriate synonyms based on the context, and selects synonyms by referencing dictionaries such as WordNet.

[1260] Step 9:

[1261] The server reconstructs each sentence, examining the order of verb, object, and subject, and rearranging them to make them easier for the listener to understand.

[1262] Step 10:

[1263] The server repeats the above process recursively, refining the sentence so that each cycle further improves the results.

[1264] Step 11:

[1265] The server saves the final converted text to a text file and outputs the saved data in an appropriate format.

[1266] Step 12:

[1267] The server sends the converted text data to the terminal, using an appropriate format such as JSON.

[1268] Step 13:

[1269] The device passes the received text data to the screen reader, which converts it into speech and prepares to play it back.

[1270] Step 14:

[1271] The user listens to the audio played by the screen reader, audibly receiving and understanding the improved text.

[1272] The above is a specific processing flow in the system of the present invention.

[1273] Example 1

[1274] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1275] Conventional technologies for converting text obtained from web pages into speech format have had many problems. In particular, it has been difficult to understand the source of pronouns, properly replace homonyms, and optimize sentence structure, making it difficult for users to accurately grasp the information. Furthermore, there has been a lack of technology for automating these processes recursively. The present invention solves these problems and provides a system that allows users to more easily obtain accurate information.

[1276] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1277] In this invention, the server includes means for acquiring text data from a web page, means for preprocessing the acquired text data using a natural language processing library, means for inferring the source of a pronoun based on the context and inserting it in place of the pronoun, means for replacing homonyms with appropriate words that match the context, means for adjusting the order of verbs, objects, and subjects based on syntactic analysis, means for recursively repeating the above to improve the quality of the text data, means for storing and outputting the converted text data, and means for playing back the converted sentence data aloud. This allows the user to obtain natural and easy-to-understand audio information.

[1278] A "web page" is a document in which information published on the Internet is written in HTML format.

[1279] "Text data" refers to data that is stored or transmitted electronically as character information.

[1280] "Natural language processing library" is a general term for software tools that allow computers to process and analyze the language that humans use on a daily basis.

[1281] "Preprocessing" refers to the preprocessing of raw data to convert it into a format that can be processed by an algorithm.

[1282] A pronoun is a word used in place of a noun to refer to a person, thing, place, etc.

[1283] A "reference" is a specific noun or situation to which a pronoun refers.

[1284] "Context" refers to the entire situation that determines the meaning of words, such as the context and background information of a sentence or utterance.

[1285] "Homonyms" are words that are pronounced the same but have different meanings.

[1286] An "appropriate word" is one chosen to fit the context and convey the correct meaning.

[1287] "Syntax analysis" is the process of analyzing the grammatical structure of a sentence and clarifying the relationships between each element.

[1288] A "verb" is a part of speech that primarily expresses an action or state.

[1289] An "object" is a noun or noun phrase that is the target of an action.

[1290] A "subject" is a noun or noun phrase that indicates the agent of an action or state.

[1291] "Recursive" is an adjective that refers to the repeated application of the same process.

[1292] "Level of proficiency" is a measure of maturity and completeness of writing, technique, etc.

[1293] "Memory" refers to the long-term storage of data.

[1294] "Output" refers to providing processed data to the outside.

[1295] "Playback by voice" refers to converting text data into machine voice and reading it aloud.

[1296] The system of the present invention operates such that a server takes text data from a web page and converts it into an audio format that is easy for a user to understand.

[1297] 1. Obtaining text (server)

[1298] The server first retrieves text data from a specific web page. To do this, it uses an HTTP request to download the HTML data of the web page. The specific software used is the requests library, and the BeautifulSoup library is used to extract text from the retrieved HTML data.

[1299] 2. Preprocessing (Server)

[1300] The server preprocesses the acquired text data using a natural language processing library, such as spaCy, to split the text into sentences and words and perform syntactic analysis, which reveals the relationships between each part of the text.

[1301] 3. Pronoun Resolution (Server)

[1302] The server detects pronouns in a sentence and infers what each pronoun refers to. Using syntactic and contextual analysis, it inserts the inferred referent in place of the pronoun. As a concrete example, we use the neuralcoref library.

[1303] 4. Homophone Processing (Server)

[1304] The server detects homonyms in a sentence and replaces them with the appropriate word depending on the context, for example, by selecting the appropriate word from a list of homonyms based on the context, reducing the chance of misunderstanding.

[1305] 5. Adjusting the text structure (server)

[1306] Based on the syntactic analysis, the server adjusts the order of the subject, verb, and object, which makes the text more meaningful.

[1307] 6. Recursive Improvement (Server)

[1308] The process is repeated recursively to refine the text data, with each processing cycle adjusting the sentences until they become more natural and understandable.

[1309] 7. Generating and outputting the final converted text (server)

[1310] The server finally stores the converted text data in a storage device, prepares it in an appropriate format, and sends the converted text data to the terminal.

[1311] 8. Audio playback (device)

[1312] The device passes the text data received from the server to the screen reader and plays it aloud to the user. For example, by using the pyttsx3 library, the user can receive text information aurally.

[1313] Specific examples

[1314] Original: Whether the plan succeeds or not depends on his efforts.

[1315] The system retrieves a sentence from a web page and uses an NLP library to parse it and understand the context. It infers that the pronoun "he" refers to "Tanaka-san," and replaces the pronoun. It then checks for the absence of homonyms and adjusts the order of the verb, object, and subject while maintaining the sentence structure. It continues the recursive process, eventually arriving at the sentence, "Whether the plan will be successful or not depends on Tanaka-san's efforts." The converted sentence is sent to the device and played back by a screen reader. The user can clearly hear and understand that it is "Whether the plan will be successful or not depends on Tanaka-san's efforts."

[1316] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1317] Step 1:

[1318] The server retrieves text data from a specific web page. Specifically, it sends an HTTP request to download the HTML data of the web page. The input is the URL of the web page, and the output is the retrieved HTML data. The server then extracts the text data from the retrieved HTML data using the BeautifulSoup library.

[1319] Step 2:

[1320] The server preprocesses the retrieved text data using a natural language processing library. Specifically, it uses the spaCy library to split the text into sentences and words and perform syntactic analysis. The input is the extracted text data, and the output is a collection of split and parsed sentences.

[1321] Step 3:

[1322] The server detects pronouns in a sentence and infers what each pronoun refers to. The technology used utilizes the neuralcoref library. The input is a collection of parsed sentences, and the output is a sentence in which pronouns have been replaced with their referents. Syntactic and contextual analysis are used for inference.

[1323] Step 4:

[1324] The server detects homonyms and replaces them with the appropriate word depending on the context. It selects the appropriate word from a list of homonyms based on the context. The input is a sentence with pronouns resolved, and the output is a sentence with homonyms replaced with the appropriate word.

[1325] Step 5:

[1326] The server adjusts the order of subjects, verbs, and objects based on syntactic analysis. It reconstructs the relationships between elements in the text and places them in a natural grammatical order. The input is a sentence with homonyms processed, and the output is the adjusted sentence.

[1327] Step 6:

[1328] The server repeats these processes recursively to refine the text data, making adjustments with each processing cycle until the text becomes more natural and easier to understand. The input is the text that has undergone this series of processes, and the output is the final refined text.

[1329] Step 7:

[1330] The server stores the final converted text in a storage device and prepares it in an appropriate format. The data stored in the storage device is sent to the user's terminal. The input is the final edited text, and the output is the data stored and sent to the terminal.

[1331] Step 8:

[1332] The device passes the text data received from the server to a screen reader library (e.g., pyttsx3) and plays it aloud to the user. The input at this time is the received text data, and the output is audio playback for the user. This allows the user to receive text information aurally.

[1333] (Application example 1)

[1334] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1335] In conventional virtual stores, users had few easy ways to obtain product information, and had to rely solely on visual access. As a result, it was not possible to provide appropriate services for visually impaired people or users who have difficulty grasping a large amount of information at once. Furthermore, users had limited ways to quickly and easily access information that interested them, resulting in an incomplete in-store experience.

[1336] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1337] In this invention, the server includes means for retrieving sentences from web pages, means for inferring the source of a pronoun based on the context and inserting it in place of the pronoun, means for replacing homonyms with synonyms that fit the context, means for adjusting the order of verbs, objects, and subjects, means for recursively repeating the above to improve the quality of the sentences, means for storing and outputting the converted sentences, means for retrieving product information in a virtual space and converting the sentences into voice, and means for providing real-time voice guidance of the retrieved product information. This enables real-time voice guidance of product information in a virtual store, making it possible to provide an environment in which all users, including visually impaired people, can easily access information.

[1338] A "web page" is a document in HTML format that can be viewed on the Internet.

[1339] The "means for obtaining text" is a means for extracting text data from a specific web page.

[1340] "Methods of inferring the pronoun's referent based on the context and inserting it in place of the pronoun" refers to methods of identifying the object referred to by a pronoun in a sentence and inserting a specific noun in its place.

[1341] "Means for replacing homonyms with context-appropriate synonyms" refers to means for converting homonyms into synonyms with accurate meanings based on the context.

[1342] "Means for adjusting the order of verbs, objects, and subjects" are means for rearranging the order of verbs, objects, and subjects so that the meaning of a sentence becomes more comprehensible.

[1343] "Means for recursively refining sentences" refers to repeating the process multiple times until the sentences become natural and easy to understand.

[1344] The "means for storing and outputting converted text" refers to a means for saving converted text and outputting it as needed.

[1345] "Means for acquiring product information in a virtual space and converting text into audio" refers to means for acquiring text information about products displayed in a virtual space and converting it into audio.

[1346] The "means for providing voice guidance of acquired product information in real time" is a means for instantly providing information about the product being viewed by the user by voice.

[1347] This invention relates to a system that provides audio guidance of text retrieved from a web page within a virtual store. This system consists of a server and terminals and is designed to enable users to quickly and easily obtain product information within the virtual space.

[1348] System Overview

[1349] The server performs a series of processes to retrieve text from a web page and convert it into a format that is easy for the user to understand. The server includes the following means:

[1350] 1. How to get text from a web page

[1351] 2. A method of guessing the pronoun's source and inserting it in place of the pronoun

[1352] 3. A method for replacing homonyms with context-appropriate synonyms

[1353] 4. A means of adjusting the order of verbs, objects, and subjects

[1354] 5. Repeat the above steps recursively to improve your writing

[1355] 6. A means of storing and outputting the converted text

[1356] 7. A means of obtaining product information in virtual space and converting text into speech

[1357] 8. A means of providing real-time voice guidance of acquired product information

[1358] Program processing

[1359] The server downloads HTML data from a web page using requests and BeautifulSoup, extracts the text, and then uses natural language processing libraries such as spacy to perform syntactic and contextual analysis of the text, resolving pronouns, processing homonyms, and adjusting sentence structure. The final adjusted text is then saved and sent to the device in the appropriate format.

[1360] The device converts the received text into speech using speech synthesis software such as pyttsx3 and provides real-time guidance to the user, allowing the user to easily understand product information by listening to it in the virtual space.

[1361] Hardware and software used

[1362] Server: The requests library for sending HTTP requests, BeautifulSoup for HTML parsing, and spacy for natural language processing.

[1363] Terminal: pyttsx3 for speech synthesis

[1364] Specific examples

[1365] For example, suppose a user is viewing a product page in a virtual store, and the product description is as follows:

[1366] "This product is made from high-quality leather. The most popular color is red."

[1367] In this case, the system behaves as follows:

[1368] The server retrieves the above sentence from the web page and converts it into an intelligible format through natural language processing. The device then converts the sentence into speech, guiding the user in real time: "This product is made of high-quality leather. The most popular color is red."

[1369] Prompt Sentence Examples

[1370] "When a customer is browsing products in a virtual store, we want them to be provided with a voice guide that retrieves product descriptions from a product database. For example, if a customer using smart glasses is viewing a product page for a red leather product, we want to be provided with a real-time guide that says, 'This product is made with high-quality leather. Red is a particularly popular color.'"

[1371] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1372] Step 1:

[1373] The server uses requests to send an HTTP request from the specified web page URL and download the HTML data. The input is the URL and the output is the raw data in HTML format, which is then passed to the next processing step.

[1374] Step 2:

[1375] The server uses the BeautifulSoup library to parse the HTML data and extract the necessary text information.< / url:> Extracts text from tags, etc. The input is HTML data, and the output is extracted text data.

[1376] Step 3:

[1377] The server uses the spacy library to preprocess the extracted text data for natural language processing. It splits the text into sentences and words and performs syntactic analysis. The input is the text data, and the output is a parsed syntax tree.

[1378] Step 4:

[1379] The server detects pronouns in text data, infers the referent to which the pronoun refers based on the context, and inserts the inferred referent in place of the pronoun. The input is a syntax tree and pronoun data, and the output is the text data with the pronoun replaced.

[1380] Step 5:

[1381] The server detects homonyms and replaces them with appropriate synonyms based on the context. It uses a homonym dictionary to select words that fit the context. The input is text data with pronouns replaced, and the output is text data with homonyms appropriately replaced.

[1382] Step 6:

[1383] The server adjusts the order of verbs, objects, and subjects in the text, making the sentences more natural and easier to understand. The input is text data with homonyms replaced, and the output is text data with adjusted syntax.

[1384] Step 7:

[1385] The server repeats these processes recursively, adjusting each cycle until the sentence becomes more natural and easier to understand. The input is the text data obtained in the previous step, and the output is the optimized text data.

[1386] Step 8:

[1387] The server stores the final converted text data in a storage device and sends it to the terminal in an appropriate format. The input is optimized text data, and the output is data ready to be sent to the terminal.

[1388] Step 9:

[1389] The terminal uses the pyttsx3 library to convert the received text data into speech and provide real-time guidance to the user. The input is the text data sent from the server, and the output is the speech guidance.

[1390] Examples of specific examples and prompts

[1391] Example prompt sentence:

[1392] "When a customer is browsing products in a virtual store, we want them to be provided with a voice guide that retrieves product descriptions from a product database. For example, if a customer using smart glasses is viewing a product page for a red leather product, we want to be provided with a real-time guide that says, 'This product is made with high-quality leather. Red is a particularly popular color.'"

[1393] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1394] The system of the present invention is configured to retrieve text from a web page and convert it into a speech format that is easy for users to understand. Furthermore, by combining it with an emotion engine, the system can adjust the text according to the user's emotions, providing a better user experience.

[1395] About program processing

[1396] Text acquisition and initial analysis (server)

[1397] The server retrieves text data from a specific web page via an HTTP request. It then uses an HTML parser to extract the text data and preprocesses it with an NLP (Natural Language Processing) library. This splits the text into sentences and words, and analyzes dependencies through syntactic analysis.

[1398] Pronoun Resolution (Server)

[1399] The server detects pronouns in a sentence and uses syntactic and contextual analysis algorithms to infer their referents, which are then inserted in place of the pronouns to improve the clarity of the sentence.

[1400] Homophone Processing (Server)

[1401] The server then detects homonyms and replaces them with appropriate synonyms based on the context, referencing a thesaurus such as WordNet, resulting in a sentence with less room for misinterpretation.

[1402] Text structure adjustment (server)

[1403] The server parses each sentence and reframes the order of verb, object, and subject to make it easier for the listener to understand.

[1404] Using the Emotion Engine (Server)

[1405] The server uses an emotion engine to recognize the user's emotions. Specifically, it analyzes the user's voice input and facial expression data to identify the user's current emotional state, and adjusts the text accordingly.

[1406] Recursive Improvements (Server)

[1407] The above processing steps are repeated recursively to refine the sentence. By incorporating an emotion engine into this process, optimal sentences that reflect the user's emotions are generated.

[1408] Generation and output of final converted text (server)

[1409] The completed text data is stored in a storage device and prepared in an appropriate format. The server sends this data to the terminal.

[1410] Playback via audio (device)

[1411] The device passes the received text data to the screen reader, which converts it into speech and plays it back, allowing the user to receive it aurally and understand the information effectively.

[1412] Specific examples

[1413] Original text:

[1414] Whether the plan will be successful or not depends on his efforts.

[1415] Converted example:

[1416] 1. The server retrieves the text from the web page.

[1417] 2. Perform syntactic analysis and understand the context.

[1418] 3. Assuming that "he" is "Tanaka-san" (hypothesis), replace the pronoun.

[1419] 4. Ensure there are no homonyms.

[1420] 5. Adjust the sentence structure and optimize the order of verbs, objects, and subjects.

[1421] 6. The emotion engine recognizes that the user is tired and adjusts the expression to be concise and reassuring.

[1422] 7. Continue recursively to optimize the text.

[1423] 8. Finally, say, "Whether the plan will be successful or not depends on Tanaka's efforts."

[1424] 9. Send the converted text to the device and play it with a screen reader.

[1425] By listening to this optimized voice, users can quickly and accurately understand information. In particular, by utilizing the emotion engine, sentences are provided that take into consideration the user's emotional state, resulting in a better user experience.

[1426] The processing flow will be explained below.

[1427] Step 1:

[1428] The server retrieves text data from a specific web page by issuing an HTTP request and downloading the HTML content of the web page.

[1429] Step 2:

[1430] The server parses the HTML data and uses an HTML parser library such as BeautifulSoup to extract the required text data.

[1431] Step 3:

[1432] The server splits the extracted text data into sentences, and uses an NLP library to tokenize the sentences and identify their components.

[1433] Step 4:

[1434] The server detects pronouns in the sentence using grammatical analysis to identify their positions.

[1435] Step 5:

[1436] The server infers the pronoun's referent, using syntactic and contextual analysis algorithms to identify the specific noun or noun phrase to which the pronoun refers.

[1437] Step 6:

[1438] The server inserts the inferred reference in place of the pronoun, and clarifies the sentence by removing the pronoun and replacing it with a concrete noun.

[1439] Step 7:

[1440] The server detects homonyms, consults a homonym dictionary, and lists target words in the sentence.

[1441] Step 8:

[1442] The server replaces homonyms with appropriate synonyms depending on the context. It analyzes the context and selects the synonym with the most appropriate meaning.

[1443] Step 9:

[1444] The server analyzes the order of verbs, objects, and subjects in each sentence and rearranges them in an order that is easy for the listener to understand. It also uses syntactic analysis to understand the structure of the sentence and rearranges the order as necessary.

[1445] Step 10:

[1446] The server uses an emotion engine to recognize the user's emotions. It collects the user's voice input and facial expression data and uses an emotion analysis model to identify the user's emotional state.

[1447] Step 11:

[1448] The server adjusts the sentences based on the user's emotions, for example, if the user is tired, it adapts the sentences to a concise and reassuring format.

[1449] Step 12:

[1450] The server recursively repeats each of the above processing steps to refine the sentence, repeating the process until the sentence is optimized.

[1451] Step 13:

[1452] The server then stores the converted text data in a storage device and converts it into an appropriate format for output, such as a text file or JSON format.

[1453] Step 14:

[1454] The server sends the converted text data to the terminal, which then sends the data to the terminal via the network.

[1455] Step 15:

[1456] The device passes the received text data to the screen reader, which converts the text into speech and prepares it for playback.

[1457] Step 16:

[1458] The user listens to the audio played by the screen reader, receives the improved text aurally, and can understand the information quickly and accurately.

[1459] Example 2

[1460] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1461] Conventional text conversion systems often use text extracted from web pages as is, rarely converting it into a format that is easy for users to understand. Furthermore, pronoun references can be ambiguous, and interpretations of homonyms can be misleading. Furthermore, the text structure can be complex, and the system does not respond to the user's emotional state, resulting in a poor user experience. Therefore, a text conversion system that takes into account the clarity of the entire text and the user's emotions is needed.

[1462] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1463] In this invention, the server includes means for retrieving sentences from web pages, means for inferring the source of a pronoun based on the context and inserting it in place of the pronoun, means for replacing homonyms with synonyms that match the context, means for analyzing the user's emotional state and adjusting the sentences according to the emotion, means for adjusting the order of verbs, objects, and subjects, means for recursively repeating the above to improve the quality of the sentences, and means for storing and outputting the converted sentences. This makes it possible to convert sentences into ones that are easy for the user to understand, and provides sentences that correspond to the user's emotional state, thereby realizing a better user experience.

[1464] A "web page" is a type of document published on the Internet and constructed using HTML and other web technologies.

[1465] A "sentence" is text data that combines characters and symbols to form meaning.

[1466] A "pronoun" is an element of language used to refer to a particular noun or noun phrase, such as "he," "it," or "I."

[1467] "Context" refers to the context or situation in a sentence or speech that allows us to understand the meaning and intention of words.

[1468] "Homonyms" are words that are pronounced the same but have different meanings.

[1469] "Synonyms" are words that have similar meanings or concepts, but are technically different.

[1470] "Emotional state" refers to the user's current mood or psychological state, including happiness, sadness, anger, fatigue, etc.

[1471] "Syntax analysis" is a natural language processing technique that analyzes the grammatical structure of sentences and phrases and clarifies the relationships between each element.

[1472] "Recursive" refers to the process in which a process or operation is repeatedly performed based on its results.

[1473] "Degree of sophistication" refers to the degree of completion and high quality of a piece of writing or a product.

[1474] The present invention is a system for acquiring text from a web page and converting it into a speech format that is easy for users to understand. The configuration and specific processing steps of the system are described below.

[1475] System configuration

[1476] This system mainly consists of a server and a terminal. The server acquires, analyzes, and converts text, and the terminal plays the converted text in audio format.

[1477] Specific examples of hardware and software used

[1478] The server is built on a common web server platform and requires the following software:

[1479] HTML parsers (e.g. BeautifulSoup, lxml)

[1480] Natural language processing libraries (e.g. spaCy, NLTK)

[1481] Sentiment analysis SDK (e.g. Microsoft Azure's Face API, Google Cloud Speech-to-Text)

[1482] Database (e.g. MySQL, PostgreSQL)

[1483] A terminal is a device capable of playing audio and uses the following software:

[1484] Screen reader APIs (e.g., TalkBack on Android, VoiceOver on iOS)

[1485] Specific examples

[1486] Processing flow

[1487] 1. Text Acquisition and Initial Analysis

[1488] The server sends an HTTP request to the URL provided by the user to retrieve the HTML source code of the web page.

[1489] The obtained HTML source code is analyzed using an HTML parser (e.g., BeautifulSoup) to extract key text data.

[1490] The extracted text data is preprocessed using a natural language processing library (e.g., spaCy), where it is split into sentences and words and parsed syntactically.

[1491] 2. Pronoun Resolution

[1492] The server uses syntactic and contextual analysis algorithms to identify the source of a pronoun in a sentence. For example, it infers that the pronoun "he" refers to "Tanaka-san" and replaces the pronoun with a hypothetical real name.

[1493] 3. Homophone Processing

[1494] The server refers to a thesaurus such as WordNet and replaces detected homonyms with appropriate words depending on the context. For example, it determines whether "kami" means "god" or "paper" depending on the context.

[1495] 4. Adjusting sentence structure

[1496] The server reorders the sentence based on the results of the parsing. For example, it reorders the verb, object, and subject, adjusting the sentence "Whether the plan will be successful or not depends on his efforts" to "Whether the plan will be successful or not depends on Tanaka's efforts."

[1497] 5. Use of Emotion Engine

[1498] The server performs sentiment analysis to identify the user's emotional state, and adjusts the text to be more concise and friendly depending on the user's emotional state (e.g., tired).

[1499] 6. Recursive Improvement

[1500] The server recursively executes the above processing steps to refine the sentence.

[1501] 7. Generating and outputting the final converted text

[1502] The server stores the finalized text in a database and sends it to the device.

[1503] 8. Audio playback

[1504] The device uses a screen reader API (e.g., TalkBack) to convert the received text data into speech and play it back to the user.

[1505] Prompt Sentence Examples

[1506] Here are some examples of prompts that a generative AI model might use:

[1507] "Outline a system that takes specific text from a web page and converts it into a clear, understandable audio format."

[1508] This concludes the description of the preferred embodiment of the present invention. The system allows for the capture of text from a web page and converting it into an easily understandable speech format that is tailored to the user's emotional state, thereby improving the user experience and effectively conveying information.

[1509] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1510] Step 1:

[1511] The server sends an HTTP request to the URL of a web page provided by the user. The input is the URL provided by the user, and the output is the HTML source code of the web page. Specifically, the server makes an HTTP GET request and receives HTML data in response. This HTML data is used in the next analysis step.

[1512] Step 2:

[1513] The server parses the retrieved HTML source code using an HTML parser (e.g., BeautifulSoup, lxml) to extract text data. The input is the HTML source code, and the output is the extracted text data. Specifically, the HTML parser parses the document tree to identify and extract useful text content. This data is pure text with unnecessary HTML tags removed.

[1514] Step 3:

[1515] The server preprocesses the extracted text data using a natural language processing library (e.g., spaCy, NLTK). The input is the extracted text data, and the output is tokenized and parsed data. Specifically, the text is divided into sentences and words, and parsed to identify the relationships between grammatical elements (nouns, verbs, adjectives, etc.). The results of this analysis make it easier to manipulate the text in subsequent processing steps.

[1516] Step 4:

[1517] The server infers the pronoun's referent based on the context and inserts it in place of the pronoun. The input is parsed data, and the output is text with pronouns resolved. Specifically, it uses syntactic analysis and core reference analysis to identify the entity to which the pronoun refers, and then substitutes the identified entity name for the pronoun. This process improves the clarity of the text.

[1518] Step 5:

[1519] The server replaces homonyms with synonyms that fit the context. The input is text with pronouns resolved, and the output is text with homonyms appropriately replaced. Specifically, it references a thesaurus such as WordNet and performs context analysis to select the appropriate word. For example, it determines whether the word "kami" means "god" or "paper" depending on the context, and replaces it with the appropriate word.

[1520] Step 6:

[1521] The server reconstructs the order of verbs, objects, and subjects based on the results of the syntactic analysis. The input is text with homonyms replaced, and the output is a syntactically optimized sentence. Specifically, it uses phrase structure information from an NLP library to reorder the words into subject-verb-object. This process makes the sentence easier for the listener to understand.

[1522] Step 7:

[1523] The server uses an emotion engine to analyze and adjust the user's emotional state. The input is the user's voice input and facial expression data, and the output is text that has been adjusted based on the user's emotional information. Specifically, it uses a voice recognition API and facial expression recognition SDK to analyze the user's current emotional state and adjusts the text based on that information. For example, if the user is tired, the text is changed to a simpler, more friendly expression.

[1524] Step 8:

[1525] The server recursively executes each of the above processing steps to refine the sentence. The input is the sentence refined by the emotion engine, and the output is a further optimized sentence. Specifically, it performs syntactic analysis and sentiment analysis again to evaluate whether additional improvements are possible. If necessary, it performs pronoun resolution, homonym processing, and structural adjustment again.

[1526] Step 9:

[1527] The server saves the final adjusted text in a storage device and sends it to the terminal. The input is the final adjusted text, and the output is the saved data and the text sent to the terminal. Specifically, the text is saved in a database and data is sent to the terminal using a REST API or WebSocket.

[1528] Step 10:

[1529] The device passes the received text data to a screen reader API (e.g. TalkBack, VoiceOver), which converts it into speech and plays it back. The input is text data received from the server, and the output is speech data. Specifically, the screen reader API analyzes the received text and converts it into speech using a speech synthesis engine. The user listens to this speech to understand the information.

[1530] Through the above processing steps, the present system provides information in a voice format that is easy for users to understand and takes into consideration their emotions.

[1531] (Application example 2)

[1532] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1533] There is a huge amount of information on the Internet, and it is not easy to quickly and accurately understand the necessary information. Furthermore, while it is important to properly understand product details, especially in virtual stores, there are limitations to understanding information from text alone. Furthermore, there is a lack of information provided that reflects the user's emotional state, making it difficult to provide an effective user experience. To solve these issues, a system is needed that adjusts text content based on emotions and delivers it in audio format.

[1534] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1535] In this invention, the server includes means for retrieving sentences from web pages, means for inferring the source of a pronoun based on the context and inserting it in place of the pronoun, means for replacing homonyms with synonyms that fit the context, means for adjusting the order of verbs, objects, and subjects, means for recursively repeating the above to improve the quality of the sentences, means for storing and outputting the converted sentences, means for recognizing the user's emotions and adjusting the sentences based on the emotions, and means for outputting the sentences to the user in audio format. This allows the user to accurately understand detailed information about products in a virtual store and to receive appropriate information according to their emotional state.

[1536] A "means of extracting text from a web page" is a technique or method for collecting text data from a specific website on the Internet.

[1537] "Means of inferring the source of a pronoun based on the context and inserting it in place of the pronoun" refers to techniques or methods for identifying the specific noun referred to by a pronoun used in a sentence based on context analysis and replacing the pronoun with the specific noun.

[1538] A "method of replacing homonyms with context-appropriate synonyms" is a technique or method that uses context analysis to replace homonyms in a sentence with words that have the appropriate meaning.

[1539] "Methods for adjusting the order of verbs, objects, and subjects" are techniques or methods for reanalyzing sentence structure and rearranging the order of verbs, objects, and subjects to make them easier for listeners or readers to understand.

[1540] "Means for improving the quality of writing by recursively repeating" refers to techniques and methods for improving the quality of writing by repeating each of the above processes multiple times.

[1541] "Means for storing and outputting converted text" refers to the technology or method for storing text data after processing and providing it to an external party as needed.

[1542] "Means for recognizing a user's emotions and adjusting text based on those emotions" refers to technologies and methods for analyzing voice input and facial expression data to identify a user's emotional state and adjusting text in accordance with that state.

[1543] "Means for outputting text to a user in audio format" refers to techniques or methods for converting text data into audio and providing it to a user.

[1544] The present invention is a system for taking text from a web page and converting it into a user-friendly speech format. The system is implemented through the following steps:

[1545] Hardware and software used

[1546] The server uses the following key software libraries and technologies:

[1547] Requests library: An HTTP client for retrieving HTML data from web pages.

[1548] BeautifulSoup Library: A library for parsing HTML data and extracting text data.

[1549] NLTK (Natural Language Processing Library): A library for dividing text into sentences and words and performing syntactic and contextual analysis.

[1550] WordNet (Thesaurus): A dictionary database for replacing homonyms with context-appropriate synonyms.

[1551] Emotion Recognition Library: A library for collecting and analyzing a user's voice and facial expression data to identify their emotional state.

[1552] Pyttsx3 Library: A speech synthesis library for converting text data into speech and playing it back to the user.

[1553] System configuration

[1554] The system is configured as follows:

[1555] 1. Text Acquisition and Initial Analysis: The server acquires text data from a web page via an HTTP request, then extracts the text data using an HTML parser and preprocesses it using a natural language processing library.

[1556] 2. Pronoun resolution: The server detects pronouns in the sentence and uses context and syntactic analysis algorithms to infer their origin and insert them in place of the pronoun.

[1557] 3. Homophone Processing: The server then detects homonyms and replaces them with appropriate synonyms that fit the context by consulting a thesaurus such as WordNet.

[1558] 4. Sentence structure adjustment: The server parses each sentence and reframes the order of verb, object, and subject to make it easier for the listener to understand.

[1559] 5. Using the emotion engine: The server uses the emotion engine to recognize the user's emotions. Specifically, it analyzes the user's voice input and facial expression data to identify the user's current emotional state. Based on this, it adjusts the sentence.

[1560] 6. Audio playback: The user device passes the received text data to the screen reader, which converts it into audio and plays it back. The user receives this audio through their ears and can effectively understand the information.

[1561] Specific examples

[1562] For example, product explanations in a virtual store are carried out as follows.

[1563] Example

[1564] 1. When a user looks at a particular product, the smart glasses or head-mounted display retrieves detailed information about that product from a web page.

[1565] 2. The server analyzes the text data and processes pronouns and homonyms.

[1566] 3. Recognize the user's emotional state and adjust the sentences based on the emotion.

[1567] 4. The optimized text data is synthesized into speech and played back to the user.

[1568] Prompt Sentence Examples

[1569] Take a sentence from a web page, format it to be user-friendly, and then use an emotion engine to adapt it to the user's emotions. For example, if the user is tired, use a concise and relaxing phrase. Optimize the following sentence for speech: 'This product is made of high-quality materials and is very durable, but I'm a little worried because he's never used it before.'

[1570] This allows the user to effectively understand the information in the virtual store and receive appropriate information according to their emotions.

[1571] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1572] Step 1:

[1573] The server retrieves text data from a web page. The server first sends an HTTP request to a specific website and receives HTML formatted data. To parse the received HTML data, the BeautifulSoup library is used to extract the required text data. The input of this process is the website URL, and the output is the extracted text data.

[1574] Step 2:

[1575] The server performs an initial analysis of the extracted text data using a natural language processing library (such as NLTK). First, the text is divided into sentences and words. Next, syntactic analysis is performed to analyze the dependencies between sentences. The input of this process is the extracted text data, and the output is the divided sentences and words, as well as information about their dependencies.

[1576] Step 3:

[1577] The server detects pronouns in a sentence and infers their origin based on the context. It uses syntactic and contextual analysis algorithms to determine what the pronoun specifically refers to and inserts the actual noun in its place. The input to this process is the parsed result, and the output is the text with the pronoun replaced.

[1578] Step 4:

[1579] The server detects homonyms and replaces them with appropriate synonyms based on the context, referencing a thesaurus such as WordNet. It checks the meaning of words in a sentence based on the context and converts them to appropriate synonyms. The input to this process is the text with pronouns replaced, and the output is the text with homonyms replaced with appropriate synonyms.

[1580] Step 5:

[1581] The server parses each sentence and reconstructs the order of verbs, objects, and subjects, adjusting the order of the sentences to make them easier for the listener to understand. The input to this process is the text with homonyms replaced, and the output is the reconstructed text.

[1582] Step 6:

[1583] The server uses an emotion engine to recognize the user's emotions. It collects voice input and facial expression data from the user's device and analyzes them using an emotion recognition model. Based on the acquired emotional state information, the system adjusts the text to reflect that emotion. The input for this process is the user's voice and facial expression data, and the output is text adjusted based on the user's emotional state.

[1584] Step 7:

[1585] The server then converts the converted and adjusted text data into audio format using a speech synthesis library (such as Pyttsx3). The generated audio file is then sent to the user's device, which then plays the audio using a screen reader. The input of this process is the adjusted text data, and the output is the generated audio file.

[1586] Step 8:

[1587] Users can effectively understand information by hearing the audio played from the device. In the context of a virtual store, users can accurately understand product details and make appropriate purchasing decisions.

[1588] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1589] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1590] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1591] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1592] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1593] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1594] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1595] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1596] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1597] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1598] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1599] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1600] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1601] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1602] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1603] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1604] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1605] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1606] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1607] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1608] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1609] The following is further disclosed regarding the above embodiment.

[1610] (Claim 1)

[1611] a means for retrieving text from a web page;

[1612] A means of inferring the pronoun's reference source based on the context and inserting it in place of the pronoun;

[1613] A means of replacing homonyms with context-appropriate synonyms;

[1614] a means of adjusting the order of verbs, objects, and subjects;

[1615] A method for improving the quality of writing by repeating the above recursively,

[1616] The system includes means for storing and outputting the converted text.

[1617] (Claim 2)

[1618] 10. The system of claim 1, wherein syntactic analysis is used to infer the referent of a pronoun.

[1619] (Claim 3)

[1620] 10. The system of claim 1, wherein a thesaurus is utilized when replacing homonyms.

[1621] "Example 1"

[1622] (Claim 1)

[1623] means for obtaining text data from a web page;

[1624] A means for preprocessing the acquired text data using a natural language processing library;

[1625] A means of inferring the pronoun's reference source based on the context and inserting it in place of the pronoun;

[1626] A means of replacing homonyms with appropriate words that fit the context;

[1627] means for adjusting the order of verbs, objects, and subjects based on syntactic analysis;

[1628] A means for recursively repeating the above to improve the quality of the text data;

[1629] means for storing and outputting the converted text data;

[1630] means for playing the converted text data aloud;

[1631] A system including:

[1632] (Claim 2)

[1633] 10. The system of claim 1, wherein syntactic and contextual analysis is used to infer the referent of a pronoun.

[1634] (Claim 3)

[1635] 10. The system of claim 1, wherein the system utilizes a synonym list to select an appropriate word based on context when replacing homonyms.

[1636] "Application Example 1"

[1637] (Claim 1)

[1638] a means for retrieving text from a web page;

[1639] A means of inferring the pronoun's reference source based on the context and inserting it in place of the pronoun;

[1640] A means of replacing homonyms with context-appropriate synonyms;

[1641] a means of adjusting the order of verbs, objects, and subjects;

[1642] A method for improving the quality of writing by repeating the above recursively,

[1643] means for storing and outputting the converted text;

[1644] A means of acquiring product information in the virtual space and converting text into voice;

[1645] The system includes a means for providing voice guidance of acquired product information in real time.

[1646] (Claim 2)

[1647] 10. The system of claim 1, wherein syntactic analysis is used to infer the referent of a pronoun.

[1648] (Claim 3)

[1649] 10. The system of claim 1, wherein a thesaurus is utilized when replacing homonyms.

[1650] "Example 2: Combining Emotion Engines"

[1651] (Claim 1)

[1652] a means for retrieving text from a web page;

[1653] A means of inferring the pronoun's reference source based on the context and inserting it in place of the pronoun;

[1654] A means of replacing homonyms with context-appropriate synonyms;

[1655] a means of adjusting the order of verbs, objects, and subjects;

[1656] means for analyzing the emotional state of a user and adjusting the sentences according to the emotion;

[1657] A method for improving the quality of writing by repeating the above recursively,

[1658] The system includes means for storing and outputting the converted text.

[1659] (Claim 2)

[1660] 10. The system of claim 1, wherein syntactic analysis is used to infer the referent of a pronoun.

[1661] (Claim 3)

[1662] 10. The system of claim 1, wherein a thesaurus is utilized when replacing homonyms.

[1663] "Application example 2 when combining emotion engines"

[1664] (Claim 1)

[1665] a means for retrieving text from a web page;

[1666] A means of inferring the pronoun's reference source based on the context and inserting it in place of the pronoun;

[1667] A means of replacing homonyms with context-appropriate synonyms;

[1668] a means of adjusting the order of verbs, objects, and subjects;

[1669] A method for improving the quality of writing by repeating the above recursively,

[1670] means for storing and outputting the converted text;

[1671] means for recognizing a user's emotion and adjusting the sentence based on the emotion;

[1672] A system including means for outputting text to a user in audio form.

[1673] (Claim 2)

[1674] 10. The system of claim 1, wherein syntactic analysis is used to infer the referent of a pronoun.

[1675] (Claim 3)

[1676] 10. The system of claim 1, wherein a thesaurus is utilized when replacing homonyms. [Explanation of symbols]

[1677] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:>

Claims

1. a means for retrieving text from a web page; A means of inferring the pronoun's reference source based on the context and inserting it in place of the pronoun; A means of replacing homonyms with context-appropriate synonyms; a means of adjusting the order of verbs, objects, and subjects; A method for improving the quality of writing by repeating the above recursively, The system includes means for storing and outputting the converted text.

2. 10. The system of claim 1, wherein syntactic analysis is used to infer the referent of a pronoun.

3. 10. The system of claim 1, wherein a thesaurus is utilized when replacing homonyms.

Images