System

Abstract

A system is provided.SOLUTION: The system includes an input means for inputting information by a user, a transmission means for transmitting the input information to a server, a conversion means for converting the information received by the server into braille, a transmission means for transmitting the converted braille information to a terminal, and a display means for displaying the braille information to the user by the terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In the past, visually impaired people faced many barriers when obtaining store information and service manuals. Specifically, only visual information was often provided, and there was no efficient system for providing this information in Braille. As a result, it was difficult for visually impaired people to obtain information independently, and they often experienced inconvenience and anxiety when using stores. Therefore, there is a need for a system that allows visually impaired people to easily obtain the information they need in Braille. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing a system including an input means for a user to input information, a transmission means for transmitting the input information to a server, a conversion means for converting the information received by the server into Braille, a transmission means for transmitting the converted Braille information to a terminal, and a display means for the terminal to display the Braille information to the user. The transmission means also includes a means for retrieving corresponding service manual information from a database based on the information received from the input means, and the conversion means includes a means for using an algorithm to convert text information into Braille, thereby providing an environment in which visually impaired people can efficiently obtain information.

[0006] "User" means a visually impaired person who uses the system to obtain information.

[0007] "Input means" refers to a device or interface for a user to input information, and corresponds to text input or voice input.

[0008] "Transmission means" refers to a communication means for transmitting input information to a server, and is a wired or wireless network communication.

[0009] A "server" is a computer device that receives and processes information sent by a user.

[0010] "Conversion means" refers to the algorithm or process by which the server converts received information into Braille.

[0011] "Braille" is a special writing system that can be read by touch by the visually impaired.

[0012] A "database" is a collection of information that stores the service manual information and other information required by the server.

[0013] "Display means" refers to a device or interface for displaying the converted Braille information to a user, such as a Braille display device or Braille printer. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] The present invention relates to a system that enables visually impaired users to obtain store information and service manuals in Braille. This system assists visually impaired users in obtaining information by inputting information, converting it into Braille, and providing it to them.

[0036] Overall system picture

[0037] The system consists of the following main components:

[0038] 1. An input method for users to enter information

[0039] 2. A means by which the terminal receives user input and sends it to the server

[0040] 3. Transformation means for processing the information received by the server

[0041] 4. A means for transmitting the converted Braille information to the terminal

[0042] 5. Display method for displaying braille information on the device

[0043] Description of Specific Embodiments

[0044] 1. User enters information

[0045] A user accesses a terminal in a store and inputs information. This input method includes text input and voice input.

[0046] Examples:

[0047] When a user requests "how to handle the front desk" by voice input, the terminal converts this request into text data.

[0048] 2. The device sends a request to the server

[0049] The terminal receives the user's request and sends it to the server using a transmission means. The data sent here is JSON format data that includes the request content, user ID, etc.

[0050] Examples:

[0051] The terminal sends "Request content: How the front desk should respond" and "User ID" to the server.

[0052] 3. The server receives and interprets the request

[0053] The server receives the request from the device, analyzes its contents, and retrieves the relevant information from the database.

[0054] Examples:

[0055] The server analyzes the request "How should the front desk handle this?" and retrieves the corresponding text information from the database.

[0056] 4. The server converts the text into Braille

[0057] The server converts the acquired text into Braille, applying a Braille conversion algorithm to convert characters into Braille codes.

[0058] Examples:

[0059] The server converts the "front desk's response instructions" into Braille. For example, "Fu" is converted to ⠋, "Ro" to ⠚, "N" to ⠴, and "To" to ⠞.

[0060] 5. The server sends the Braille data to the device.

[0061] The server transmits the data converted into Braille to the terminal, and the transmission means is used to provide the formatted Braille data to the terminal.

[0062] Examples:

[0063] The braille data is formatted as "⠋⠚⠴⠞(Front)..." and sent to the device.

[0064] 6. The device displays the braille information

[0065] The terminal displays the received Braille data to the user. The display means includes a Braille display device and a Braille printer.

[0066] Examples:

[0067] The terminal displays the Braille information of "How to handle the front desk" to the user via the Braille display device.

[0068] Program processing flow explanation

[0069] The processing flow of the program of this system will now be described in detail.

[0070] 1. User enters information:

[0071] The user enters text or voice input.

[0072] The terminal receives the input information and generates request data.

[0073] 2. The device sends a request to the server:

[0074] The terminal converts the request data into JSON format and transmits it to the server using the transmission means.

[0075] 3. The server receives and interprets the request:

[0076] The server receives the request, analyzes the contents, and retrieves the relevant information from the database.

[0077] 4. Server converts to Braille:

[0078] The text information acquired by the server is converted into Braille using a Braille conversion algorithm.

[0079] 5. The server sends the Braille data:

[0080] The server sends the Braille data to the terminal.

[0081] 6. The device displays the braille information:

[0082] The terminal displays the received braille data on a braille display device or braille printer.

[0083] The system of the present invention allows visually impaired users to easily obtain the information they need, improving the convenience of using stores.

[0084] The processing flow will be explained below.

[0085] Step 1:

[0086] The user accesses the store's terminal and requests the information they want to receive by voice input or text input. For example, if the user requests "how to handle the front desk" by voice input, the terminal converts this request into text data.

[0087] Step 2:

[0088] The device converts the textual request content into JSON format data along with identification information such as the user ID. This data is in preparation for sending the request content to the server.

[0089] Step 3:

[0090] The terminal sends JSON format data to the server using a transmission means. Specifically, it is sent as an HTTP request via the Internet or a local network.

[0091] Step 4:

[0092] The server receives the HTTP request and analyzes the request using text parsing technology to identify the type of service manual and specific items requested.

[0093] Step 5:

[0094] The server accesses the database and retrieves text information corresponding to the parsed request content. For example, text information corresponding to "how the front desk should respond" is retrieved from the database.

[0095] Step 6:

[0096] The server converts the acquired text into Braille using a Braille conversion algorithm, which converts each character into its corresponding Braille code.

[0097] Step 7:

[0098] The server converts the Braille data back into JSON format and sends it to the terminal using a transmission means, again as an HTTP response.

[0099] Step 8:

[0100] The device receives the HTTP response from the server, extracts the braille data, and then sends the data to a braille display device or braille printer, preparing to display the information to the user.

[0101] Step 9:

[0102] The terminal uses a Braille display device or Braille printer to display Braille information to the user. For example, the Braille information "⠋⠚⠴⠞... (How to handle the front desk)" is physically displayed to the user.

[0103] Step 10:

[0104] The user reads the braille information from the braille display device or braille printer and obtains the desired information. Through this process, visually impaired users can independently obtain the information they need when using a store.

[0105] Example 1

[0106] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0107] When visually impaired users need information, it is necessary to provide visual information in Braille format. However, current systems have limitations in how quickly and easily users can obtain the information they need in Braille while in a store. In particular, the entire process from inputting information to converting it into Braille and displaying it must be carried out efficiently. Another challenge is to improve user convenience.

[0108] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0109] In this invention, the server includes an analysis means for analyzing user input, an acquisition means for acquiring relevant information from a database, and a conversion means for converting the acquired information into Braille, thereby enabling visually impaired users to quickly and easily acquire necessary information in Braille, improving convenience when using a store.

[0110] "User" means a visually impaired individual who utilizes the system to obtain information.

[0111] "Input means" refers to a device such as a text input device or a voice input device that allows a user to input information.

[0112] "Terminal" refers to an electronic device that receives user input information and transmits it to a server as needed.

[0113] "Transmission means" refers to the communication means or protocol for transmitting information from the terminal to the server.

[0114] "Server" refers to a central management system that receives requests from users and processes the required information.

[0115] "Means for obtaining" refers to the functions and processes by which the server obtains the relevant information from the database.

[0116] "Analysis means" refers to the algorithms and processes that the server uses to analyze the information received from the terminal and understand its content.

[0117] "Conversion means" refers to the algorithm or system used by the server to convert the text information it acquires into Braille.

[0118] "Display means" refers to a device such as a braille display or braille printer that the terminal uses to display braille information to the user.

[0119] "Database" refers to a data storage that stores information desired by users and that a server accesses to retrieve that information.

[0120] MODE FOR CARRYING OUT THE INVENTION

[0121] This invention relates to a system that allows visually impaired users to obtain store information and service manuals in Braille format. This system receives user input, converts it into Braille information, and provides it to visually impaired users, making it easier for them to obtain information.

[0122] Overall system configuration

[0123] The system consists of the following main components:

[0124] 1. An input method for users to enter information

[0125] 2. A means for the terminal to receive input information and send it to the server

[0126] 3. A means for the server to analyze the received information and retrieve the information from the database

[0127] 4. A means for converting the information acquired by the server into Braille

[0128] 5. A means for the server to transmit the converted Braille information to the terminal.

[0129] 6. Display means for the terminal to display the transmitted Braille information to the user.

[0130] Input Method

[0131] Users use the store's terminals to input information. Input methods include text input and voice input. For example, if a user voice-inputs a request such as "How do I contact the front desk?", the terminal uses voice recognition software to convert this voice data into text.

[0132] Transmission method

[0133] The terminal sends the user's input data to the server. The data sent is in JSON format and includes information such as the request content and user ID. For example, the terminal sends "Request content: How the front desk should respond" and "User ID: 12345" to the server.

[0134] Analysis and acquisition methods

[0135] The server includes an analysis means for analyzing the received information. The analysis means analyzes the request, and the server uses an acquisition means for acquiring the relevant information from the database. For example, the server analyzes the request "How should the front desk handle this?" and acquires the relevant text information from the database.

[0136] Conversion Method

[0137] The server has a conversion means for converting the acquired text information into Braille. The Braille conversion algorithm converts the text into Braille code. For example, when "How to handle the front desk" is converted into Braille, it is converted as follows: "Fu" ⇒ ⠋, "Ro" ⇒ ⠚, "N" ⇒ ⠴, "To" ⇒ ⠞.

[0138] Braille data transmission method

[0139] The server has a transmitting means for transmitting the converted Braille data to the terminal. For example, the Braille data "⠋⠚⠴⠞(Front)..." is transmitted to the terminal.

[0140] Display means

[0141] The terminal has a display means for displaying the received Braille data to the user. The display means includes a Braille display device and a Braille printer. For example, the terminal displays "⠋⠚⠴⠞..." on the Braille display device.

[0142] Specific examples

[0143] As a concrete example, consider the following scenario:

[0144] 1. The user speaks "How should the front desk help?"

[0145] 2. The device converts the voice data into text data and sends it to the server.

[0146] 3. The server analyzes the received data and retrieves the relevant information from the database.

[0147] 4. The server converts the acquired information into Braille and sends it to the device.

[0148] 5. The device displays the braille information to the user.

[0149] Prompt Sentence Examples

[0150] Text input: "Please translate front desk instructions into Braille."

[0151] Voice input: "How can I help the front desk?"

[0152] The system of the present invention enables visually impaired users to quickly and easily obtain necessary information in Braille while in a store, improving convenience when using the store.

[0153] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0154] Step 1:

[0155] The user enters information

[0156] The user inputs information using an input means. Specifically, the user uses the device's voice input function to say, "How do I contact the front desk?" The device converts this voice data into text data. The input data is voice data, which is then converted into text data for further processing. Specifically, the voice recognition software analyzes the voice and generates the corresponding text.

[0157] Step 2:

[0158] The device sends a request to the server

[0159] The terminal receives the text data and prepares to send it to the server. Specifically, the terminal generates JSON data including "Request content: Front desk response method" and "User ID: 12345" and sends it to the server using the transmission means. The input data is text data, and the output data is JSON-formatted request data for the server.

[0160] Step 3:

[0161] The server receives and parses the request

[0162] The server receives the JSON formatted data sent from the terminal and analyzes its contents. The analysis means identifies that the request content is "how the front desk should respond." The input data is the JSON formatted request data, and the output data is the analyzed request content information. Specifically, the server parses the JSON data and identifies the request content.

[0163] Step 4:

[0164] The server retrieves information from the database

[0165] Based on the analysis results, the server retrieves the relevant information from the database. Specifically, the server executes a database query to retrieve the information "how the front desk should respond." The input data is the parsed request content, and the output data is the retrieved text information. Specifically, the server generates an SQL statement and executes the query against the database.

[0166] Step 5:

[0167] The server converts the information obtained into Braille.

[0168] The server converts the text information retrieved from the database into Braille. Using a conversion means, the text information is converted into Braille code to generate Braille data. The input data is text information, and the output data is Braille data. Specifically, the server applies a Braille conversion algorithm to convert the text into Braille code.

[0169] Step 6:

[0170] The server sends the Braille data to the device.

[0171] The server transmits the converted Braille data to the terminal. The Braille data is transmitted to the terminal using a transmission means. The input data is Braille data, and the output data is Braille data for the terminal. In concrete terms, the server generates formatted Braille data and transmits it to the terminal.

[0172] Step 7:

[0173] The device displays the braille information.

[0174] The terminal displays the received Braille data to the user. Using the display means, the Braille data is presented to the user via a Braille display device or Braille printer. The input data is Braille data, and the output data is information displayed in Braille. In concrete terms, the terminal displays the Braille information on a Braille display.

[0175] These specific processing steps enable visually impaired users to quickly and easily obtain the information they need in Braille, improving convenience when using stores.

[0176] (Application example 1)

[0177] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0178] One issue that visually impaired people face is that it has been difficult for them to easily obtain the information they need in public places and commercial facilities. Furthermore, the lack of a system that provides store information or service manuals in Braille limits access to information for visually impaired people. The current situation, where the usefulness of voice input interfaces is not being fully utilized, is also an issue that needs to be resolved.

[0179] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0180] In this invention, the server includes input means for a user to input voice or text, transmission means for transmitting the input information to the server, acquisition means for the server to analyze the received information and acquire corresponding data, conversion means for converting the acquired data into Braille, transmission means for transmitting the converted Braille information to the terminal, display means for the terminal to display the Braille information to the user, means for using an algorithm to convert voice data into text, and means for using an algorithm to convert text information into Braille. This enables visually impaired people to acquire necessary information using voice input and receive it in Braille.

[0181] "Voice" or "text" refers to the means used by a user to input information, where voice input refers to spoken sounds and text input refers to characters such as those on a keyboard.

[0182] "Input means" refers to a means by which a user inputs information into a terminal, and includes a voice recognition device, a keyboard, a touch screen, and the like.

[0183] "Transmission means" refers to a device or function for converting information received from the input means into a digital format and transmitting it to the server.

[0184] "Acquisition means" refers to the function of the server analyzing the received information and acquiring the corresponding data from a database or the like.

[0185] "Conversion means" refers to an algorithm or device for converting acquired data into another format, and specifically includes an algorithm for converting text data into Braille.

[0186] "Display means" refers to a device or function for displaying final information to the user, and includes a Braille display device and a Braille printer.

[0187] "Server" refers to a central computer system that receives requests from users, analyzes, converts, and transmits information.

[0188] The present invention relates to a system that enables visually impaired users to obtain store information and service information in Braille. This system assists visually impaired users in obtaining information by inputting information, converting it into Braille, and providing it to them.

[0189] Key Components of the System

[0190] The system consists of the following main components:

[0191] 1. Input means: A means by which a user inputs voice or text, including a voice recognition device, keyboard, or touch screen.

[0192] 2. Transmission means: A means for converting information received from the input means into a digital format and transmitting it to the server.

[0193] 3. Acquisition method: The method by which the server analyzes the information received and acquires the corresponding data from a database, etc.

[0194] 4. Conversion means: The algorithm or device that converts the acquired data into Braille.

[0195] 5. Display means: A Braille display device or Braille printer to display the final information to the user.

[0196] Program processing flow

[0197] 1. User inputs information by voice: The user launches the smartphone application and inputs the information they want to know by voice (e.g., "Please tell me the product description for shelf A"). The voice data is converted into text data using the smartphone's voice recognition function (e.g., Google® Cloud Speech-to-Text API).

[0198] 2. The device sends the request to the server: The smartphone converts this text data into JSON format and sends it to the store's server.

[0199] 3. The server receives and interprets the request: The server analyzes the received JSON data and retrieves information corresponding to the request content from the database.

[0200] 4. The server converts the text information into Braille: The server applies an algorithm (e.g., Braille translation software such as Duxbury) to convert the acquired text data into Braille data.

[0201] 5. The server sends the Braille data: The server formats the Braille data into JSON format and sends it to the smartphone.

[0202] 6. The smartphone displays the braille information: The smartphone displays the received braille data on the screen using a braille display module (for example, an app such as BrailleBack).

[0203] Hardware and software used

[0204] Hardware:

[0205] Smartphone (iOS or ANDROID (registered trademark))

[0206] Braille display devices (such as BrailleBack accessories for smartphones)

[0207] software:

[0208] Google Cloud Speech-to-Text API (voice recognition)

[0209] RESTful API (data communication with server)

[0210] Duxbury (Braille conversion)

[0211] Braille display app (such as BrailleBack)

[0212] Specific examples

[0213] For example, if a user says, "Tell me the product description for shelf A," the speech is converted to text using the Google Cloud Speech-to-Text API. The smartphone then sends this text data in JSON format to a server, which converts it into Braille and sends it back to the smartphone. The BrailleBack module on the smartphone displays the Braille information, allowing the user to read it.

[0214] Prompt Sentence Examples

[0215] "The user's voice input, 'Tell me the product description for shelf A,' is converted to text using the Google Cloud Speech-to-Text API and sent to the server in JSON format. The server retrieves the text information from the database and converts it into Braille using Duxbury. The Braille data is then sent to the smartphone and displayed in Braille using the BrailleBack module."

[0216] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0217] Step 1:

[0218] The user launches the smartphone application and inputs the information they want to know by voice. Using a speech recognition function (for example, Google Cloud Speech-to-Text API), the voice data is converted into text data. The input is the user's voice data, and the output is text data generated by speech recognition. Specifically, when the user says, "Tell me the product description for shelf A," this is displayed as text on the smartphone.

[0219] Step 2:

[0220] The device acquires the text data generated by speech recognition and converts it into JSON format. The input is the text data generated by speech recognition, and the output is JSON format data. Specifically, the text data "Please tell me the product description for shelf A" is converted into JSON format such as "request_text":"Please tell me the product description for shelf A".

[0221] Step 3:

[0222] The terminal sends the converted JSON format data to the server. Data communication is carried out by the transmission means. The input is JSON format data, and the output is sent to the server. Specifically, the JSON data "request_text":"Please tell me the product description for shelf A" is sent over the network.

[0223] Step 4:

[0224] The server analyzes the received JSON data and retrieves the corresponding data from the database. The database is operated by the retrieval means. The input is the JSON data received from the terminal, and the output is text data retrieved from the database. Specifically, the request "Tell me the product description for shelf A" is analyzed, and the corresponding product description is retrieved from the database.

[0225] Step 5:

[0226] The server converts the acquired text data into Braille. A conversion means applies a Braille translation algorithm (e.g., Duxbury). The input is text data acquired from the database, and the output is Braille data. Specifically, the text "Product description on shelf A" is converted into Braille code.

[0227] Step 6:

[0228] The server formats the data converted into Braille into JSON format and sends it to the terminal. Data communication is carried out by the transmission means. The input is Braille data, and the output is transmission to the terminal. Specifically, the Braille code is formatted into JSON format such as "Braille data":"⠑⠭⠁⠍⠏⠇⠑⠃⠐⠰..." and sent.

[0229] Step 7:

[0230] The terminal displays the received Braille data. A Braille display module (e.g., BrailleBack) is used to provide information to visually impaired people. The input is the Braille data received from the server, and the output is the display in Braille. Specifically, the user can read "Product description on shelf A" by touching the Braille on the smartphone screen.

[0231] In this way, visually impaired people can easily obtain information and improve convenience in stores.

[0232] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0233] This invention provides a system that enables visually impaired users to efficiently obtain store information and service manuals in Braille. Furthermore, this system incorporates an emotion engine that recognizes the user's emotions and provides optimal information based on the user's emotions.

[0234] Overall system picture

[0235] The system consists of the following main components:

[0236] 1. An input method for users to enter information

[0237] 2. A means by which the terminal receives user input and sends it to the server

[0238] 3. Transformation means for processing the information received by the server

[0239] 4. Emotion engine that recognizes user emotions

[0240] 5. Means for transmitting the converted Braille information to the terminal

[0241] 6. Display means by which the device displays braille information to the user

[0242] Description of Specific Embodiments

[0243] 1. User enters information

[0244] The user accesses the store's terminal and inputs information. This input method includes text input and voice input. The emotion engine also recognizes emotions from the user's voice and input content.

[0245] Examples:

[0246] When a user requests "how to handle the front desk" by voice input, the device converts this request into text data, and the emotion engine analyzes the user's emotions from the voice.

[0247] 2. The device sends a request to the server

[0248] The device sends the textual request content and the user's emotion data to the server using a transmission means. The data sent here is in JSON format and includes the request content, user ID, emotion data, etc.

[0249] Examples:

[0250] The terminal sends the "Request content: How the front desk should respond", "User ID", and "Emotion data" to the server.

[0251] 3. The server receives and interprets the request

[0252] The server receives the request from the device and analyzes its contents. This analysis includes text parsing technology and emotional data analysis. The server then retrieves the relevant service manual information from the database.

[0253] Examples:

[0254] The server analyzes the request "How should the front desk handle this?", retrieves the corresponding text information from the database, and determines the approach to provide the information based on the user's emotional data.

[0255] 4. The server converts the text into Braille

[0256] The server converts the acquired text into Braille using a Braille conversion algorithm, which converts each character into the corresponding Braille code.

[0257] Examples:

[0258] The server converts "How to handle the front desk" into Braille, converting it as follows: "Fu" ⇒ ⠋, "Ro" ⇒ ⠚, "N" ⇒ ⠴, "To" ⇒ ⠞.

[0259] 5. The server sends the Braille data to the device.

[0260] The server converts the Braille data back into JSON format and sends it to the device using a transmission method. This is also sent as an HTTP response. Additional information based on emotions may also be included.

[0261] Examples:

[0262] The braille data is formatted as "⠋⠚⠴⠞(Front)..." and sent to the device. Based on the emotion data, additional information on how the user can relax is also sent.

[0263] 6. The device displays the braille information

[0264] The terminal displays the received Braille data to the user. The display means includes a Braille display device and a Braille printer. In addition, information based on feedback from the emotion engine is also displayed.

[0265] Examples:

[0266] The terminal displays Braille information on "how to handle the front desk" to the user via a Braille display device, and also displays Braille information on "approaches to relax."

[0267] 7. The user receives the information

[0268] The user reads the braille information from a braille display device or braille printer and obtains the desired information. This process allows visually impaired users to efficiently obtain information and receive appropriate information according to their emotions.

[0269] The system of the present invention enables visually impaired users to efficiently obtain the information they need and provides optimal information according to their emotions, thereby increasing convenience and satisfaction.

[0270] The processing flow will be explained below.

[0271] Step 1:

[0272] A user accesses a terminal in a store and requests the necessary information by text input or voice input. At this time, the emotion engine analyzes the user's emotions from the user's voice and input content. For example, if a user requests "how to handle the front desk" by voice input, the terminal converts the request into text data, and the emotion engine extracts emotional data from the voice.

[0273] Step 2:

[0274] The device generates JSON format data containing the text of the request and the extracted emotion data. This data includes the request content, user ID, emotion data, etc.

[0275] Step 3:

[0276] The terminal sends JSON format data to the server using a transmission means. Specifically, it is sent as an HTTP request over the network.

[0277] Step 4:

[0278] The server receives the HTTP request and analyzes the request content and sentiment data using text parsing technology and sentiment analysis algorithms to identify the type of service manual requested and the user's sentiment.

[0279] Step 5:

[0280] Based on the analysis results, the server retrieves the relevant service manual information from the database. For example, text information on "how to handle a customer at the front desk" is retrieved from the database.

[0281] Step 6:

[0282] The server converts the acquired text into Braille using a Braille conversion algorithm, which converts each character into its corresponding Braille code.

[0283] Step 7:

[0284] The emotion engine determines the additional information needed based on the user's emotional data. For example, if the user is feeling stressed, information on approaches to relaxation will be added.

[0285] Step 8:

[0286] The server generates JSON data containing the Braille data and any additional information determined by the emotion engine, ready to be sent to the device.

[0287] Step 9:

[0288] The server generates JSON format data and sends it to the terminal using a transmission method. This is also sent as an HTTP response via the network.

[0289] Step 10:

[0290] The device receives the HTTP response from the server, extracts the braille data and additional information, and then sends the data to a braille display device or braille printer, preparing to display the information to the user.

[0291] Step 11:

[0292] The terminal uses a Braille display device or Braille printer to display Braille information to the user. Specifically, the Braille information "⠋⠚⠴⠞ (how the front desk handles the situation)" and the Braille information "approach to relaxation" based on emotions are displayed.

[0293] Step 12:

[0294] The user reads the braille information from a braille display or braille printer and obtains the desired information and additional useful information. This process allows visually impaired users to efficiently obtain the information they need and receive appropriate information according to their emotions.

[0295] Example 2

[0296] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0297] There are issues with the lack of efficient means for visually impaired users to obtain information, making it difficult to provide appropriate information. In addition, there is a lack of optimal information provision that responds to the user's emotions, which results in problems with how information is received and its convenience.

[0298] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0299] In this invention, the server includes an analysis means for analyzing received information and acquiring user emotion data, a conversion means for converting the analyzed information into Braille, and a transmission means for transmitting the converted Braille information to the terminal, thereby enabling appropriate information to be provided based on the user's emotion, and enabling visually impaired users to efficiently acquire information.

[0300] "Input means" refers to a means for a user to input information, and includes text input, voice input, and the like.

[0301] The "transmission means" is a means for transmitting the input information to the server, and typically transmits data via an HTTP request.

[0302] The "analysis means" refers to a means for analyzing the information received by the server and obtaining the user's emotional data, and includes text parsing and emotion analysis techniques.

[0303] The "conversion means" is a means for converting the information analyzed by the server into Braille, and uses a Braille conversion algorithm.

[0304] "Display means" refers to means by which the terminal displays Braille information to the user, and includes a Braille display device and a Braille printer.

[0305] MODE FOR CARRYING OUT THE INVENTION

[0306] This invention relates to a system that enables visually impaired users to efficiently obtain store information and service manuals in Braille. Furthermore, this system incorporates an emotion engine that recognizes the user's emotions and provides appropriate information based on the user's emotions.

[0307] Specific hardware and software configurations

[0308] The system consists of the following main components:

[0309] 1. Input Method: The means by which users input information includes text input and voice input, specifically voice recognition software (e.g., Google Speech-to-Text) and keyboards.

[0310] 2. Transmission method: An HTTP request is used to transmit the entered information to the server.

[0311] 3. Analysis means: The server analyzes the received information and obtains the user's emotional data using natural language processing (NLP) technology or emotion analysis software (e.g., Microsoft® Azure® Emotion API).

[0312] 4. Conversion method: A Braille conversion algorithm (e.g., Liblouis) is used as a means for converting the information analyzed by the server into Braille.

[0313] 5. Transmission method: HTTP responses are again used as a means to transmit the converted Braille information to the terminal.

[0314] 6. Display means: A Braille display device (e.g., Refreshable Braille Display) or a Braille printer is used as a means for the terminal to display Braille information to the user.

[0315] Specific example of system operation

[0316] For example, if a user accesses a store terminal and requests "how to handle the front desk" by voice input, the following operation will occur.

[0317] 1. The device is equipped with voice recognition software that converts the speech into text data. At the same time, an emotion engine analyzes the tone and speed of the user's voice to determine the user's emotions.

[0318] 2. The device packs the acquired text data and emotion data into JSON format and sends an HTTP POST request to the server.

[0319] 3. The server receives the JSON data and analyzes its contents. First, it uses natural language processing technology to understand the request and retrieves the relevant service manual information from the database. It also processes the emotion data and determines how to provide information based on the user's emotion.

[0320] 4. A Braille conversion algorithm is used to convert the text information obtained by the server into Braille.

[0321] 5. The generated braille data is packed again in JSON format and sent to the device as an HTTP response, optionally including additional information based on the emotion data.

[0322] 6. The terminal sends the received Braille data to a Braille display device for display to the user, which displays the Braille data so that the user can read it.

[0323] Examples of prompt statements

[0324] "This system allows visually impaired users to efficiently retrieve store information and service manuals in Braille. Please provide a typical user scenario."

[0325] "Explain how a user can request assistance from the front desk by voice, and a server will translate that information into Braille and provide it to them."

[0326] This allows visually impaired users to acquire information efficiently and provides appropriate information according to their emotions.

[0327] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0328] Step 1:

[0329] The user inputs information. The user requests "How should the front desk respond?" by voice into the terminal. The terminal uses voice recognition software to convert this voice into text data (input: voice data, output: text data). At the same time, the emotion engine analyzes the user's tone of voice and speaking style to obtain emotional data (input: voice data, output: emotional data). Specifically, when the user voice-inputs "How should the front desk respond?", the terminal's microphone captures the voice data, which is then converted into text data by the voice recognition software. At the same time, the emotion engine obtains the emotional data.

[0330] Step 2:

[0331] The device sends the request content and emotion data to the server. The device then packs the acquired text data and emotion data into JSON format and sends an HTTP POST request to the server (input: text data, emotion data, output: JSON format data). Specifically, the text data "How should the front desk handle this?" and the emotion data are sent to the server in JSON format as "{\"userID\": 12345, \"request\": \"How should the front desk handle this\", \"emotion\": \"neutral\"}".

[0332] Step 3:

[0333] The server receives and analyzes the request. The server receives the JSON data sent from the terminal and analyzes the content (input: JSON data, output: analyzed text data and emotion data). First, it uses natural language processing technology to understand the request content and retrieves the corresponding service manual information from a database (input: request content, output: service manual information). It then processes the emotion data and determines how to provide information based on the user's emotion (input: emotion data, output: information provision method). Specifically, the server analyzes the request for "how the front desk should respond," retrieves the corresponding text information from the database, and determines the appropriate information provision method based on the emotion data "neutral."

[0334] Step 4:

[0335] The server converts the acquired text information into Braille. The server converts the acquired text information into Braille code using a Braille conversion algorithm (input: text information, output: Braille data). Specifically, the server converts the string "How the front desk responds" into Braille as follows: "Fu" ⇒ ⠋, "Ro" ⇒ ⠚, "N" ⇒ ⠴, "To" ⇒ ⠞.

[0336] Step 5:

[0337] The server sends the braille data to the device. The generated braille data is packed again in JSON format and sent to the device as an HTTP response (input: braille data, output: JSON format data). If necessary, additional information based on the emotion data may be included. Specifically, the braille data "⠋⠚⠴⠞(front)..." is packed in JSON format and sent to the device as "{\"braille_data\": \"⠋⠚⠴⠞...\", \"additional_info\": \"Approaches to Relaxation\"}".

[0338] Step 6:

[0339] The device displays the braille data to the user. The device sends the received braille data to the braille display device and displays it to the user (input: braille data in JSON format, output: braille display). Specifically, the device sends the data "⠋⠚⠴⠞(front)..." to the braille display device, which then displays it to the user. Braille data for the additional information "Approaches to Relaxation" is also displayed.

[0340] Step 7:

[0341] The user reads the braille information. The user obtains the necessary information by reading the braille information from the braille display device (input: braille display data, output: information obtained by the user). Specifically, the user reads the braille information on "how the front desk will handle the situation" and the braille data on "approaches to relaxing" from the braille display device.

[0342] (Application example 2)

[0343] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0344] When visually impaired users efficiently obtain the information they need in stores and facilities, that information needs to be optimized according to their emotions. However, current systems are unable to provide information that takes the user's emotions into account, resulting in a lack of convenience and satisfaction. In addition, there are situations where it is difficult for users to quickly receive specific guidance information in Braille or audio.

[0345] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an input means for a user to input information, a transmission means for transmitting the input information to the server, a conversion means for converting the received information into Braille, a transmission means for transmitting the converted Braille information to the terminal, a display means for displaying the Braille information to the user, and an emotion recognition means for recognizing the user's emotion and providing optimal information. This enables visually impaired users to efficiently obtain the information they need in a store or facility in a form optimized according to their emotion.

[0346] An "input means" is a device or interface through which a user inputs information.

[0347] The "transmitting means" is a communication device or mechanism for transmitting input information to the server.

[0348] A "conversion means" is a device or algorithm that converts information received by the server into Braille.

[0349] "Display means" refers to a device or mechanism by which a terminal displays Braille information to a user.

[0350] The "emotion recognition means" is a device or system that recognizes the user's emotions and provides optimal information based on those emotions.

[0351] "Guide information" refers to information that a user needs in a facility or store, and specifically includes store information, service manuals, and the like.

[0352] The "server" is a computer system that analyzes the received information, converts it into Braille, and performs emotion recognition.

[0353] The realization of this system involves the following major components:

[0354] Overall system configuration

[0355] 1. "Input means" for users to enter information

[0356] 2. "Transmission method" to send the entered information to the server

[0357] 3. "Conversion means" that converts the information received by the server into Braille

[0358] 4. "Transmission means" for transmitting the converted Braille information to the terminal

[0359] 5. "Display means" by which the device displays Braille information to the user

[0360] 6. "Emotion recognition means" that recognizes the user's emotions and provides optimal information

[0361] Specific Embodiments of the System

[0362] 1. Enter your information

[0363] Users input their voice via smart glasses or a smartphone. The voice is converted into text data by speech recognition software. At this time, an emotion recognition engine analyzes the user's emotions from the voice data. The specific software used is EmotionRecognizer and a speech recognition library.

[0364] 2. Transmission of information

[0365] Once speech recognition and sentiment analysis are complete, the data is packaged in JSON format and sent to a server using the HTTP protocol. The server receives and analyzes this data using software such as a request sending library.

[0366] 3. Analysis of Information

[0367] The server analyzes the received request and retrieves the appropriate guidance information from a database. This analysis uses Natural Language Processing (NLP) technology, and determines the appropriate information provision approach based on the user's emotional data.

[0368] 4. Conversion to Braille

[0369] The acquired guidance information is converted into Braille data using a Braille conversion algorithm. Specifically, the BrailleConverter library is used. The conversion results are written as Braille code.

[0370] 5. Resending Information

[0371] The server then converts the Braille data back into JSON format and sends it to the device as an HTTP response, which also includes additional feedback information based on the emotion data.

[0372] 6. Display of Information

[0373] The device then provides the received Braille data to the user using a Braille display or Braille printer. Feedback based on emotion recognition is also displayed. This allows the user to read the necessary guidance information in Braille and receive optimal feedback based on their emotions.

[0374] Examples of concrete examples and prompts

[0375] For example, if a user speaks "What is the timetable for the next train?" at a train station information desk, the system will take the following steps:

[0376] 1. Convert the voice input "Please tell me the timetable for the next train" into text.

[0377] 2. The emotion recognition engine detects the user's impatience from the tone of the voice.

[0378] 3. The server now provides information about the next train time and adds a message to reassure anxious users.

[0379] 4. Smart glasses and smartphones will provide information through braille displays and audio guidance.

[0380] Example prompt for a generative AI model:

[0381] "A user says, 'What is the timetable for the next train?' Convert this speech to text, recognize the user's emotions, and suggest the best way to respond."

[0382] This format allows visually impaired users to efficiently obtain guidance information, and also enables optimal information provision that takes into consideration their emotions at the time.

[0383] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0384] Step 1:

[0385] The user uses smart glasses or a smartphone to input voice. The input voice is converted into text data by speech recognition software. Specifically, the user inputs "Please tell me the timetable for the next train," which is converted into text "Please tell me the timetable for the next train." At the same time, an emotion recognition engine analyzes the voice data and recognizes the user's emotion (in this case, impatience). The input is voice data and emotion data, and the output is text data and analyzed emotion data.

[0386] Step 2:

[0387] The device packages text data and emotion data in JSON format and sends it to the server via the HTTP protocol. Specifically, the user's voice is sent as text "Please tell me the timetable for the next train," along with emotion data. The input is text data and emotion data, and the output is an HTTP request to the server.

[0388] Step 3:

[0389] The server analyzes the received JSON data. The server uses Natural Language Processing (NLP) technology to understand the content of the text data and retrieves the corresponding guidance information from the database. For example, the server recognizes the request "next train timetable" and retrieves the next train timetable from the database. The input is JSON data, and the output is guidance information.

[0390] Step 4:

[0391] The server converts the acquired guidance information into Braille data using a Braille conversion algorithm. In this process, the BrailleConverter library is used to convert the text "next train timetable" into Braille. The input is the text data of the guidance information, and the output is Braille data.

[0392] Step 5:

[0393] The server converts the feedback information based on the Braille data and emotion data back into JSON format and sends it to the terminal as an HTTP response. Specifically, it contains the emotion feedback "⠞⠝⠋⠥. (Braille)" and "Please stay calm and wait for the next train." The input is the Braille data and emotion feedback information, and the output is an HTTP response.

[0394] Step 6:

[0395] The device analyzes the received data and displays the information to the user through a braille display or audio output. The user can read "⠞⠝⠋⠥." on the braille display or receive information about the next train time and a message to relax via audio. The input is an HTTP response, and the output is information in a format (braille or audio) that can be understood by visually impaired users.

[0396] As a format

[0397] It explains that each step in the process has data inputs and outputs, and details how the information that the user ultimately receives is generated.

[0398] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0399] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0400] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0401] [Second embodiment]

[0402] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0403] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0404] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0405] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0406] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0407] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0408] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0409] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0410] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0411] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0412] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0413] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0414] The present invention relates to a system that enables visually impaired users to obtain store information and service manuals in Braille. This system assists visually impaired users in obtaining information by inputting information, converting it into Braille, and providing it to them.

[0415] Overall system picture

[0416] The system consists of the following main components:

[0417] 1. An input method for users to enter information

[0418] 2. A means by which the terminal receives user input and sends it to the server

[0419] 3. Transformation means for processing the information received by the server

[0420] 4. A means for transmitting the converted Braille information to the terminal

[0421] 5. Display method for displaying braille information on the device

[0422] Description of Specific Embodiments

[0423] 1. User enters information

[0424] A user accesses a terminal in a store and inputs information. This input method includes text input and voice input.

[0425] Examples:

[0426] When a user requests "how to handle the front desk" by voice input, the terminal converts this request into text data.

[0427] 2. The device sends a request to the server

[0428] The terminal receives the user's request and sends it to the server using a transmission means. The data sent here is JSON format data that includes the request content, user ID, etc.

[0429] Examples:

[0430] The terminal sends "Request content: How the front desk should respond" and "User ID" to the server.

[0431] 3. The server receives and interprets the request

[0432] The server receives the request from the device, analyzes its contents, and retrieves the relevant information from the database.

[0433] Examples:

[0434] The server analyzes the request "How should the front desk handle this?" and retrieves the corresponding text information from the database.

[0435] 4. The server converts the text into Braille

[0436] The server converts the acquired text into Braille, applying a Braille conversion algorithm to convert characters into Braille codes.

[0437] Examples:

[0438] The server converts the "front desk's response instructions" into Braille. For example, "Fu" is converted to ⠋, "Ro" to ⠚, "N" to ⠴, and "To" to ⠞.

[0439] 5. The server sends the Braille data to the device.

[0440] The server transmits the data converted into Braille to the terminal, and the transmission means is used to provide the formatted Braille data to the terminal.

[0441] Examples:

[0442] The braille data is formatted as "⠋⠚⠴⠞(Front)..." and sent to the device.

[0443] 6. The device displays the braille information

[0444] The terminal displays the received Braille data to the user. The display means includes a Braille display device and a Braille printer.

[0445] Examples:

[0446] The terminal displays the Braille information of "How to handle the front desk" to the user via the Braille display device.

[0447] Program processing flow explanation

[0448] The processing flow of the program of this system will now be described in detail.

[0449] 1. User enters information:

[0450] The user enters text or voice input.

[0451] The terminal receives the input information and generates request data.

[0452] 2. The device sends a request to the server:

[0453] The terminal converts the request data into JSON format and transmits it to the server using the transmission means.

[0454] 3. The server receives and interprets the request:

[0455] The server receives the request, analyzes the contents, and retrieves the relevant information from the database.

[0456] 4. Server converts to Braille:

[0457] The text information acquired by the server is converted into Braille using a Braille conversion algorithm.

[0458] 5. The server sends the Braille data:

[0459] The server sends the Braille data to the terminal.

[0460] 6. The device displays the braille information:

[0461] The terminal displays the received braille data on a braille display device or braille printer.

[0462] The system of the present invention allows visually impaired users to easily obtain the information they need, improving the convenience of using stores.

[0463] The processing flow will be explained below.

[0464] Step 1:

[0465] The user accesses the store's terminal and requests the information they want to receive by voice input or text input. For example, if the user requests "how to handle the front desk" by voice input, the terminal converts this request into text data.

[0466] Step 2:

[0467] The device converts the textual request content into JSON format data along with identification information such as the user ID. This data is in preparation for sending the request content to the server.

[0468] Step 3:

[0469] The terminal sends JSON format data to the server using a transmission means. Specifically, it is sent as an HTTP request via the Internet or a local network.

[0470] Step 4:

[0471] The server receives the HTTP request and analyzes the request using text parsing technology to identify the type of service manual and specific items requested.

[0472] Step 5:

[0473] The server accesses the database and retrieves text information corresponding to the parsed request content. For example, text information corresponding to "how the front desk should respond" is retrieved from the database.

[0474] Step 6:

[0475] The server converts the acquired text into Braille using a Braille conversion algorithm, which converts each character into its corresponding Braille code.

[0476] Step 7:

[0477] The server converts the Braille data back into JSON format and sends it to the terminal using a transmission means, again as an HTTP response.

[0478] Step 8:

[0479] The device receives the HTTP response from the server, extracts the braille data, and then sends the data to a braille display device or braille printer, preparing to display the information to the user.

[0480] Step 9:

[0481] The terminal uses a Braille display device or Braille printer to display Braille information to the user. For example, the Braille information "⠋⠚⠴⠞... (How to handle the front desk)" is physically displayed to the user.

[0482] Step 10:

[0483] The user reads the braille information from the braille display device or braille printer and obtains the desired information. Through this process, visually impaired users can independently obtain the information they need when using a store.

[0484] Example 1

[0485] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0486] When visually impaired users need information, it is necessary to provide visual information in Braille format. However, current systems have limitations in how quickly and easily users can obtain the information they need in Braille while in a store. In particular, the entire process from inputting information to converting it into Braille and displaying it must be carried out efficiently. Another challenge is to improve user convenience.

[0487] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0488] In this invention, the server includes an analysis means for analyzing user input, an acquisition means for acquiring relevant information from a database, and a conversion means for converting the acquired information into Braille, thereby enabling visually impaired users to quickly and easily acquire necessary information in Braille, improving convenience when using a store.

[0489] "User" means a visually impaired individual who utilizes the system to obtain information.

[0490] "Input means" refers to a device such as a text input device or a voice input device that allows a user to input information.

[0491] "Terminal" refers to an electronic device that receives user input information and transmits it to a server as needed.

[0492] "Transmission means" refers to the communication means or protocol for transmitting information from the terminal to the server.

[0493] "Server" refers to a central management system that receives requests from users and processes the required information.

[0494] "Means for obtaining" refers to the functions and processes by which the server obtains the relevant information from the database.

[0495] "Analysis means" refers to the algorithms and processes that the server uses to analyze the information received from the terminal and understand its content.

[0496] "Conversion means" refers to the algorithm or system used by the server to convert the text information it acquires into Braille.

[0497] "Display means" refers to a device such as a braille display or braille printer that the terminal uses to display braille information to the user.

[0498] "Database" refers to a data storage that stores information desired by users and that a server accesses to retrieve that information.

[0499] MODE FOR CARRYING OUT THE INVENTION

[0500] This invention relates to a system that allows visually impaired users to obtain store information and service manuals in Braille format. This system receives user input, converts it into Braille information, and provides it to visually impaired users, making it easier for them to obtain information.

[0501] Overall system configuration

[0502] The system consists of the following main components:

[0503] 1. An input method for users to enter information

[0504] 2. A means for the terminal to receive input information and send it to the server

[0505] 3. A means for the server to analyze the received information and retrieve the information from the database

[0506] 4. A means for converting the information acquired by the server into Braille

[0507] 5. A means for the server to transmit the converted Braille information to the terminal.

[0508] 6. Display means for the terminal to display the transmitted Braille information to the user.

[0509] Input Method

[0510] Users use the store's terminals to input information. Input methods include text input and voice input. For example, if a user voice-inputs a request such as "How do I contact the front desk?", the terminal uses voice recognition software to convert this voice data into text.

[0511] Transmission method

[0512] The terminal sends the user's input data to the server. The data sent is in JSON format and includes information such as the request content and user ID. For example, the terminal sends "Request content: How the front desk should respond" and "User ID: 12345" to the server.

[0513] Analysis and acquisition methods

[0514] The server includes an analysis means for analyzing the received information. The analysis means analyzes the request, and the server uses an acquisition means for acquiring the relevant information from the database. For example, the server analyzes the request "How should the front desk handle this?" and acquires the relevant text information from the database.

[0515] Conversion Method

[0516] The server has a conversion means for converting the acquired text information into Braille. The Braille conversion algorithm converts the text into Braille code. For example, when "How to handle the front desk" is converted into Braille, it is converted as follows: "Fu" ⇒ ⠋, "Ro" ⇒ ⠚, "N" ⇒ ⠴, "To" ⇒ ⠞.

[0517] Braille data transmission method

[0518] The server has a transmitting means for transmitting the converted Braille data to the terminal. For example, the Braille data "⠋⠚⠴⠞(Front)..." is transmitted to the terminal.

[0519] Display means

[0520] The terminal has a display means for displaying the received Braille data to the user. The display means includes a Braille display device and a Braille printer. For example, the terminal displays "⠋⠚⠴⠞..." on the Braille display device.

[0521] Specific examples

[0522] As a concrete example, consider the following scenario:

[0523] 1. The user speaks "How should the front desk help?"

[0524] 2. The device converts the voice data into text data and sends it to the server.

[0525] 3. The server analyzes the received data and retrieves the relevant information from the database.

[0526] 4. The server converts the acquired information into Braille and sends it to the device.

[0527] 5. The device displays the braille information to the user.

[0528] Prompt Sentence Examples

[0529] Text input: "Please translate front desk instructions into Braille."

[0530] Voice input: "How can I help the front desk?"

[0531] The system of the present invention enables visually impaired users to quickly and easily obtain necessary information in Braille while in a store, improving convenience when using the store.

[0532] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0533] Step 1:

[0534] The user enters information

[0535] The user inputs information using an input means. Specifically, the user uses the device's voice input function to say, "How do I contact the front desk?" The device converts this voice data into text data. The input data is voice data, which is then converted into text data for further processing. Specifically, the voice recognition software analyzes the voice and generates the corresponding text.

[0536] Step 2:

[0537] The device sends a request to the server

[0538] The terminal receives the text data and prepares to send it to the server. Specifically, the terminal generates JSON data including "Request content: Front desk response method" and "User ID: 12345" and sends it to the server using the transmission means. The input data is text data, and the output data is JSON-formatted request data for the server.

[0539] Step 3:

[0540] The server receives and parses the request

[0541] The server receives the JSON formatted data sent from the terminal and analyzes its contents. The analysis means identifies that the request content is "how the front desk should respond." The input data is the JSON formatted request data, and the output data is the analyzed request content information. Specifically, the server parses the JSON data and identifies the request content.

[0542] Step 4:

[0543] The server retrieves information from the database

[0544] Based on the analysis results, the server retrieves the relevant information from the database. Specifically, the server executes a database query to retrieve the information "how the front desk should respond." The input data is the parsed request content, and the output data is the retrieved text information. Specifically, the server generates an SQL statement and executes the query against the database.

[0545] Step 5:

[0546] The server converts the information obtained into Braille.

[0547] The server converts the text information retrieved from the database into Braille. Using a conversion means, the text information is converted into Braille code to generate Braille data. The input data is text information, and the output data is Braille data. Specifically, the server applies a Braille conversion algorithm to convert the text into Braille code.

[0548] Step 6:

[0549] The server sends the Braille data to the device.

[0550] The server transmits the converted Braille data to the terminal. The Braille data is transmitted to the terminal using a transmission means. The input data is Braille data, and the output data is Braille data for the terminal. In concrete terms, the server generates formatted Braille data and transmits it to the terminal.

[0551] Step 7:

[0552] The device displays the braille information.

[0553] The terminal displays the received Braille data to the user. Using the display means, the Braille data is presented to the user via a Braille display device or Braille printer. The input data is Braille data, and the output data is information displayed in Braille. In concrete terms, the terminal displays the Braille information on a Braille display.

[0554] These specific processing steps enable visually impaired users to quickly and easily obtain the information they need in Braille, improving convenience when using stores.

[0555] (Application example 1)

[0556] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0557] One issue that visually impaired people face is that it has been difficult for them to easily obtain the information they need in public places and commercial facilities. Furthermore, the lack of a system that provides store information or service manuals in Braille limits access to information for visually impaired people. The current situation, where the usefulness of voice input interfaces is not being fully utilized, is also an issue that needs to be resolved.

[0558] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0559] In this invention, the server includes input means for a user to input voice or text, transmission means for transmitting the input information to the server, acquisition means for the server to analyze the received information and acquire corresponding data, conversion means for converting the acquired data into Braille, transmission means for transmitting the converted Braille information to the terminal, display means for the terminal to display the Braille information to the user, means for using an algorithm to convert voice data into text, and means for using an algorithm to convert text information into Braille. This enables visually impaired people to acquire necessary information using voice input and receive it in Braille.

[0560] "Voice" or "text" refers to the means used by a user to input information, where voice input refers to spoken sounds and text input refers to characters such as those on a keyboard.

[0561] "Input means" refers to a means by which a user inputs information into a terminal, and includes a voice recognition device, a keyboard, a touch screen, and the like.

[0562] "Transmission means" refers to a device or function for converting information received from the input means into a digital format and transmitting it to the server.

[0563] "Acquisition means" refers to the function of the server analyzing the received information and acquiring the corresponding data from a database or the like.

[0564] "Conversion means" refers to an algorithm or device for converting acquired data into another format, and specifically includes an algorithm for converting text data into Braille.

[0565] "Display means" refers to a device or function for displaying final information to the user, and includes a Braille display device and a Braille printer.

[0566] "Server" refers to a central computer system that receives requests from users, analyzes, converts, and transmits information.

[0567] The present invention relates to a system that enables visually impaired users to obtain store information and service information in Braille. This system assists visually impaired users in obtaining information by inputting information, converting it into Braille, and providing it to them.

[0568] Key Components of the System

[0569] The system consists of the following main components:

[0570] 1. Input means: A means by which a user inputs voice or text, including a voice recognition device, keyboard, or touch screen.

[0571] 2. Transmission means: A means for converting information received from the input means into a digital format and transmitting it to the server.

[0572] 3. Acquisition method: The method by which the server analyzes the information received and acquires the corresponding data from a database, etc.

[0573] 4. Conversion means: The algorithm or device that converts the acquired data into Braille.

[0574] 5. Display means: A Braille display device or Braille printer to display the final information to the user.

[0575] Program processing flow

[0576] 1. User inputs information by voice: The user launches the smartphone application and inputs the information they want to know by voice (e.g., "Please tell me the product description for shelf A"). The voice data is converted into text data using the smartphone's voice recognition function (e.g., Google Cloud Speech-to-Text API).

[0577] 2. The device sends the request to the server: The smartphone converts this text data into JSON format and sends it to the store's server.

[0578] 3. The server receives and interprets the request: The server analyzes the received JSON data and retrieves information corresponding to the request content from the database.

[0579] 4. The server converts the text information into Braille: The server applies an algorithm (e.g., Braille translation software such as Duxbury) to convert the acquired text data into Braille data.

[0580] 5. The server sends the Braille data: The server formats the Braille data into JSON format and sends it to the smartphone.

[0581] 6. The smartphone displays the braille information: The smartphone displays the received braille data on the screen using a braille display module (for example, an app such as BrailleBack).

[0582] Hardware and software used

[0583] Hardware:

[0584] Smartphone (iOS or Android)

[0585] Braille display devices (such as BrailleBack accessories for smartphones)

[0586] software:

[0587] Google Cloud Speech-to-Text API (voice recognition)

[0588] RESTful API (data communication with server)

[0589] Duxbury (Braille conversion)

[0590] Braille display app (such as BrailleBack)

[0591] Specific examples

[0592] For example, if a user says, "Tell me the product description for shelf A," the speech is converted to text using the Google Cloud Speech-to-Text API. The smartphone then sends this text data in JSON format to a server, which converts it into Braille and sends it back to the smartphone. The BrailleBack module on the smartphone displays the Braille information, allowing the user to read it.

[0593] Prompt Sentence Examples

[0594] "The user's voice input, 'Tell me the product description for shelf A,' is converted to text using the Google Cloud Speech-to-Text API and sent to the server in JSON format. The server retrieves the text information from the database and converts it into Braille using Duxbury. The Braille data is then sent to the smartphone and displayed in Braille using the BrailleBack module."

[0595] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0596] Step 1:

[0597] The user launches the smartphone application and inputs the information they want to know by voice. Using a speech recognition function (for example, Google Cloud Speech-to-Text API), the voice data is converted into text data. The input is the user's voice data, and the output is text data generated by speech recognition. Specifically, when the user says, "Tell me the product description for shelf A," this is displayed as text on the smartphone.

[0598] Step 2:

[0599] The device acquires the text data generated by speech recognition and converts it into JSON format. The input is the text data generated by speech recognition, and the output is JSON format data. Specifically, the text data "Please tell me the product description for shelf A" is converted into JSON format such as "request_text":"Please tell me the product description for shelf A".

[0600] Step 3:

[0601] The terminal sends the converted JSON format data to the server. Data communication is carried out by the transmission means. The input is JSON format data, and the output is sent to the server. Specifically, the JSON data "request_text":"Please tell me the product description for shelf A" is sent over the network.

[0602] Step 4:

[0603] The server analyzes the received JSON data and retrieves the corresponding data from the database. The database is operated by the retrieval means. The input is the JSON data received from the terminal, and the output is text data retrieved from the database. Specifically, the request "Tell me the product description for shelf A" is analyzed, and the corresponding product description is retrieved from the database.

[0604] Step 5:

[0605] The server converts the acquired text data into Braille. A conversion means applies a Braille translation algorithm (e.g., Duxbury). The input is text data acquired from the database, and the output is Braille data. Specifically, the text "Product description on shelf A" is converted into Braille code.

[0606] Step 6:

[0607] The server formats the data converted into Braille into JSON format and sends it to the terminal. Data communication is carried out by the transmission means. The input is Braille data, and the output is transmission to the terminal. Specifically, the Braille code is formatted into JSON format such as "Braille data":"⠑⠭⠁⠍⠏⠇⠑⠃⠐⠰..." and sent.

[0608] Step 7:

[0609] The terminal displays the received Braille data. A Braille display module (e.g., BrailleBack) is used to provide information to visually impaired people. The input is the Braille data received from the server, and the output is the display in Braille. Specifically, the user can read "Product description on shelf A" by touching the Braille on the smartphone screen.

[0610] In this way, visually impaired people can easily obtain information and improve convenience in stores.

[0611] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0612] This invention provides a system that enables visually impaired users to efficiently obtain store information and service manuals in Braille. Furthermore, this system incorporates an emotion engine that recognizes the user's emotions and provides optimal information based on the user's emotions.

[0613] Overall system picture

[0614] The system consists of the following main components:

[0615] 1. An input method for users to enter information

[0616] 2. A means by which the terminal receives user input and sends it to the server

[0617] 3. Transformation means for processing the information received by the server

[0618] 4. Emotion engine that recognizes user emotions

[0619] 5. Means for transmitting the converted Braille information to the terminal

[0620] 6. Display means by which the device displays braille information to the user

[0621] Description of Specific Embodiments

[0622] 1. User enters information

[0623] The user accesses the store's terminal and inputs information. This input method includes text input and voice input. The emotion engine also recognizes emotions from the user's voice and input content.

[0624] Examples:

[0625] When a user requests "how to handle the front desk" by voice input, the device converts this request into text data, and the emotion engine analyzes the user's emotions from the voice.

[0626] 2. The device sends a request to the server

[0627] The device sends the textual request content and the user's emotion data to the server using a transmission means. The data sent here is in JSON format and includes the request content, user ID, emotion data, etc.

[0628] Examples:

[0629] The terminal sends the "Request content: How the front desk should respond", "User ID", and "Emotion data" to the server.

[0630] 3. The server receives and interprets the request

[0631] The server receives the request from the device and analyzes its contents. This analysis includes text parsing technology and emotional data analysis. The server then retrieves the relevant service manual information from the database.

[0632] Examples:

[0633] The server analyzes the request "How should the front desk handle this?", retrieves the corresponding text information from the database, and determines the approach to provide the information based on the user's emotional data.

[0634] 4. The server converts the text into Braille

[0635] The server converts the acquired text into Braille using a Braille conversion algorithm, which converts each character into the corresponding Braille code.

[0636] Examples:

[0637] The server converts "How to handle the front desk" into Braille, converting it as follows: "Fu" ⇒ ⠋, "Ro" ⇒ ⠚, "N" ⇒ ⠴, "To" ⇒ ⠞.

[0638] 5. The server sends the Braille data to the device.

[0639] The server converts the Braille data back into JSON format and sends it to the device using a transmission method. This is also sent as an HTTP response. Additional information based on emotions may also be included.

[0640] Examples:

[0641] The braille data is formatted as "⠋⠚⠴⠞(Front)..." and sent to the device. Based on the emotion data, additional information on how the user can relax is also sent.

[0642] 6. The device displays the braille information

[0643] The terminal displays the received Braille data to the user. The display means includes a Braille display device and a Braille printer. In addition, information based on feedback from the emotion engine is also displayed.

[0644] Examples:

[0645] The terminal displays Braille information on "how to handle the front desk" to the user via a Braille display device, and also displays Braille information on "approaches to relax."

[0646] 7. The user receives the information

[0647] The user reads the braille information from a braille display device or braille printer and obtains the desired information. This process allows visually impaired users to efficiently obtain information and receive appropriate information according to their emotions.

[0648] The system of the present invention enables visually impaired users to efficiently obtain the information they need and provides optimal information according to their emotions, thereby increasing convenience and satisfaction.

[0649] The processing flow will be explained below.

[0650] Step 1:

[0651] A user accesses a terminal in a store and requests the necessary information by text input or voice input. At this time, the emotion engine analyzes the user's emotions from the user's voice and input content. For example, if a user requests "how to handle the front desk" by voice input, the terminal converts the request into text data, and the emotion engine extracts emotional data from the voice.

[0652] Step 2:

[0653] The device generates JSON format data containing the text of the request and the extracted emotion data. This data includes the request content, user ID, emotion data, etc.

[0654] Step 3:

[0655] The terminal sends JSON format data to the server using a transmission means. Specifically, it is sent as an HTTP request over the network.

[0656] Step 4:

[0657] The server receives the HTTP request and analyzes the request content and sentiment data using text parsing technology and sentiment analysis algorithms to identify the type of service manual requested and the user's sentiment.

[0658] Step 5:

[0659] Based on the analysis results, the server retrieves the relevant service manual information from the database. For example, text information on "how to handle a customer at the front desk" is retrieved from the database.

[0660] Step 6:

[0661] The server converts the acquired text into Braille using a Braille conversion algorithm, which converts each character into its corresponding Braille code.

[0662] Step 7:

[0663] The emotion engine determines the additional information needed based on the user's emotional data. For example, if the user is feeling stressed, information on approaches to relaxation will be added.

[0664] Step 8:

[0665] The server generates JSON data containing the Braille data and any additional information determined by the emotion engine, ready to be sent to the device.

[0666] Step 9:

[0667] The server generates JSON format data and sends it to the terminal using a transmission method. This is also sent as an HTTP response via the network.

[0668] Step 10:

[0669] The device receives the HTTP response from the server, extracts the braille data and additional information, and then sends the data to a braille display device or braille printer, preparing to display the information to the user.

[0670] Step 11:

[0671] The terminal uses a Braille display device or Braille printer to display Braille information to the user. Specifically, the Braille information "⠋⠚⠴⠞ (how the front desk handles the situation)" and the Braille information "approach to relaxation" based on emotions are displayed.

[0672] Step 12:

[0673] The user reads the braille information from a braille display or braille printer and obtains the desired information and additional useful information. This process allows visually impaired users to efficiently obtain the information they need and receive appropriate information according to their emotions.

[0674] Example 2

[0675] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0676] There are issues with the lack of efficient means for visually impaired users to obtain information, making it difficult to provide appropriate information. In addition, there is a lack of optimal information provision that responds to the user's emotions, which results in problems with how information is received and its convenience.

[0677] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0678] In this invention, the server includes an analysis means for analyzing received information and acquiring user emotion data, a conversion means for converting the analyzed information into Braille, and a transmission means for transmitting the converted Braille information to the terminal, thereby enabling appropriate information to be provided based on the user's emotion, and enabling visually impaired users to efficiently acquire information.

[0679] "Input means" refers to a means for a user to input information, and includes text input, voice input, and the like.

[0680] The "transmission means" is a means for transmitting the input information to the server, and typically transmits data via an HTTP request.

[0681] The "analysis means" refers to a means for analyzing the information received by the server and obtaining the user's emotional data, and includes text parsing and emotion analysis techniques.

[0682] The "conversion means" is a means for converting the information analyzed by the server into Braille, and uses a Braille conversion algorithm.

[0683] "Display means" refers to means by which the terminal displays Braille information to the user, and includes a Braille display device and a Braille printer.

[0684] MODE FOR CARRYING OUT THE INVENTION

[0685] This invention relates to a system that enables visually impaired users to efficiently obtain store information and service manuals in Braille. Furthermore, this system incorporates an emotion engine that recognizes the user's emotions and provides appropriate information based on the user's emotions.

[0686] Specific hardware and software configurations

[0687] The system consists of the following main components:

[0688] 1. Input Method: The means by which users input information includes text input and voice input, specifically voice recognition software (e.g., Google Speech-to-Text) and keyboards.

[0689] 2. Transmission method: An HTTP request is used to transmit the entered information to the server.

[0690] 3. Analysis method: The server analyzes the received information and obtains the user's emotional data using natural language processing (NLP) technology or emotion analysis software (e.g., Microsoft Azure Emotion API).

[0691] 4. Conversion method: A Braille conversion algorithm (e.g., Liblouis) is used as a means for converting the information analyzed by the server into Braille.

[0692] 5. Transmission method: HTTP responses are again used as a means to transmit the converted Braille information to the terminal.

[0693] 6. Display means: A Braille display device (e.g., Refreshable Braille Display) or a Braille printer is used as a means for the terminal to display Braille information to the user.

[0694] Specific example of system operation

[0695] For example, if a user accesses a store terminal and requests "how to handle the front desk" by voice input, the following operation will occur.

[0696] 1. The device is equipped with voice recognition software that converts the speech into text data. At the same time, an emotion engine analyzes the tone and speed of the user's voice to determine the user's emotions.

[0697] 2. The device packs the acquired text data and emotion data into JSON format and sends an HTTP POST request to the server.

[0698] 3. The server receives the JSON data and analyzes its contents. First, it uses natural language processing technology to understand the request and retrieves the relevant service manual information from the database. It also processes the emotion data and determines how to provide information based on the user's emotion.

[0699] 4. A Braille conversion algorithm is used to convert the text information obtained by the server into Braille.

[0700] 5. The generated braille data is packed again in JSON format and sent to the device as an HTTP response, optionally including additional information based on the emotion data.

[0701] 6. The terminal sends the received Braille data to a Braille display device for display to the user, which displays the Braille data so that the user can read it.

[0702] Examples of prompt statements

[0703] "This system allows visually impaired users to efficiently retrieve store information and service manuals in Braille. Please provide a typical user scenario."

[0704] "Explain how a user can request assistance from the front desk by voice, and a server will translate that information into Braille and provide it to them."

[0705] This allows visually impaired users to acquire information efficiently and provides appropriate information according to their emotions.

[0706] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0707] Step 1:

[0708] The user inputs information. The user requests "How should the front desk respond?" by voice into the terminal. The terminal uses voice recognition software to convert this voice into text data (input: voice data, output: text data). At the same time, the emotion engine analyzes the user's tone of voice and speaking style to obtain emotional data (input: voice data, output: emotional data). Specifically, when the user voice-inputs "How should the front desk respond?", the terminal's microphone captures the voice data, which is then converted into text data by the voice recognition software. At the same time, the emotion engine obtains the emotional data.

[0709] Step 2:

[0710] The device sends the request content and emotion data to the server. The device then packs the acquired text data and emotion data into JSON format and sends an HTTP POST request to the server (input: text data, emotion data, output: JSON format data). Specifically, the text data "How should the front desk handle this?" and the emotion data are sent to the server in JSON format as "{\"userID\": 12345, \"request\": \"How should the front desk handle this\", \"emotion\": \"neutral\"}".

[0711] Step 3:

[0712] The server receives and analyzes the request. The server receives the JSON data sent from the terminal and analyzes the content (input: JSON data, output: analyzed text data and emotion data). First, it uses natural language processing technology to understand the request content and retrieves the corresponding service manual information from a database (input: request content, output: service manual information). It then processes the emotion data and determines how to provide information based on the user's emotion (input: emotion data, output: information provision method). Specifically, the server analyzes the request for "how the front desk should respond," retrieves the corresponding text information from the database, and determines the appropriate information provision method based on the emotion data "neutral."

[0713] Step 4:

[0714] The server converts the acquired text information into Braille. The server converts the acquired text information into Braille code using a Braille conversion algorithm (input: text information, output: Braille data). Specifically, the server converts the string "How the front desk responds" into Braille as follows: "Fu" ⇒ ⠋, "Ro" ⇒ ⠚, "N" ⇒ ⠴, "To" ⇒ ⠞.

[0715] Step 5:

[0716] The server sends the braille data to the device. The generated braille data is packed again in JSON format and sent to the device as an HTTP response (input: braille data, output: JSON format data). If necessary, additional information based on the emotion data may be included. Specifically, the braille data "⠋⠚⠴⠞(front)..." is packed in JSON format and sent to the device as "{\"braille_data\": \"⠋⠚⠴⠞...\", \"additional_info\": \"Approaches to Relaxation\"}".

[0717] Step 6:

[0718] The device displays the braille data to the user. The device sends the received braille data to the braille display device and displays it to the user (input: braille data in JSON format, output: braille display). Specifically, the device sends the data "⠋⠚⠴⠞(front)..." to the braille display device, which then displays it to the user. Braille data for the additional information "Approaches to Relaxation" is also displayed.

[0719] Step 7:

[0720] The user reads the braille information. The user obtains the necessary information by reading the braille information from the braille display device (input: braille display data, output: information obtained by the user). Specifically, the user reads the braille information on "how the front desk will handle the situation" and the braille data on "approaches to relaxing" from the braille display device.

[0721] (Application example 2)

[0722] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0723] When visually impaired users efficiently obtain the information they need in stores and facilities, that information needs to be optimized according to their emotions. However, current systems are unable to provide information that takes the user's emotions into account, resulting in a lack of convenience and satisfaction. In addition, there are situations where it is difficult for users to quickly receive specific guidance information in Braille or audio.

[0724] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an input means for a user to input information, a transmission means for transmitting the input information to the server, a conversion means for converting the received information into Braille, a transmission means for transmitting the converted Braille information to the terminal, a display means for displaying the Braille information to the user, and an emotion recognition means for recognizing the user's emotion and providing optimal information. This enables visually impaired users to efficiently obtain the information they need in a store or facility in a form optimized according to their emotion.

[0725] An "input means" is a device or interface through which a user inputs information.

[0726] The "transmitting means" is a communication device or mechanism for transmitting input information to the server.

[0727] A "conversion means" is a device or algorithm that converts information received by the server into Braille.

[0728] "Display means" refers to a device or mechanism by which a terminal displays Braille information to a user.

[0729] The "emotion recognition means" is a device or system that recognizes the user's emotions and provides optimal information based on those emotions.

[0730] "Guide information" refers to information that a user needs in a facility or store, and specifically includes store information, service manuals, and the like.

[0731] The "server" is a computer system that analyzes the received information, converts it into Braille, and performs emotion recognition.

[0732] The realization of this system involves the following major components:

[0733] Overall system configuration

[0734] 1. "Input means" for users to enter information

[0735] 2. "Transmission method" to send the entered information to the server

[0736] 3. "Conversion means" that converts the information received by the server into Braille

[0737] 4. "Transmission means" for transmitting the converted Braille information to the terminal

[0738] 5. "Display means" by which the device displays Braille information to the user

[0739] 6. "Emotion recognition means" that recognizes the user's emotions and provides optimal information

[0740] Specific Embodiments of the System

[0741] 1. Enter your information

[0742] Users input their voice via smart glasses or a smartphone. The voice is converted into text data by speech recognition software. At this time, an emotion recognition engine analyzes the user's emotions from the voice data. The specific software used is EmotionRecognizer and a speech recognition library.

[0743] 2. Transmission of information

[0744] Once speech recognition and sentiment analysis are complete, the data is packaged in JSON format and sent to a server using the HTTP protocol. The server receives and analyzes this data using software such as a request sending library.

[0745] 3. Analysis of Information

[0746] The server analyzes the received request and retrieves the appropriate guidance information from a database. This analysis uses Natural Language Processing (NLP) technology, and determines the appropriate information provision approach based on the user's emotional data.

[0747] 4. Conversion to Braille

[0748] The acquired guidance information is converted into Braille data using a Braille conversion algorithm. Specifically, the BrailleConverter library is used. The conversion results are written as Braille code.

[0749] 5. Resending Information

[0750] The server then converts the Braille data back into JSON format and sends it to the device as an HTTP response, which also includes additional feedback information based on the emotion data.

[0751] 6. Display of Information

[0752] The device then provides the received Braille data to the user using a Braille display or Braille printer. Feedback based on emotion recognition is also displayed. This allows the user to read the necessary guidance information in Braille and receive optimal feedback based on their emotions.

[0753] Examples of concrete examples and prompts

[0754] For example, if a user speaks "What is the timetable for the next train?" at a train station information desk, the system will take the following steps:

[0755] 1. Convert the voice input "Please tell me the timetable for the next train" into text.

[0756] 2. The emotion recognition engine detects the user's impatience from the tone of the voice.

[0757] 3. The server now provides information about the next train time and adds a message to reassure anxious users.

[0758] 4. Smart glasses and smartphones will provide information through braille displays and audio guidance.

[0759] Example prompt for a generative AI model:

[0760] "A user says, 'What is the timetable for the next train?' Convert this speech to text, recognize the user's emotions, and suggest the best way to respond."

[0761] This format allows visually impaired users to efficiently obtain guidance information, and also enables optimal information provision that takes into consideration their emotions at the time.

[0762] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0763] Step 1:

[0764] The user uses smart glasses or a smartphone to input voice. The input voice is converted into text data by speech recognition software. Specifically, the user inputs "Please tell me the timetable for the next train," which is converted into text "Please tell me the timetable for the next train." At the same time, an emotion recognition engine analyzes the voice data and recognizes the user's emotion (in this case, impatience). The input is voice data and emotion data, and the output is text data and analyzed emotion data.

[0765] Step 2:

[0766] The device packages text data and emotion data in JSON format and sends it to the server via the HTTP protocol. Specifically, the user's voice is sent as text "Please tell me the timetable for the next train," along with emotion data. The input is text data and emotion data, and the output is an HTTP request to the server.

[0767] Step 3:

[0768] The server analyzes the received JSON data. The server uses Natural Language Processing (NLP) technology to understand the content of the text data and retrieves the corresponding guidance information from the database. For example, the server recognizes the request "next train timetable" and retrieves the next train timetable from the database. The input is JSON data, and the output is guidance information.

[0769] Step 4:

[0770] The server converts the acquired guidance information into Braille data using a Braille conversion algorithm. In this process, the BrailleConverter library is used to convert the text "next train timetable" into Braille. The input is the text data of the guidance information, and the output is Braille data.

[0771] Step 5:

[0772] The server converts the feedback information based on the Braille data and emotion data back into JSON format and sends it to the terminal as an HTTP response. Specifically, it contains the emotion feedback "⠞⠝⠋⠥. (Braille)" and "Please stay calm and wait for the next train." The input is the Braille data and emotion feedback information, and the output is an HTTP response.

[0773] Step 6:

[0774] The device analyzes the received data and displays the information to the user through a braille display or audio output. The user can read "⠞⠝⠋⠥." on the braille display or receive information about the next train time and a message to relax via audio. The input is an HTTP response, and the output is information in a format (braille or audio) that can be understood by visually impaired users.

[0775] As a format

[0776] It explains that each step in the process has data inputs and outputs, and details how the information that the user ultimately receives is generated.

[0777] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0778] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0779] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0780] [Third embodiment]

[0781] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0782] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0783] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0784] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0785] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0786] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0787] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0788] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0789] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0790] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0791] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0792] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0793] The present invention relates to a system that enables visually impaired users to obtain store information and service manuals in Braille. This system assists visually impaired users in obtaining information by inputting information, converting it into Braille, and providing it to them.

[0794] Overall system picture

[0795] The system consists of the following main components:

[0796] 1. An input method for users to enter information

[0797] 2. A means by which the terminal receives user input and sends it to the server

[0798] 3. Transformation means for processing the information received by the server

[0799] 4. A means for transmitting the converted Braille information to the terminal

[0800] 5. Display method for displaying braille information on the device

[0801] Description of Specific Embodiments

[0802] 1. User enters information

[0803] A user accesses a terminal in a store and inputs information. This input method includes text input and voice input.

[0804] Examples:

[0805] When a user requests "how to handle the front desk" by voice input, the terminal converts this request into text data.

[0806] 2. The device sends a request to the server

[0807] The terminal receives the user's request and sends it to the server using a transmission means. The data sent here is JSON format data that includes the request content, user ID, etc.

[0808] Examples:

[0809] The terminal sends "Request content: How the front desk should respond" and "User ID" to the server.

[0810] 3. The server receives and interprets the request

[0811] The server receives the request from the device, analyzes its contents, and retrieves the relevant information from the database.

[0812] Examples:

[0813] The server analyzes the request "How should the front desk handle this?" and retrieves the corresponding text information from the database.

[0814] 4. The server converts the text into Braille

[0815] The server converts the acquired text into Braille, applying a Braille conversion algorithm to convert characters into Braille codes.

[0816] Examples:

[0817] The server converts the "front desk's response instructions" into Braille. For example, "Fu" is converted to ⠋, "Ro" to ⠚, "N" to ⠴, and "To" to ⠞.

[0818] 5. The server sends the Braille data to the device.

[0819] The server transmits the data converted into Braille to the terminal, and the transmission means is used to provide the formatted Braille data to the terminal.

[0820] Examples:

[0821] The braille data is formatted as "⠋⠚⠴⠞(Front)..." and sent to the device.

[0822] 6. The device displays the braille information

[0823] The terminal displays the received Braille data to the user. The display means includes a Braille display device and a Braille printer.

[0824] Examples:

[0825] The terminal displays the Braille information of "How to handle the front desk" to the user via the Braille display device.

[0826] Program processing flow explanation

[0827] The processing flow of the program of this system will now be described in detail.

[0828] 1. User enters information:

[0829] The user enters text or voice input.

[0830] The terminal receives the input information and generates request data.

[0831] 2. The device sends a request to the server:

[0832] The terminal converts the request data into JSON format and transmits it to the server using the transmission means.

[0833] 3. The server receives and interprets the request:

[0834] The server receives the request, analyzes the contents, and retrieves the relevant information from the database.

[0835] 4. Server converts to Braille:

[0836] The text information acquired by the server is converted into Braille using a Braille conversion algorithm.

[0837] 5. The server sends the Braille data:

[0838] The server sends the Braille data to the terminal.

[0839] 6. The device displays the braille information:

[0840] The terminal displays the received braille data on a braille display device or braille printer.

[0841] The system of the present invention allows visually impaired users to easily obtain the information they need, improving the convenience of using stores.

[0842] The processing flow will be explained below.

[0843] Step 1:

[0844] The user accesses the store's terminal and requests the information they want to receive by voice input or text input. For example, if the user requests "how to handle the front desk" by voice input, the terminal converts this request into text data.

[0845] Step 2:

[0846] The device converts the textual request content into JSON format data along with identification information such as the user ID. This data is in preparation for sending the request content to the server.

[0847] Step 3:

[0848] The terminal sends JSON format data to the server using a transmission means. Specifically, it is sent as an HTTP request via the Internet or a local network.

[0849] Step 4:

[0850] The server receives the HTTP request and analyzes the request using text parsing technology to identify the type of service manual and specific items requested.

[0851] Step 5:

[0852] The server accesses the database and retrieves text information corresponding to the parsed request content. For example, text information corresponding to "how the front desk should respond" is retrieved from the database.

[0853] Step 6:

[0854] The server converts the acquired text into Braille using a Braille conversion algorithm, which converts each character into its corresponding Braille code.

[0855] Step 7:

[0856] The server converts the Braille data back into JSON format and sends it to the terminal using a transmission means, again as an HTTP response.

[0857] Step 8:

[0858] The device receives the HTTP response from the server, extracts the braille data, and then sends the data to a braille display device or braille printer, preparing to display the information to the user.

[0859] Step 9:

[0860] The terminal uses a Braille display device or Braille printer to display Braille information to the user. For example, the Braille information "⠋⠚⠴⠞... (How to handle the front desk)" is physically displayed to the user.

[0861] Step 10:

[0862] The user reads the braille information from the braille display device or braille printer and obtains the desired information. Through this process, visually impaired users can independently obtain the information they need when using a store.

[0863] Example 1

[0864] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0865] When visually impaired users need information, it is necessary to provide visual information in Braille format. However, current systems have limitations in how quickly and easily users can obtain the information they need in Braille while in a store. In particular, the entire process from inputting information to converting it into Braille and displaying it must be carried out efficiently. Another challenge is to improve user convenience.

[0866] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0867] In this invention, the server includes an analysis means for analyzing user input, an acquisition means for acquiring relevant information from a database, and a conversion means for converting the acquired information into Braille, thereby enabling visually impaired users to quickly and easily acquire necessary information in Braille, improving convenience when using a store.

[0868] "User" means a visually impaired individual who utilizes the system to obtain information.

[0869] "Input means" refers to a device such as a text input device or a voice input device that allows a user to input information.

[0870] "Terminal" refers to an electronic device that receives user input information and transmits it to a server as needed.

[0871] "Transmission means" refers to the communication means or protocol for transmitting information from the terminal to the server.

[0872] "Server" refers to a central management system that receives requests from users and processes the required information.

[0873] "Means for obtaining" refers to the functions and processes by which the server obtains the relevant information from the database.

[0874] "Analysis means" refers to the algorithms and processes that the server uses to analyze the information received from the terminal and understand its content.

[0875] "Conversion means" refers to the algorithm or system used by the server to convert the text information it acquires into Braille.

[0876] "Display means" refers to a device such as a braille display or braille printer that the terminal uses to display braille information to the user.

[0877] "Database" refers to a data storage that stores information desired by users and that a server accesses to retrieve that information.

[0878] MODE FOR CARRYING OUT THE INVENTION

[0879] This invention relates to a system that allows visually impaired users to obtain store information and service manuals in Braille format. This system receives user input, converts it into Braille information, and provides it to visually impaired users, making it easier for them to obtain information.

[0880] Overall system configuration

[0881] The system consists of the following main components:

[0882] 1. An input method for users to enter information

[0883] 2. A means for the terminal to receive input information and send it to the server

[0884] 3. A means for the server to analyze the received information and retrieve the information from the database

[0885] 4. A means for converting the information acquired by the server into Braille

[0886] 5. A means for the server to transmit the converted Braille information to the terminal.

[0887] 6. Display means for the terminal to display the transmitted Braille information to the user.

[0888] Input Method

[0889] Users use the store's terminals to input information. Input methods include text input and voice input. For example, if a user voice-inputs a request such as "How do I contact the front desk?", the terminal uses voice recognition software to convert this voice data into text.

[0890] Transmission method

[0891] The terminal sends the user's input data to the server. The data sent is in JSON format and includes information such as the request content and user ID. For example, the terminal sends "Request content: How the front desk should respond" and "User ID: 12345" to the server.

[0892] Analysis and acquisition methods

[0893] The server includes an analysis means for analyzing the received information. The analysis means analyzes the request, and the server uses an acquisition means for acquiring the relevant information from the database. For example, the server analyzes the request "How should the front desk handle this?" and acquires the relevant text information from the database.

[0894] Conversion Method

[0895] The server has a conversion means for converting the acquired text information into Braille. The Braille conversion algorithm converts the text into Braille code. For example, when "How to handle the front desk" is converted into Braille, it is converted as follows: "Fu" ⇒ ⠋, "Ro" ⇒ ⠚, "N" ⇒ ⠴, "To" ⇒ ⠞.

[0896] Braille data transmission method

[0897] The server has a transmitting means for transmitting the converted Braille data to the terminal. For example, the Braille data "⠋⠚⠴⠞(Front)..." is transmitted to the terminal.

[0898] Display means

[0899] The terminal has a display means for displaying the received Braille data to the user. The display means includes a Braille display device and a Braille printer. For example, the terminal displays "⠋⠚⠴⠞..." on the Braille display device.

[0900] Specific examples

[0901] As a concrete example, consider the following scenario:

[0902] 1. The user speaks "How should the front desk help?"

[0903] 2. The device converts the voice data into text data and sends it to the server.

[0904] 3. The server analyzes the received data and retrieves the relevant information from the database.

[0905] 4. The server converts the acquired information into Braille and sends it to the device.

[0906] 5. The device displays the braille information to the user.

[0907] Prompt Sentence Examples

[0908] Text input: "Please translate front desk instructions into Braille."

[0909] Voice input: "How can I help the front desk?"

[0910] The system of the present invention enables visually impaired users to quickly and easily obtain necessary information in Braille while in a store, improving convenience when using the store.

[0911] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0912] Step 1:

[0913] The user enters information

[0914] The user inputs information using an input means. Specifically, the user uses the device's voice input function to say, "How do I contact the front desk?" The device converts this voice data into text data. The input data is voice data, which is then converted into text data for further processing. Specifically, the voice recognition software analyzes the voice and generates the corresponding text.

[0915] Step 2:

[0916] The device sends a request to the server

[0917] The terminal receives the text data and prepares to send it to the server. Specifically, the terminal generates JSON data including "Request content: Front desk response method" and "User ID: 12345" and sends it to the server using the transmission means. The input data is text data, and the output data is JSON-formatted request data for the server.

[0918] Step 3:

[0919] The server receives and parses the request

[0920] The server receives the JSON formatted data sent from the terminal and analyzes its contents. The analysis means identifies that the request content is "how the front desk should respond." The input data is the JSON formatted request data, and the output data is the analyzed request content information. Specifically, the server parses the JSON data and identifies the request content.

[0921] Step 4:

[0922] The server retrieves information from the database

[0923] Based on the analysis results, the server retrieves the relevant information from the database. Specifically, the server executes a database query to retrieve the information "how the front desk should respond." The input data is the parsed request content, and the output data is the retrieved text information. Specifically, the server generates an SQL statement and executes the query against the database.

[0924] Step 5:

[0925] The server converts the information obtained into Braille.

[0926] The server converts the text information retrieved from the database into Braille. Using a conversion means, the text information is converted into Braille code to generate Braille data. The input data is text information, and the output data is Braille data. Specifically, the server applies a Braille conversion algorithm to convert the text into Braille code.

[0927] Step 6:

[0928] The server sends the Braille data to the device.

[0929] The server transmits the converted Braille data to the terminal. The Braille data is transmitted to the terminal using a transmission means. The input data is Braille data, and the output data is Braille data for the terminal. In concrete terms, the server generates formatted Braille data and transmits it to the terminal.

[0930] Step 7:

[0931] The device displays the braille information.

[0932] The terminal displays the received Braille data to the user. Using the display means, the Braille data is presented to the user via a Braille display device or Braille printer. The input data is Braille data, and the output data is information displayed in Braille. In concrete terms, the terminal displays the Braille information on a Braille display.

[0933] These specific processing steps enable visually impaired users to quickly and easily obtain the information they need in Braille, improving convenience when using stores.

[0934] (Application example 1)

[0935] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0936] One issue that visually impaired people face is that it has been difficult for them to easily obtain the information they need in public places and commercial facilities. Furthermore, the lack of a system that provides store information or service manuals in Braille limits access to information for visually impaired people. The current situation, where the usefulness of voice input interfaces is not being fully utilized, is also an issue that needs to be resolved.

[0937] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0938] In this invention, the server includes input means for a user to input voice or text, transmission means for transmitting the input information to the server, acquisition means for the server to analyze the received information and acquire corresponding data, conversion means for converting the acquired data into Braille, transmission means for transmitting the converted Braille information to the terminal, display means for the terminal to display the Braille information to the user, means for using an algorithm to convert voice data into text, and means for using an algorithm to convert text information into Braille. This enables visually impaired people to acquire necessary information using voice input and receive it in Braille.

[0939] "Voice" or "text" refers to the means used by a user to input information, where voice input refers to spoken sounds and text input refers to characters such as those on a keyboard.

[0940] "Input means" refers to a means by which a user inputs information into a terminal, and includes a voice recognition device, a keyboard, a touch screen, and the like.

[0941] "Transmission means" refers to a device or function for converting information received from the input means into a digital format and transmitting it to the server.

[0942] "Acquisition means" refers to the function of the server analyzing the received information and acquiring the corresponding data from a database or the like.

[0943] "Conversion means" refers to an algorithm or device for converting acquired data into another format, and specifically includes an algorithm for converting text data into Braille.

[0944] "Display means" refers to a device or function for displaying final information to the user, and includes a Braille display device and a Braille printer.

[0945] "Server" refers to a central computer system that receives requests from users, analyzes, converts, and transmits information.

[0946] The present invention relates to a system that enables visually impaired users to obtain store information and service information in Braille. This system assists visually impaired users in obtaining information by inputting information, converting it into Braille, and providing it to them.

[0947] Key Components of the System

[0948] The system consists of the following main components:

[0949] 1. Input means: A means by which a user inputs voice or text, including a voice recognition device, keyboard, or touch screen.

[0950] 2. Transmission means: A means for converting information received from the input means into a digital format and transmitting it to the server.

[0951] 3. Acquisition method: The method by which the server analyzes the information received and acquires the corresponding data from a database, etc.

[0952] 4. Conversion means: The algorithm or device that converts the acquired data into Braille.

[0953] 5. Display means: A Braille display device or Braille printer to display the final information to the user.

[0954] Program processing flow

[0955] 1. User inputs information by voice: The user launches the smartphone application and inputs the information they want to know by voice (e.g., "Please tell me the product description for shelf A"). The voice data is converted into text data using the smartphone's voice recognition function (e.g., Google Cloud Speech-to-Text API).

[0956] 2. The device sends the request to the server: The smartphone converts this text data into JSON format and sends it to the store's server.

[0957] 3. The server receives and interprets the request: The server analyzes the received JSON data and retrieves information corresponding to the request content from the database.

[0958] 4. The server converts the text information into Braille: The server applies an algorithm (e.g., Braille translation software such as Duxbury) to convert the acquired text data into Braille data.

[0959] 5. The server sends the Braille data: The server formats the Braille data into JSON format and sends it to the smartphone.

[0960] 6. The smartphone displays the braille information: The smartphone displays the received braille data on the screen using a braille display module (for example, an app such as BrailleBack).

[0961] Hardware and software used

[0962] Hardware:

[0963] Smartphone (iOS or Android)

[0964] Braille display devices (such as BrailleBack accessories for smartphones)

[0965] software:

[0966] Google Cloud Speech-to-Text API (voice recognition)

[0967] RESTful API (data communication with server)

[0968] Duxbury (Braille conversion)

[0969] Braille display app (such as BrailleBack)

[0970] Specific examples

[0971] For example, if a user says, "Tell me the product description for shelf A," the speech is converted to text using the Google Cloud Speech-to-Text API. The smartphone then sends this text data in JSON format to a server, which converts it into Braille and sends it back to the smartphone. The BrailleBack module on the smartphone displays the Braille information, allowing the user to read it.

[0972] Prompt Sentence Examples

[0973] "The user's voice input, 'Tell me the product description for shelf A,' is converted to text using the Google Cloud Speech-to-Text API and sent to the server in JSON format. The server retrieves the text information from the database and converts it into Braille using Duxbury. The Braille data is then sent to the smartphone and displayed in Braille using the BrailleBack module."

[0974] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0975] Step 1:

[0976] The user launches the smartphone application and inputs the information they want to know by voice. Using a speech recognition function (for example, Google Cloud Speech-to-Text API), the voice data is converted into text data. The input is the user's voice data, and the output is text data generated by speech recognition. Specifically, when the user says, "Tell me the product description for shelf A," this is displayed as text on the smartphone.

[0977] Step 2:

[0978] The device acquires the text data generated by speech recognition and converts it into JSON format. The input is the text data generated by speech recognition, and the output is JSON format data. Specifically, the text data "Please tell me the product description for shelf A" is converted into JSON format such as "request_text":"Please tell me the product description for shelf A".

[0979] Step 3:

[0980] The terminal sends the converted JSON format data to the server. Data communication is carried out by the transmission means. The input is JSON format data, and the output is sent to the server. Specifically, the JSON data "request_text":"Please tell me the product description for shelf A" is sent over the network.

[0981] Step 4:

[0982] The server analyzes the received JSON data and retrieves the corresponding data from the database. The database is operated by the retrieval means. The input is the JSON data received from the terminal, and the output is text data retrieved from the database. Specifically, the request "Tell me the product description for shelf A" is analyzed, and the corresponding product description is retrieved from the database.

[0983] Step 5:

[0984] The server converts the acquired text data into Braille. A conversion means applies a Braille translation algorithm (e.g., Duxbury). The input is text data acquired from the database, and the output is Braille data. Specifically, the text "Product description on shelf A" is converted into Braille code.

[0985] Step 6:

[0986] The server formats the data converted into Braille into JSON format and sends it to the terminal. Data communication is carried out by the transmission means. The input is Braille data, and the output is transmission to the terminal. Specifically, the Braille code is formatted into JSON format such as "Braille data":"⠑⠭⠁⠍⠏⠇⠑⠃⠐⠰..." and sent.

[0987] Step 7:

[0988] The terminal displays the received Braille data. A Braille display module (e.g., BrailleBack) is used to provide information to visually impaired people. The input is the Braille data received from the server, and the output is the display in Braille. Specifically, the user can read "Product description on shelf A" by touching the Braille on the smartphone screen.

[0989] In this way, visually impaired people can easily obtain information and improve convenience in stores.

[0990] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0991] This invention provides a system that enables visually impaired users to efficiently obtain store information and service manuals in Braille. Furthermore, this system incorporates an emotion engine that recognizes the user's emotions and provides optimal information based on the user's emotions.

[0992] Overall system picture

[0993] The system consists of the following main components:

[0994] 1. An input method for users to enter information

[0995] 2. A means by which the terminal receives user input and sends it to the server

[0996] 3. Transformation means for processing the information received by the server

[0997] 4. Emotion engine that recognizes user emotions

[0998] 5. Means for transmitting the converted Braille information to the terminal

[0999] 6. Display means by which the device displays braille information to the user

[1000] Description of Specific Embodiments

[1001] 1. User enters information

[1002] The user accesses the store's terminal and inputs information. This input method includes text input and voice input. The emotion engine also recognizes emotions from the user's voice and input content.

[1003] Examples:

[1004] When a user requests "how to handle the front desk" by voice input, the device converts this request into text data, and the emotion engine analyzes the user's emotions from the voice.

[1005] 2. The device sends a request to the server

[1006] The device sends the textual request content and the user's emotion data to the server using a transmission means. The data sent here is in JSON format and includes the request content, user ID, emotion data, etc.

[1007] Examples:

[1008] The terminal sends the "Request content: How the front desk should respond", "User ID", and "Emotion data" to the server.

[1009] 3. The server receives and interprets the request

[1010] The server receives the request from the device and analyzes its contents. This analysis includes text parsing technology and emotional data analysis. The server then retrieves the relevant service manual information from the database.

[1011] Examples:

[1012] The server analyzes the request "How should the front desk handle this?", retrieves the corresponding text information from the database, and determines the approach to provide the information based on the user's emotional data.

[1013] 4. The server converts the text into Braille

[1014] The server converts the acquired text into Braille using a Braille conversion algorithm, which converts each character into the corresponding Braille code.

[1015] Examples:

[1016] The server converts "How to handle the front desk" into Braille, converting it as follows: "Fu" ⇒ ⠋, "Ro" ⇒ ⠚, "N" ⇒ ⠴, "To" ⇒ ⠞.

[1017] 5. The server sends the Braille data to the device.

[1018] The server converts the Braille data back into JSON format and sends it to the device using a transmission method. This is also sent as an HTTP response. Additional information based on emotions may also be included.

[1019] Examples:

[1020] The braille data is formatted as "⠋⠚⠴⠞(Front)..." and sent to the device. Based on the emotion data, additional information on how the user can relax is also sent.

[1021] 6. The device displays the braille information

[1022] The terminal displays the received Braille data to the user. The display means includes a Braille display device and a Braille printer. In addition, information based on feedback from the emotion engine is also displayed.

[1023] Examples:

[1024] The terminal displays Braille information on "how to handle the front desk" to the user via a Braille display device, and also displays Braille information on "approaches to relax."

[1025] 7. The user receives the information

[1026] The user reads the braille information from a braille display device or braille printer and obtains the desired information. This process allows visually impaired users to efficiently obtain information and receive appropriate information according to their emotions.

[1027] The system of the present invention enables visually impaired users to efficiently obtain the information they need and provides optimal information according to their emotions, thereby increasing convenience and satisfaction.

[1028] The processing flow will be explained below.

[1029] Step 1:

[1030] A user accesses a terminal in a store and requests the necessary information by text input or voice input. At this time, the emotion engine analyzes the user's emotions from the user's voice and input content. For example, if a user requests "how to handle the front desk" by voice input, the terminal converts the request into text data, and the emotion engine extracts emotional data from the voice.

[1031] Step 2:

[1032] The device generates JSON format data containing the text of the request and the extracted emotion data. This data includes the request content, user ID, emotion data, etc.

[1033] Step 3:

[1034] The terminal sends JSON format data to the server using a transmission means. Specifically, it is sent as an HTTP request over the network.

[1035] Step 4:

[1036] The server receives the HTTP request and analyzes the request content and sentiment data using text parsing technology and sentiment analysis algorithms to identify the type of service manual requested and the user's sentiment.

[1037] Step 5:

[1038] Based on the analysis results, the server retrieves the relevant service manual information from the database. For example, text information on "how to handle a customer at the front desk" is retrieved from the database.

[1039] Step 6:

[1040] The server converts the acquired text into Braille using a Braille conversion algorithm, which converts each character into its corresponding Braille code.

[1041] Step 7:

[1042] The emotion engine determines the additional information needed based on the user's emotional data. For example, if the user is feeling stressed, information on approaches to relaxation will be added.

[1043] Step 8:

[1044] The server generates JSON data containing the Braille data and any additional information determined by the emotion engine, ready to be sent to the device.

[1045] Step 9:

[1046] The server generates JSON format data and sends it to the terminal using a transmission method. This is also sent as an HTTP response via the network.

[1047] Step 10:

[1048] The device receives the HTTP response from the server, extracts the braille data and additional information, and then sends the data to a braille display device or braille printer, preparing to display the information to the user.

[1049] Step 11:

[1050] The terminal uses a Braille display device or Braille printer to display Braille information to the user. Specifically, the Braille information "⠋⠚⠴⠞ (how the front desk handles the situation)" and the Braille information "approach to relaxation" based on emotions are displayed.

[1051] Step 12:

[1052] The user reads the braille information from a braille display or braille printer and obtains the desired information and additional useful information. This process allows visually impaired users to efficiently obtain the information they need and receive appropriate information according to their emotions.

[1053] Example 2

[1054] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1055] There are issues with the lack of efficient means for visually impaired users to obtain information, making it difficult to provide appropriate information. In addition, there is a lack of optimal information provision that responds to the user's emotions, which results in problems with how information is received and its convenience.

[1056] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1057] In this invention, the server includes an analysis means for analyzing received information and acquiring user emotion data, a conversion means for converting the analyzed information into Braille, and a transmission means for transmitting the converted Braille information to the terminal, thereby enabling appropriate information to be provided based on the user's emotion, and enabling visually impaired users to efficiently acquire information.

[1058] "Input means" refers to a means for a user to input information, and includes text input, voice input, and the like.

[1059] The "transmission means" is a means for transmitting the input information to the server, and typically transmits data via an HTTP request.

[1060] The "analysis means" refers to a means for analyzing the information received by the server and obtaining the user's emotional data, and includes text parsing and emotion analysis techniques.

[1061] The "conversion means" is a means for converting the information analyzed by the server into Braille, and uses a Braille conversion algorithm.

[1062] "Display means" refers to means by which the terminal displays Braille information to the user, and includes a Braille display device and a Braille printer.

[1063] MODE FOR CARRYING OUT THE INVENTION

[1064] This invention relates to a system that enables visually impaired users to efficiently obtain store information and service manuals in Braille. Furthermore, this system incorporates an emotion engine that recognizes the user's emotions and provides appropriate information based on the user's emotions.

[1065] Specific hardware and software configurations

[1066] The system consists of the following main components:

[1067] 1. Input Method: The means by which users input information includes text input and voice input, specifically voice recognition software (e.g., Google Speech-to-Text) and keyboards.

[1068] 2. Transmission method: An HTTP request is used to transmit the entered information to the server.

[1069] 3. Analysis method: The server analyzes the received information and obtains the user's emotional data using natural language processing (NLP) technology or emotion analysis software (e.g., Microsoft Azure Emotion API).

[1070] 4. Conversion method: A Braille conversion algorithm (e.g., Liblouis) is used as a means for converting the information analyzed by the server into Braille.

[1071] 5. Transmission method: HTTP responses are again used as a means to transmit the converted Braille information to the terminal.

[1072] 6. Display means: A Braille display device (e.g., Refreshable Braille Display) or a Braille printer is used as a means for the terminal to display Braille information to the user.

[1073] Specific example of system operation

[1074] For example, if a user accesses a store terminal and requests "how to handle the front desk" by voice input, the following operation will occur.

[1075] 1. The device is equipped with voice recognition software that converts the speech into text data. At the same time, an emotion engine analyzes the tone and speed of the user's voice to determine the user's emotions.

[1076] 2. The device packs the acquired text data and emotion data into JSON format and sends an HTTP POST request to the server.

[1077] 3. The server receives the JSON data and analyzes its contents. First, it uses natural language processing technology to understand the request and retrieves the relevant service manual information from the database. It also processes the emotion data and determines how to provide information based on the user's emotion.

[1078] 4. A Braille conversion algorithm is used to convert the text information obtained by the server into Braille.

[1079] 5. The generated braille data is packed again in JSON format and sent to the device as an HTTP response, optionally including additional information based on the emotion data.

[1080] 6. The terminal sends the received Braille data to a Braille display device for display to the user, which displays the Braille data so that the user can read it.

[1081] Examples of prompt statements

[1082] "This system allows visually impaired users to efficiently retrieve store information and service manuals in Braille. Please provide a typical user scenario."

[1083] "Explain how a user can request assistance from the front desk by voice, and a server will translate that information into Braille and provide it to them."

[1084] This allows visually impaired users to acquire information efficiently and provides appropriate information according to their emotions.

[1085] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1086] Step 1:

[1087] The user inputs information. The user requests "How should the front desk respond?" by voice into the terminal. The terminal uses voice recognition software to convert this voice into text data (input: voice data, output: text data). At the same time, the emotion engine analyzes the user's tone of voice and speaking style to obtain emotional data (input: voice data, output: emotional data). Specifically, when the user voice-inputs "How should the front desk respond?", the terminal's microphone captures the voice data, which is then converted into text data by the voice recognition software. At the same time, the emotion engine obtains the emotional data.

[1088] Step 2:

[1089] The device sends the request content and emotion data to the server. The device then packs the acquired text data and emotion data into JSON format and sends an HTTP POST request to the server (input: text data, emotion data, output: JSON format data). Specifically, the text data "How should the front desk handle this?" and the emotion data are sent to the server in JSON format as "{\"userID\": 12345, \"request\": \"How should the front desk handle this\", \"emotion\": \"neutral\"}".

[1090] Step 3:

[1091] The server receives and analyzes the request. The server receives the JSON data sent from the terminal and analyzes the content (input: JSON data, output: analyzed text data and emotion data). First, it uses natural language processing technology to understand the request content and retrieves the corresponding service manual information from a database (input: request content, output: service manual information). It then processes the emotion data and determines how to provide information based on the user's emotion (input: emotion data, output: information provision method). Specifically, the server analyzes the request for "how the front desk should respond," retrieves the corresponding text information from the database, and determines the appropriate information provision method based on the emotion data "neutral."

[1092] Step 4:

[1093] The server converts the acquired text information into Braille. The server converts the acquired text information into Braille code using a Braille conversion algorithm (input: text information, output: Braille data). Specifically, the server converts the string "How the front desk responds" into Braille as follows: "Fu" ⇒ ⠋, "Ro" ⇒ ⠚, "N" ⇒ ⠴, "To" ⇒ ⠞.

[1094] Step 5:

[1095] The server sends the braille data to the device. The generated braille data is packed again in JSON format and sent to the device as an HTTP response (input: braille data, output: JSON format data). If necessary, additional information based on the emotion data may be included. Specifically, the braille data "⠋⠚⠴⠞(front)..." is packed in JSON format and sent to the device as "{\"braille_data\": \"⠋⠚⠴⠞...\", \"additional_info\": \"Approaches to Relaxation\"}".

[1096] Step 6:

[1097] The device displays the braille data to the user. The device sends the received braille data to the braille display device and displays it to the user (input: braille data in JSON format, output: braille display). Specifically, the device sends the data "⠋⠚⠴⠞(front)..." to the braille display device, which then displays it to the user. Braille data for the additional information "Approaches to Relaxation" is also displayed.

[1098] Step 7:

[1099] The user reads the braille information. The user obtains the necessary information by reading the braille information from the braille display device (input: braille display data, output: information obtained by the user). Specifically, the user reads the braille information on "how the front desk will handle the situation" and the braille data on "approaches to relaxing" from the braille display device.

[1100] (Application example 2)

[1101] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1102] When visually impaired users efficiently obtain the information they need in stores and facilities, that information needs to be optimized according to their emotions. However, current systems are unable to provide information that takes the user's emotions into account, resulting in a lack of convenience and satisfaction. In addition, there are situations where it is difficult for users to quickly receive specific guidance information in Braille or audio.

[1103] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an input means for a user to input information, a transmission means for transmitting the input information to the server, a conversion means for converting the received information into Braille, a transmission means for transmitting the converted Braille information to the terminal, a display means for displaying the Braille information to the user, and an emotion recognition means for recognizing the user's emotion and providing optimal information. This enables visually impaired users to efficiently obtain the information they need in a store or facility in a form optimized according to their emotion.

[1104] An "input means" is a device or interface through which a user inputs information.

[1105] The "transmitting means" is a communication device or mechanism for transmitting input information to the server.

[1106] A "conversion means" is a device or algorithm that converts information received by the server into Braille.

[1107] "Display means" refers to a device or mechanism by which a terminal displays Braille information to a user.

[1108] The "emotion recognition means" is a device or system that recognizes the user's emotions and provides optimal information based on those emotions.

[1109] "Guide information" refers to information that a user needs in a facility or store, and specifically includes store information, service manuals, and the like.

[1110] The "server" is a computer system that analyzes the received information, converts it into Braille, and performs emotion recognition.

[1111] The realization of this system involves the following major components:

[1112] Overall system configuration

[1113] 1. "Input means" for users to enter information

[1114] 2. "Transmission method" to send the entered information to the server

[1115] 3. "Conversion means" that converts the information received by the server into Braille

[1116] 4. "Transmission means" for transmitting the converted Braille information to the terminal

[1117] 5. "Display means" by which the device displays Braille information to the user

[1118] 6. "Emotion recognition means" that recognizes the user's emotions and provides optimal information

[1119] Specific Embodiments of the System

[1120] 1. Enter your information

[1121] Users input their voice via smart glasses or a smartphone. The voice is converted into text data by speech recognition software. At this time, an emotion recognition engine analyzes the user's emotions from the voice data. The specific software used is EmotionRecognizer and a speech recognition library.

[1122] 2. Transmission of information

[1123] Once speech recognition and sentiment analysis are complete, the data is packaged in JSON format and sent to a server using the HTTP protocol. The server receives and analyzes this data using software such as a request sending library.

[1124] 3. Analysis of Information

[1125] The server analyzes the received request and retrieves the appropriate guidance information from a database. This analysis uses Natural Language Processing (NLP) technology, and determines the appropriate information provision approach based on the user's emotional data.

[1126] 4. Conversion to Braille

[1127] The acquired guidance information is converted into Braille data using a Braille conversion algorithm. Specifically, the BrailleConverter library is used. The conversion results are written as Braille code.

[1128] 5. Resending Information

[1129] The server then converts the Braille data back into JSON format and sends it to the device as an HTTP response, which also includes additional feedback information based on the emotion data.

[1130] 6. Display of Information

[1131] The device then provides the received Braille data to the user using a Braille display or Braille printer. Feedback based on emotion recognition is also displayed. This allows the user to read the necessary guidance information in Braille and receive optimal feedback based on their emotions.

[1132] Examples of concrete examples and prompts

[1133] For example, if a user speaks "What is the timetable for the next train?" at a train station information desk, the system will take the following steps:

[1134] 1. Convert the voice input "Please tell me the timetable for the next train" into text.

[1135] 2. The emotion recognition engine detects the user's impatience from the tone of the voice.

[1136] 3. The server now provides information about the next train time and adds a message to reassure anxious users.

[1137] 4. Smart glasses and smartphones will provide information through braille displays and audio guidance.

[1138] Example prompt for a generative AI model:

[1139] "A user says, 'What is the timetable for the next train?' Convert this speech to text, recognize the user's emotions, and suggest the best way to respond."

[1140] This format allows visually impaired users to efficiently obtain guidance information, and also enables optimal information provision that takes into consideration their emotions at the time.

[1141] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1142] Step 1:

[1143] The user uses smart glasses or a smartphone to input voice. The input voice is converted into text data by speech recognition software. Specifically, the user inputs "Please tell me the timetable for the next train," which is converted into text "Please tell me the timetable for the next train." At the same time, an emotion recognition engine analyzes the voice data and recognizes the user's emotion (in this case, impatience). The input is voice data and emotion data, and the output is text data and analyzed emotion data.

[1144] Step 2:

[1145] The device packages text data and emotion data in JSON format and sends it to the server via the HTTP protocol. Specifically, the user's voice is sent as text "Please tell me the timetable for the next train," along with emotion data. The input is text data and emotion data, and the output is an HTTP request to the server.

[1146] Step 3:

[1147] The server analyzes the received JSON data. The server uses Natural Language Processing (NLP) technology to understand the content of the text data and retrieves the corresponding guidance information from the database. For example, the server recognizes the request "next train timetable" and retrieves the next train timetable from the database. The input is JSON data, and the output is guidance information.

[1148] Step 4:

[1149] The server converts the acquired guidance information into Braille data using a Braille conversion algorithm. In this process, the BrailleConverter library is used to convert the text "next train timetable" into Braille. The input is the text data of the guidance information, and the output is Braille data.

[1150] Step 5:

[1151] The server converts the feedback information based on the Braille data and emotion data back into JSON format and sends it to the terminal as an HTTP response. Specifically, it contains the emotion feedback "⠞⠝⠋⠥. (Braille)" and "Please stay calm and wait for the next train." The input is the Braille data and emotion feedback information, and the output is an HTTP response.

[1152] Step 6:

[1153] The device analyzes the received data and displays the information to the user through a braille display or audio output. The user can read "⠞⠝⠋⠥." on the braille display or receive information about the next train time and a message to relax via audio. The input is an HTTP response, and the output is information in a format (braille or audio) that can be understood by visually impaired users.

[1154] As a format

[1155] It explains that each step in the process has data inputs and outputs, and details how the information that the user ultimately receives is generated.

[1156] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1157] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1158] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1159] [Fourth embodiment]

[1160] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1161] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1162] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1163] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1164] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1165] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1166] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1167] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1168] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1169] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1170] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1171] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1172] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1173] The present invention relates to a system that enables visually impaired users to obtain store information and service manuals in Braille. This system assists visually impaired users in obtaining information by inputting information, converting it into Braille, and providing it to them.

[1174] Overall system picture

[1175] The system consists of the following main components:

[1176] 1. An input method for users to enter information

[1177] 2. A means by which the terminal receives user input and sends it to the server

[1178] 3. Transformation means for processing the information received by the server

[1179] 4. A means for transmitting the converted Braille information to the terminal

[1180] 5. Display method for displaying braille information on the device

[1181] Description of Specific Embodiments

[1182] 1. User enters information

[1183] A user accesses a terminal in a store and inputs information. This input method includes text input and voice input.

[1184] Examples:

[1185] When a user requests "how to handle the front desk" by voice input, the terminal converts this request into text data.

[1186] 2. The device sends a request to the server

[1187] The terminal receives the user's request and sends it to the server using a transmission means. The data sent here is JSON format data that includes the request content, user ID, etc.

[1188] Examples:

[1189] The terminal sends "Request content: How the front desk should respond" and "User ID" to the server.

[1190] 3. The server receives and interprets the request

[1191] The server receives the request from the device, analyzes its contents, and retrieves the relevant information from the database.

[1192] Examples:

[1193] The server analyzes the request "How should the front desk handle this?" and retrieves the corresponding text information from the database.

[1194] 4. The server converts the text into Braille

[1195] The server converts the acquired text into Braille, applying a Braille conversion algorithm to convert characters into Braille codes.

[1196] Examples:

[1197] The server converts the "front desk's response instructions" into Braille. For example, "Fu" is converted to ⠋, "Ro" to ⠚, "N" to ⠴, and "To" to ⠞.

[1198] 5. The server sends the Braille data to the device.

[1199] The server transmits the data converted into Braille to the terminal, and the transmission means is used to provide the formatted Braille data to the terminal.

[1200] Examples:

[1201] The braille data is formatted as "⠋⠚⠴⠞(Front)..." and sent to the device.

[1202] 6. The device displays the braille information

[1203] The terminal displays the received Braille data to the user. The display means includes a Braille display device and a Braille printer.

[1204] Examples:

[1205] The terminal displays the Braille information of "How to handle the front desk" to the user via the Braille display device.

[1206] Program processing flow explanation

[1207] The processing flow of the program of this system will now be described in detail.

[1208] 1. User enters information:

[1209] The user enters text or voice input.

[1210] The terminal receives the input information and generates request data.

[1211] 2. The device sends a request to the server:

[1212] The terminal converts the request data into JSON format and transmits it to the server using the transmission means.

[1213] 3. The server receives and interprets the request:

[1214] The server receives the request, analyzes the contents, and retrieves the relevant information from the database.

[1215] 4. Server converts to Braille:

[1216] The text information acquired by the server is converted into Braille using a Braille conversion algorithm.

[1217] 5. The server sends the Braille data:

[1218] The server sends the Braille data to the terminal.

[1219] 6. The device displays the braille information:

[1220] The terminal displays the received braille data on a braille display device or braille printer.

[1221] The system of the present invention allows visually impaired users to easily obtain the information they need, improving the convenience of using stores.

[1222] The processing flow will be explained below.

[1223] Step 1:

[1224] The user accesses the store's terminal and requests the information they want to receive by voice input or text input. For example, if the user requests "how to handle the front desk" by voice input, the terminal converts this request into text data.

[1225] Step 2:

[1226] The device converts the textual request content into JSON format data along with identification information such as the user ID. This data is in preparation for sending the request content to the server.

[1227] Step 3:

[1228] The terminal sends JSON format data to the server using a transmission means. Specifically, it is sent as an HTTP request via the Internet or a local network.

[1229] Step 4:

[1230] The server receives the HTTP request and analyzes the request using text parsing technology to identify the type of service manual and specific items requested.

[1231] Step 5:

[1232] The server accesses the database and retrieves text information corresponding to the parsed request content. For example, text information corresponding to "how the front desk should respond" is retrieved from the database.

[1233] Step 6:

[1234] The server converts the acquired text into Braille using a Braille conversion algorithm, which converts each character into its corresponding Braille code.

[1235] Step 7:

[1236] The server converts the Braille data back into JSON format and sends it to the terminal using a transmission means, again as an HTTP response.

[1237] Step 8:

[1238] The device receives the HTTP response from the server, extracts the braille data, and then sends the data to a braille display device or braille printer, preparing to display the information to the user.

[1239] Step 9:

[1240] The terminal uses a Braille display device or Braille printer to display Braille information to the user. For example, the Braille information "⠋⠚⠴⠞... (How to handle the front desk)" is physically displayed to the user.

[1241] Step 10:

[1242] The user reads the braille information from the braille display device or braille printer and obtains the desired information. Through this process, visually impaired users can independently obtain the information they need when using a store.

[1243] Example 1

[1244] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1245] When visually impaired users need information, it is necessary to provide visual information in Braille format. However, current systems have limitations in how quickly and easily users can obtain the information they need in Braille while in a store. In particular, the entire process from inputting information to converting it into Braille and displaying it must be carried out efficiently. Another challenge is to improve user convenience.

[1246] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1247] In this invention, the server includes an analysis means for analyzing user input, an acquisition means for acquiring relevant information from a database, and a conversion means for converting the acquired information into Braille, thereby enabling visually impaired users to quickly and easily acquire necessary information in Braille, improving convenience when using a store.

[1248] "User" means a visually impaired individual who utilizes the system to obtain information.

[1249] "Input means" refers to a device such as a text input device or a voice input device that allows a user to input information.

[1250] "Terminal" refers to an electronic device that receives user input information and transmits it to a server as needed.

[1251] "Transmission means" refers to the communication means or protocol for transmitting information from the terminal to the server.

[1252] "Server" refers to a central management system that receives requests from users and processes the required information.

[1253] "Means for obtaining" refers to the functions and processes by which the server obtains the relevant information from the database.

[1254] "Analysis means" refers to the algorithms and processes that the server uses to analyze the information received from the terminal and understand its content.

[1255] "Conversion means" refers to the algorithm or system used by the server to convert the text information it acquires into Braille.

[1256] "Display means" refers to a device such as a braille display or braille printer that the terminal uses to display braille information to the user.

[1257] "Database" refers to a data storage that stores information desired by users and that a server accesses to retrieve that information.

[1258] MODE FOR CARRYING OUT THE INVENTION

[1259] This invention relates to a system that allows visually impaired users to obtain store information and service manuals in Braille format. This system receives user input, converts it into Braille information, and provides it to visually impaired users, making it easier for them to obtain information.

[1260] Overall system configuration

[1261] The system consists of the following main components:

[1262] 1. An input method for users to enter information

[1263] 2. A means for the terminal to receive input information and send it to the server

[1264] 3. A means for the server to analyze the received information and retrieve the information from the database

[1265] 4. A means for converting the information acquired by the server into Braille

[1266] 5. A means for the server to transmit the converted Braille information to the terminal.

[1267] 6. Display means for the terminal to display the transmitted Braille information to the user.

[1268] Input Method

[1269] Users use the store's terminals to input information. Input methods include text input and voice input. For example, if a user voice-inputs a request such as "How do I contact the front desk?", the terminal uses voice recognition software to convert this voice data into text.

[1270] Transmission method

[1271] The terminal sends the user's input data to the server. The data sent is in JSON format and includes information such as the request content and user ID. For example, the terminal sends "Request content: How the front desk should respond" and "User ID: 12345" to the server.

[1272] Analysis and acquisition methods

[1273] The server includes an analysis means for analyzing the received information. The analysis means analyzes the request, and the server uses an acquisition means for acquiring the relevant information from the database. For example, the server analyzes the request "How should the front desk handle this?" and acquires the relevant text information from the database.

[1274] Conversion Method

[1275] The server has a conversion means for converting the acquired text information into Braille. The Braille conversion algorithm converts the text into Braille code. For example, when "How to handle the front desk" is converted into Braille, it is converted as follows: "Fu" ⇒ ⠋, "Ro" ⇒ ⠚, "N" ⇒ ⠴, "To" ⇒ ⠞.

[1276] Braille data transmission method

[1277] The server has a transmitting means for transmitting the converted Braille data to the terminal. For example, the Braille data "⠋⠚⠴⠞(Front)..." is transmitted to the terminal.

[1278] Display means

[1279] The terminal has a display means for displaying the received Braille data to the user. The display means includes a Braille display device and a Braille printer. For example, the terminal displays "⠋⠚⠴⠞..." on the Braille display device.

[1280] Specific examples

[1281] As a concrete example, consider the following scenario:

[1282] 1. The user speaks "How should the front desk help?"

[1283] 2. The device converts the voice data into text data and sends it to the server.

[1284] 3. The server analyzes the received data and retrieves the relevant information from the database.

[1285] 4. The server converts the acquired information into Braille and sends it to the device.

[1286] 5. The device displays the braille information to the user.

[1287] Prompt Sentence Examples

[1288] Text input: "Please translate front desk instructions into Braille."

[1289] Voice input: "How can I help the front desk?"

[1290] The system of the present invention enables visually impaired users to quickly and easily obtain necessary information in Braille while in a store, improving convenience when using the store.

[1291] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1292] Step 1:

[1293] The user enters information

[1294] The user inputs information using an input means. Specifically, the user uses the device's voice input function to say, "How do I contact the front desk?" The device converts this voice data into text data. The input data is voice data, which is then converted into text data for further processing. Specifically, the voice recognition software analyzes the voice and generates the corresponding text.

[1295] Step 2:

[1296] The device sends a request to the server

[1297] The terminal receives the text data and prepares to send it to the server. Specifically, the terminal generates JSON data including "Request content: Front desk response method" and "User ID: 12345" and sends it to the server using the transmission means. The input data is text data, and the output data is JSON-formatted request data for the server.

[1298] Step 3:

[1299] The server receives and parses the request

[1300] The server receives the JSON formatted data sent from the terminal and analyzes its contents. The analysis means identifies that the request content is "how the front desk should respond." The input data is the JSON formatted request data, and the output data is the analyzed request content information. Specifically, the server parses the JSON data and identifies the request content.

[1301] Step 4:

[1302] The server retrieves information from the database

[1303] Based on the analysis results, the server retrieves the relevant information from the database. Specifically, the server executes a database query to retrieve the information "how the front desk should respond." The input data is the parsed request content, and the output data is the retrieved text information. Specifically, the server generates an SQL statement and executes the query against the database.

[1304] Step 5:

[1305] The server converts the information obtained into Braille.

[1306] The server converts the text information retrieved from the database into Braille. Using a conversion means, the text information is converted into Braille code to generate Braille data. The input data is text information, and the output data is Braille data. Specifically, the server applies a Braille conversion algorithm to convert the text into Braille code.

[1307] Step 6:

[1308] The server sends the Braille data to the device.

[1309] The server transmits the converted Braille data to the terminal. The Braille data is transmitted to the terminal using a transmission means. The input data is Braille data, and the output data is Braille data for the terminal. In concrete terms, the server generates formatted Braille data and transmits it to the terminal.

[1310] Step 7:

[1311] The device displays the braille information.

[1312] The terminal displays the received Braille data to the user. Using the display means, the Braille data is presented to the user via a Braille display device or Braille printer. The input data is Braille data, and the output data is information displayed in Braille. In concrete terms, the terminal displays the Braille information on a Braille display.

[1313] These specific processing steps enable visually impaired users to quickly and easily obtain the information they need in Braille, improving convenience when using stores.

[1314] (Application example 1)

[1315] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1316] One issue that visually impaired people face is that it has been difficult for them to easily obtain the information they need in public places and commercial facilities. Furthermore, the lack of a system that provides store information or service manuals in Braille limits access to information for visually impaired people. The current situation, where the usefulness of voice input interfaces is not being fully utilized, is also an issue that needs to be resolved.

[1317] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1318] In this invention, the server includes input means for a user to input voice or text, transmission means for transmitting the input information to the server, acquisition means for the server to analyze the received information and acquire corresponding data, conversion means for converting the acquired data into Braille, transmission means for transmitting the converted Braille information to the terminal, display means for the terminal to display the Braille information to the user, means for using an algorithm to convert voice data into text, and means for using an algorithm to convert text information into Braille. This enables visually impaired people to acquire necessary information using voice input and receive it in Braille.

[1319] "Voice" or "text" refers to the means used by a user to input information, where voice input refers to spoken sounds and text input refers to characters such as those on a keyboard.

[1320] "Input means" refers to a means by which a user inputs information into a terminal, and includes a voice recognition device, a keyboard, a touch screen, and the like.

[1321] "Transmission means" refers to a device or function for converting information received from the input means into a digital format and transmitting it to the server.

[1322] "Acquisition means" refers to the function of the server analyzing the received information and acquiring the corresponding data from a database or the like.

[1323] "Conversion means" refers to an algorithm or device for converting acquired data into another format, and specifically includes an algorithm for converting text data into Braille.

[1324] "Display means" refers to a device or function for displaying final information to the user, and includes a Braille display device and a Braille printer.

[1325] "Server" refers to a central computer system that receives requests from users, analyzes, converts, and transmits information.

[1326] The present invention relates to a system that enables visually impaired users to obtain store information and service information in Braille. This system assists visually impaired users in obtaining information by inputting information, converting it into Braille, and providing it to them.

[1327] Key Components of the System

[1328] The system consists of the following main components:

[1329] 1. Input means: A means by which a user inputs voice or text, including a voice recognition device, keyboard, or touch screen.

[1330] 2. Transmission means: A means for converting information received from the input means into a digital format and transmitting it to the server.

[1331] 3. Acquisition method: The method by which the server analyzes the information received and acquires the corresponding data from a database, etc.

[1332] 4. Conversion means: The algorithm or device that converts the acquired data into Braille.

[1333] 5. Display means: A Braille display device or Braille printer to display the final information to the user.

[1334] Program processing flow

[1335] 1. User inputs information by voice: The user launches the smartphone application and inputs the information they want to know by voice (e.g., "Please tell me the product description for shelf A"). The voice data is converted into text data using the smartphone's voice recognition function (e.g., Google Cloud Speech-to-Text API).

[1336] 2. The device sends the request to the server: The smartphone converts this text data into JSON format and sends it to the store's server.

[1337] 3. The server receives and interprets the request: The server analyzes the received JSON data and retrieves information corresponding to the request content from the database.

[1338] 4. The server converts the text information into Braille: The server applies an algorithm (e.g., Braille translation software such as Duxbury) to convert the acquired text data into Braille data.

[1339] 5. The server sends the Braille data: The server formats the Braille data into JSON format and sends it to the smartphone.

[1340] 6. The smartphone displays the braille information: The smartphone displays the received braille data on the screen using a braille display module (for example, an app such as BrailleBack).

[1341] Hardware and software used

[1342] Hardware:

[1343] Smartphone (iOS or Android)

[1344] Braille display devices (such as BrailleBack accessories for smartphones)

[1345] software:

[1346] Google Cloud Speech-to-Text API (voice recognition)

[1347] RESTful API (data communication with server)

[1348] Duxbury (Braille conversion)

[1349] Braille display app (such as BrailleBack)

[1350] Specific examples

[1351] For example, if a user says, "Tell me the product description for shelf A," the speech is converted to text using the Google Cloud Speech-to-Text API. The smartphone then sends this text data in JSON format to a server, which converts it into Braille and sends it back to the smartphone. The BrailleBack module on the smartphone displays the Braille information, allowing the user to read it.

[1352] Prompt Sentence Examples

[1353] "The user's voice input, 'Tell me the product description for shelf A,' is converted to text using the Google Cloud Speech-to-Text API and sent to the server in JSON format. The server retrieves the text information from the database and converts it into Braille using Duxbury. The Braille data is then sent to the smartphone and displayed in Braille using the BrailleBack module."

[1354] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1355] Step 1:

[1356] The user launches the smartphone application and inputs the information they want to know by voice. Using a speech recognition function (for example, Google Cloud Speech-to-Text API), the voice data is converted into text data. The input is the user's voice data, and the output is text data generated by speech recognition. Specifically, when the user says, "Tell me the product description for shelf A," this is displayed as text on the smartphone.

[1357] Step 2:

[1358] The device acquires the text data generated by speech recognition and converts it into JSON format. The input is the text data generated by speech recognition, and the output is JSON format data. Specifically, the text data "Please tell me the product description for shelf A" is converted into JSON format such as "request_text":"Please tell me the product description for shelf A".

[1359] Step 3:

[1360] The terminal sends the converted JSON format data to the server. Data communication is carried out by the transmission means. The input is JSON format data, and the output is sent to the server. Specifically, the JSON data "request_text":"Please tell me the product description for shelf A" is sent over the network.

[1361] Step 4:

[1362] The server analyzes the received JSON data and retrieves the corresponding data from the database. The database is operated by the retrieval means. The input is the JSON data received from the terminal, and the output is text data retrieved from the database. Specifically, the request "Tell me the product description for shelf A" is analyzed, and the corresponding product description is retrieved from the database.

[1363] Step 5:

[1364] The server converts the acquired text data into Braille. A conversion means applies a Braille translation algorithm (e.g., Duxbury). The input is text data acquired from the database, and the output is Braille data. Specifically, the text "Product description on shelf A" is converted into Braille code.

[1365] Step 6:

[1366] The server formats the data converted into Braille into JSON format and sends it to the terminal. Data communication is carried out by the transmission means. The input is Braille data, and the output is transmission to the terminal. Specifically, the Braille code is formatted into JSON format such as "Braille data":"⠑⠭⠁⠍⠏⠇⠑⠃⠐⠰..." and sent.

[1367] Step 7:

[1368] The terminal displays the received Braille data. A Braille display module (e.g., BrailleBack) is used to provide information to visually impaired people. The input is the Braille data received from the server, and the output is the display in Braille. Specifically, the user can read "Product description on shelf A" by touching the Braille on the smartphone screen.

[1369] In this way, visually impaired people can easily obtain information and improve convenience in stores.

[1370] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1371] This invention provides a system that enables visually impaired users to efficiently obtain store information and service manuals in Braille. Furthermore, this system incorporates an emotion engine that recognizes the user's emotions and provides optimal information based on the user's emotions.

[1372] Overall system picture

[1373] The system consists of the following main components:

[1374] 1. An input method for users to enter information

[1375] 2. A means by which the terminal receives user input and sends it to the server

[1376] 3. Transformation means for processing the information received by the server

[1377] 4. Emotion engine that recognizes user emotions

[1378] 5. Means for transmitting the converted Braille information to the terminal

[1379] 6. Display means by which the device displays braille information to the user

[1380] Description of Specific Embodiments

[1381] 1. User enters information

[1382] The user accesses the store's terminal and inputs information. This input method includes text input and voice input. The emotion engine also recognizes emotions from the user's voice and input content.

[1383] Examples:

[1384] When a user requests "how to handle the front desk" by voice input, the device converts this request into text data, and the emotion engine analyzes the user's emotions from the voice.

[1385] 2. The device sends a request to the server

[1386] The device sends the textual request content and the user's emotion data to the server using a transmission means. The data sent here is in JSON format and includes the request content, user ID, emotion data, etc.

[1387] Examples:

[1388] The terminal sends the "Request content: How the front desk should respond", "User ID", and "Emotion data" to the server.

[1389] 3. The server receives and interprets the request

[1390] The server receives the request from the device and analyzes its contents. This analysis includes text parsing technology and emotional data analysis. The server then retrieves the relevant service manual information from the database.

[1391] Examples:

[1392] The server analyzes the request "How should the front desk handle this?", retrieves the corresponding text information from the database, and determines the approach to provide the information based on the user's emotional data.

[1393] 4. The server converts the text into Braille

[1394] The server converts the acquired text into Braille using a Braille conversion algorithm, which converts each character into the corresponding Braille code.

[1395] Examples:

[1396] The server converts "How to handle the front desk" into Braille, converting it as follows: "Fu" ⇒ ⠋, "Ro" ⇒ ⠚, "N" ⇒ ⠴, "To" ⇒ ⠞.

[1397] 5. The server sends the Braille data to the device.

[1398] The server converts the Braille data back into JSON format and sends it to the device using a transmission method. This is also sent as an HTTP response. Additional information based on emotions may also be included.

[1399] Examples:

[1400] The braille data is formatted as "⠋⠚⠴⠞(Front)..." and sent to the device. Based on the emotion data, additional information on how the user can relax is also sent.

[1401] 6. The device displays the braille information

[1402] The terminal displays the received Braille data to the user. The display means includes a Braille display device and a Braille printer. In addition, information based on feedback from the emotion engine is also displayed.

[1403] Examples:

[1404] The terminal displays Braille information on "how to handle the front desk" to the user via a Braille display device, and also displays Braille information on "approaches to relax."

[1405] 7. The user receives the information

[1406] The user reads the braille information from a braille display device or braille printer and obtains the desired information. This process allows visually impaired users to efficiently obtain information and receive appropriate information according to their emotions.

[1407] The system of the present invention enables visually impaired users to efficiently obtain the information they need and provides optimal information according to their emotions, thereby increasing convenience and satisfaction.

[1408] The processing flow will be explained below.

[1409] Step 1:

[1410] A user accesses a terminal in a store and requests the necessary information by text input or voice input. At this time, the emotion engine analyzes the user's emotions from the user's voice and input content. For example, if a user requests "how to handle the front desk" by voice input, the terminal converts the request into text data, and the emotion engine extracts emotional data from the voice.

[1411] Step 2:

[1412] The device generates JSON format data containing the text of the request and the extracted emotion data. This data includes the request content, user ID, emotion data, etc.

[1413] Step 3:

[1414] The terminal sends JSON format data to the server using a transmission means. Specifically, it is sent as an HTTP request over the network.

[1415] Step 4:

[1416] The server receives the HTTP request and analyzes the request content and sentiment data using text parsing technology and sentiment analysis algorithms to identify the type of service manual requested and the user's sentiment.

[1417] Step 5:

[1418] Based on the analysis results, the server retrieves the relevant service manual information from the database. For example, text information on "how to handle a customer at the front desk" is retrieved from the database.

[1419] Step 6:

[1420] The server converts the acquired text into Braille using a Braille conversion algorithm, which converts each character into its corresponding Braille code.

[1421] Step 7:

[1422] The emotion engine determines the additional information needed based on the user's emotional data. For example, if the user is feeling stressed, information on approaches to relaxation will be added.

[1423] Step 8:

[1424] The server generates JSON data containing the Braille data and any additional information determined by the emotion engine, ready to be sent to the device.

[1425] Step 9:

[1426] The server generates JSON format data and sends it to the terminal using a transmission method. This is also sent as an HTTP response via the network.

[1427] Step 10:

[1428] The device receives the HTTP response from the server, extracts the braille data and additional information, and then sends the data to a braille display device or braille printer, preparing to display the information to the user.

[1429] Step 11:

[1430] The terminal uses a Braille display device or Braille printer to display Braille information to the user. Specifically, the Braille information "⠋⠚⠴⠞ (how the front desk handles the situation)" and the Braille information "approach to relaxation" based on emotions are displayed.

[1431] Step 12:

[1432] The user reads the braille information from a braille display or braille printer and obtains the desired information and additional useful information. This process allows visually impaired users to efficiently obtain the information they need and receive appropriate information according to their emotions.

[1433] Example 2

[1434] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1435] There are issues with the lack of efficient means for visually impaired users to obtain information, making it difficult to provide appropriate information. In addition, there is a lack of optimal information provision that responds to the user's emotions, which results in problems with how information is received and its convenience.

[1436] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1437] In this invention, the server includes an analysis means for analyzing received information and acquiring user emotion data, a conversion means for converting the analyzed information into Braille, and a transmission means for transmitting the converted Braille information to the terminal, thereby enabling appropriate information to be provided based on the user's emotion, and enabling visually impaired users to efficiently acquire information.

[1438] "Input means" refers to a means for a user to input information, and includes text input, voice input, and the like.

[1439] The "transmission means" is a means for transmitting the input information to the server, and typically transmits data via an HTTP request.

[1440] The "analysis means" refers to a means for analyzing the information received by the server and obtaining the user's emotional data, and includes text parsing and emotion analysis techniques.

[1441] The "conversion means" is a means for converting the information analyzed by the server into Braille, and uses a Braille conversion algorithm.

[1442] "Display means" refers to means by which the terminal displays Braille information to the user, and includes a Braille display device and a Braille printer.

[1443] MODE FOR CARRYING OUT THE INVENTION

[1444] This invention relates to a system that enables visually impaired users to efficiently obtain store information and service manuals in Braille. Furthermore, this system incorporates an emotion engine that recognizes the user's emotions and provides appropriate information based on the user's emotions.

[1445] Specific hardware and software configurations

[1446] The system consists of the following main components:

[1447] 1. Input Method: The means by which users input information includes text input and voice input, specifically voice recognition software (e.g., Google Speech-to-Text) and keyboards.

[1448] 2. Transmission method: An HTTP request is used to transmit the entered information to the server.

[1449] 3. Analysis method: The server analyzes the received information and obtains the user's emotional data using natural language processing (NLP) technology or emotion analysis software (e.g., Microsoft Azure Emotion API).

[1450] 4. Conversion method: A Braille conversion algorithm (e.g., Liblouis) is used as a means for converting the information analyzed by the server into Braille.

[1451] 5. Transmission method: HTTP responses are again used as a means to transmit the converted Braille information to the terminal.

[1452] 6. Display means: A Braille display device (e.g., Refreshable Braille Display) or a Braille printer is used as a means for the terminal to display Braille information to the user.

[1453] Specific example of system operation

[1454] For example, if a user accesses a store terminal and requests "how to handle the front desk" by voice input, the following operation will occur.

[1455] 1. The device is equipped with voice recognition software that converts the speech into text data. At the same time, an emotion engine analyzes the tone and speed of the user's voice to determine the user's emotions.

[1456] 2. The device packs the acquired text data and emotion data into JSON format and sends an HTTP POST request to the server.

[1457] 3. The server receives the JSON data and analyzes its contents. First, it uses natural language processing technology to understand the request and retrieves the relevant service manual information from the database. It also processes the emotion data and determines how to provide information based on the user's emotion.

[1458] 4. A Braille conversion algorithm is used to convert the text information obtained by the server into Braille.

[1459] 5. The generated braille data is packed again in JSON format and sent to the device as an HTTP response, optionally including additional information based on the emotion data.

[1460] 6. The terminal sends the received Braille data to a Braille display device for display to the user, which displays the Braille data so that the user can read it.

[1461] Examples of prompt statements

[1462] "This system allows visually impaired users to efficiently retrieve store information and service manuals in Braille. Please provide a typical user scenario."

[1463] "Explain how a user can request assistance from the front desk by voice, and a server will translate that information into Braille and provide it to them."

[1464] This allows visually impaired users to acquire information efficiently and provides appropriate information according to their emotions.

[1465] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1466] Step 1:

[1467] The user inputs information. The user requests "How should the front desk respond?" by voice into the terminal. The terminal uses voice recognition software to convert this voice into text data (input: voice data, output: text data). At the same time, the emotion engine analyzes the user's tone of voice and speaking style to obtain emotional data (input: voice data, output: emotional data). Specifically, when the user voice-inputs "How should the front desk respond?", the terminal's microphone captures the voice data, which is then converted into text data by the voice recognition software. At the same time, the emotion engine obtains the emotional data.

[1468] Step 2:

[1469] The device sends the request content and emotion data to the server. The device then packs the acquired text data and emotion data into JSON format and sends an HTTP POST request to the server (input: text data, emotion data, output: JSON format data). Specifically, the text data "How should the front desk handle this?" and the emotion data are sent to the server in JSON format as "{\"userID\": 12345, \"request\": \"How should the front desk handle this\", \"emotion\": \"neutral\"}".

[1470] Step 3:

[1471] The server receives and analyzes the request. The server receives the JSON data sent from the terminal and analyzes the content (input: JSON data, output: analyzed text data and emotion data). First, it uses natural language processing technology to understand the request content and retrieves the corresponding service manual information from a database (input: request content, output: service manual information). It then processes the emotion data and determines how to provide information based on the user's emotion (input: emotion data, output: information provision method). Specifically, the server analyzes the request for "how the front desk should respond," retrieves the corresponding text information from the database, and determines the appropriate information provision method based on the emotion data "neutral."

[1472] Step 4:

[1473] The server converts the acquired text information into Braille. The server converts the acquired text information into Braille code using a Braille conversion algorithm (input: text information, output: Braille data). Specifically, the server converts the string "How the front desk responds" into Braille as follows: "Fu" ⇒ ⠋, "Ro" ⇒ ⠚, "N" ⇒ ⠴, "To" ⇒ ⠞.

[1474] Step 5:

[1475] The server sends the braille data to the device. The generated braille data is packed again in JSON format and sent to the device as an HTTP response (input: braille data, output: JSON format data). If necessary, additional information based on the emotion data may be included. Specifically, the braille data "⠋⠚⠴⠞(front)..." is packed in JSON format and sent to the device as "{\"braille_data\": \"⠋⠚⠴⠞...\", \"additional_info\": \"Approaches to Relaxation\"}".

[1476] Step 6:

[1477] The device displays the braille data to the user. The device sends the received braille data to the braille display device and displays it to the user (input: braille data in JSON format, output: braille display). Specifically, the device sends the data "⠋⠚⠴⠞(front)..." to the braille display device, which then displays it to the user. Braille data for the additional information "Approaches to Relaxation" is also displayed.

[1478] Step 7:

[1479] The user reads the braille information. The user obtains the necessary information by reading the braille information from the braille display device (input: braille display data, output: information obtained by the user). Specifically, the user reads the braille information on "how the front desk will handle the situation" and the braille data on "approaches to relaxing" from the braille display device.

[1480] (Application example 2)

[1481] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1482] When visually impaired users efficiently obtain the information they need in stores and facilities, that information needs to be optimized according to their emotions. However, current systems are unable to provide information that takes the user's emotions into account, resulting in a lack of convenience and satisfaction. In addition, there are situations where it is difficult for users to quickly receive specific guidance information in Braille or audio.

[1483] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an input means for a user to input information, a transmission means for transmitting the input information to the server, a conversion means for converting the received information into Braille, a transmission means for transmitting the converted Braille information to the terminal, a display means for displaying the Braille information to the user, and an emotion recognition means for recognizing the user's emotion and providing optimal information. This enables visually impaired users to efficiently obtain the information they need in a store or facility in a form optimized according to their emotion.

[1484] An "input means" is a device or interface through which a user inputs information.

[1485] The "transmitting means" is a communication device or mechanism for transmitting input information to the server.

[1486] A "conversion means" is a device or algorithm that converts information received by the server into Braille.

[1487] "Display means" refers to a device or mechanism by which a terminal displays Braille information to a user.

[1488] The "emotion recognition means" is a device or system that recognizes the user's emotions and provides optimal information based on those emotions.

[1489] "Guide information" refers to information that a user needs in a facility or store, and specifically includes store information, service manuals, and the like.

[1490] The "server" is a computer system that analyzes the received information, converts it into Braille, and performs emotion recognition.

[1491] The realization of this system involves the following major components:

[1492] Overall system configuration

[1493] 1. "Input means" for users to enter information

[1494] 2. "Transmission method" to send the entered information to the server

[1495] 3. "Conversion means" that converts the information received by the server into Braille

[1496] 4. "Transmission means" for transmitting the converted Braille information to the terminal

[1497] 5. "Display means" by which the device displays Braille information to the user

[1498] 6. "Emotion recognition means" that recognizes the user's emotions and provides optimal information

[1499] Specific Embodiments of the System

[1500] 1. Enter your information

[1501] Users input their voice via smart glasses or a smartphone. The voice is converted into text data by speech recognition software. At this time, an emotion recognition engine analyzes the user's emotions from the voice data. The specific software used is EmotionRecognizer and a speech recognition library.

[1502] 2. Transmission of information

[1503] Once speech recognition and sentiment analysis are complete, the data is packaged in JSON format and sent to a server using the HTTP protocol. The server receives and analyzes this data using software such as a request sending library.

[1504] 3. Analysis of Information

[1505] The server analyzes the received request and retrieves the appropriate guidance information from a database. This analysis uses Natural Language Processing (NLP) technology, and determines the appropriate information provision approach based on the user's emotional data.

[1506] 4. Conversion to Braille

[1507] The acquired guidance information is converted into Braille data using a Braille conversion algorithm. Specifically, the BrailleConverter library is used. The conversion results are written as Braille code.

[1508] 5. Resending Information

[1509] The server then converts the Braille data back into JSON format and sends it to the device as an HTTP response, which also includes additional feedback information based on the emotion data.

[1510] 6. Display of Information

[1511] The device then provides the received Braille data to the user using a Braille display or Braille printer. Feedback based on emotion recognition is also displayed. This allows the user to read the necessary guidance information in Braille and receive optimal feedback based on their emotions.

[1512] Examples of concrete examples and prompts

[1513] For example, if a user speaks "What is the timetable for the next train?" at a train station information desk, the system will take the following steps:

[1514] 1. Convert the voice input "Please tell me the timetable for the next train" into text.

[1515] 2. The emotion recognition engine detects the user's impatience from the tone of the voice.

[1516] 3. The server now provides information about the next train time and adds a message to reassure anxious users.

[1517] 4. Smart glasses and smartphones will provide information through braille displays and audio guidance.

[1518] Example prompt for a generative AI model:

[1519] "A user says, 'What is the timetable for the next train?' Convert this speech to text, recognize the user's emotions, and suggest the best way to respond."

[1520] This format allows visually impaired users to efficiently obtain guidance information, and also enables optimal information provision that takes into consideration their emotions at the time.

[1521] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1522] Step 1:

[1523] The user uses smart glasses or a smartphone to input voice. The input voice is converted into text data by speech recognition software. Specifically, the user inputs "Please tell me the timetable for the next train," which is converted into text "Please tell me the timetable for the next train." At the same time, an emotion recognition engine analyzes the voice data and recognizes the user's emotion (in this case, impatience). The input is voice data and emotion data, and the output is text data and analyzed emotion data.

[1524] Step 2:

[1525] The device packages text data and emotion data in JSON format and sends it to the server via the HTTP protocol. Specifically, the user's voice is sent as text "Please tell me the timetable for the next train," along with emotion data. The input is text data and emotion data, and the output is an HTTP request to the server.

[1526] Step 3:

[1527] The server analyzes the received JSON data. The server uses Natural Language Processing (NLP) technology to understand the content of the text data and retrieves the corresponding guidance information from the database. For example, the server recognizes the request "next train timetable" and retrieves the next train timetable from the database. The input is JSON data, and the output is guidance information.

[1528] Step 4:

[1529] The server converts the acquired guidance information into Braille data using a Braille conversion algorithm. In this process, the BrailleConverter library is used to convert the text "next train timetable" into Braille. The input is the text data of the guidance information, and the output is Braille data.

[1530] Step 5:

[1531] The server converts the feedback information based on the Braille data and emotion data back into JSON format and sends it to the terminal as an HTTP response. Specifically, it contains the emotion feedback "⠞⠝⠋⠥. (Braille)" and "Please stay calm and wait for the next train." The input is the Braille data and emotion feedback information, and the output is an HTTP response.

[1532] Step 6:

[1533] The device analyzes the received data and displays the information to the user through a braille display or audio output. The user can read "⠞⠝⠋⠥." on the braille display or receive information about the next train time and a message to relax via audio. The input is an HTTP response, and the output is information in a format (braille or audio) that can be understood by visually impaired users.

[1534] As a format

[1535] It explains that each step in the process has data inputs and outputs, and details how the information that the user ultimately receives is generated.

[1536] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1537] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1538] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1539] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1540] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1541] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1542] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1543] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1544] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1545] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1546] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1547] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1548] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1549] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1550] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1551] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1552] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1553] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1554] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1555] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1556] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1557] The following is further disclosed regarding the above embodiment.

[1558] (Claim 1)

[1559] an input means for a user to input information;

[1560] a transmitting means for transmitting the input information to a server;

[1561] A conversion means for converting the received information into Braille;

[1562] a transmitting means for transmitting the converted Braille information to a terminal;

[1563] a display means for displaying braille information to a user on the terminal;

[1564] A system including:

[1565] (Claim 2)

[1566] 2. The system according to claim 1, further comprising means for acquiring, from a database, service manual information corresponding to the information received by the transmitting means from the input means.

[1567] (Claim 3)

[1568] 10. The system of claim 1, wherein the converting means further comprises means for using an algorithm to convert text information into Braille.

[1569] "Example 1"

[1570] (Claim 1)

[1571] an input means for a user to input information;

[1572] a transmitting means for receiving input information by the terminal and transmitting the information to the server;

[1573] an acquisition means for acquiring information corresponding to the information received by the server from the database;

[1574] a conversion means for converting the information acquired by the server into Braille;

[1575] a transmitting means for transmitting the converted Braille information to a terminal;

[1576] a display means for displaying braille information to a user on the terminal;

[1577] A system including:

[1578] (Claim 2)

[1579] 10. The system of claim 1, wherein the server further comprises analysis means for analyzing the information received from the input means.

[1580] (Claim 3)

[1581] 10. The system of claim 1, wherein the converting means further comprises means for using an algorithm to convert text information into Braille.

[1582] "Application Example 1"

[1583] (Claim 1)

[1584] an input means for a user to input voice or text;

[1585] a transmitting means for transmitting the input information to a server;

[1586] an acquisition means for the server to analyze the received information and acquire corresponding data;

[1587] A conversion means for converting the acquired data into Braille by the server;

[1588] a transmitting means for transmitting the converted Braille information to a terminal;

[1589] a display means for displaying braille information to a user on the terminal;

[1590] A system including:

[1591] (Claim 2)

[1592] 2. The system according to claim 1, further comprising means for acquiring information corresponding to the information received by the transmitting means from the input means from a database.

[1593] (Claim 3)

[1594] the converting means uses an algorithm to convert the audio data into text;

[1595] 10. The system of claim 1, further comprising means for using an algorithm to convert text information into Braille.

[1596] "Example 2: Combining Emotion Engines"

[1597] (Claim 1)

[1598] an input means for a user to input information;

[1599] a transmitting means for transmitting the input information to a server;

[1600] an analysis means for analyzing the information received by the server and acquiring emotion data of the user;

[1601] A conversion means for converting the analyzed information into Braille;

[1602] a transmitting means for transmitting the converted Braille information to a terminal;

[1603] a display means for displaying braille information to a user on the terminal;

[1604] A system including:

[1605] (Claim 2)

[1606] 2. The system according to claim 1, further comprising means for acquiring, from a database, service manual information corresponding to the information received by the transmitting means from the input means.

[1607] (Claim 3)

[1608] 10. The system of claim 1, wherein the converting means further comprises means for using an algorithm to convert text information into Braille.

[1609] "Application example 2 when combining emotion engines"

[1610] (Claim 1)

[1611] an input means for a user to input information;

[1612] a transmitting means for transmitting the input information to a server;

[1613] A conversion means for converting the received information into Braille;

[1614] a transmitting means for transmitting the converted Braille information to a terminal;

[1615] a display means for displaying braille information to a user on the terminal;

[1616] An emotion recognition means for recognizing the emotion of a user and providing optimal information;

[1617] A system including:

[1618] (Claim 2)

[1619] 2. The system according to claim 1, further comprising means for acquiring, from a database, guidance information corresponding to the information received by the transmitting means from the input means.

[1620] (Claim 3)

[1621] 10. The system of claim 1, wherein the converting means further comprises means for using an algorithm to convert text information into Braille. [Explanation of symbols]

[1622] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. an input means for a user to input information; a transmitting means for transmitting the input information to a server; A conversion means for converting the received information into Braille; a transmitting means for transmitting the converted Braille information to a terminal; a display means for displaying braille information to a user on the terminal; A system including:

2. 2. The system according to claim 1, further comprising means for retrieving corresponding service manual information from a database based on the information received by the transmitting means from the input means.

3. 2. The system of claim 1, wherein the converting means further comprises means for using an algorithm to convert text information into Braille.

Images