system

Abstract

We provide the system. [Solution] A multilingual portal for use by foreign residents, A means for users to register their basic information, A means for the user to enter the inquiry details, A means of sending the entered query content to the server, The server analyzes the query content and generates a response based on the user's native language, A means for sending and displaying the generated response on the user's terminal, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The number of foreign residents in Japan is increasing year by year, and the main problems are language barriers and lack of information in their native languages. As a result, it is difficult to obtain necessary information in daily life, and appropriate responses may not be possible in emergencies or important situations. In addition, since it is difficult for foreigners to obtain information in their native languages, the stress of living in Japan increases. In order to solve such problems, a system that incorporates a high-performance conversational AI into a multilingual support portal for foreign residents is required.

Means for Solving the Problems

[0005] This invention relates to a multilingual portal for use by foreign residents. The system includes means for users to register basic information, means for users to input inquiry content, means for sending the inputted inquiry content to a server, means for the server to analyze the inquiry content and generate a response based on the user's native language, and means for sending and displaying the generated response on the user's terminal. Furthermore, the server uses a generation AI model to generate sophisticated conversational responses based on the user's native language and has a user login authentication means so that only authenticated users can input inquiry content. As a result, foreign residents can easily obtain the information they need in their native language and receive support for their daily lives.

[0006] "Foreign residents" refers to foreign nationals who are staying in Japan.

[0007] "Multilingual support" refers to the ability to communicate in multiple different languages.

[0008] A "portal" refers to a website or application that serves as a gateway for users to access information and utilize services.

[0009] "Basic information" refers to personally identifiable information such as the user's name, email address, password, and native language.

[0010] "Inquiry content" refers to questions, problems, or matters of consultation entered by the user.

[0011] A "server" refers to a remote computer system that stores and processes data.

[0012] A "generative AI model" refers to an algorithm or program that uses artificial intelligence technology to generate an appropriate response to a given input.

[0013] "Login authentication" refers to the procedure that verifies that a user is a registered and legitimate user when accessing a system.

[0014] "Response" refers to the answer or information that the system provides in response to an inquiry. [Brief explanation of the drawing]

[0015] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13]It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Mode for Carrying Out the Invention

[0016] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0017] First, the language used in the following description will be explained.

[0018] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be one arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be one type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0019] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0020] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters and the like. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0021] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0023] [First Embodiment]

[0024] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0025] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0028] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0031] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0035] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0036] This invention is a system that incorporates a high-performance conversational AI into a multilingual portal, thereby solving the language barriers and lack of information that foreigners residing in Japan face in their daily lives. Specific embodiments of this system are described below.

[0037] 1. User registration and login

[0038] The user installs the application and starts the registration process by pressing the "Register" button upon first launch. The device displays a form for entering basic information such as name, email address, password, and native language, prompting the user to fill it out. The entered data is sent to the server.

[0039] The server saves the received data to its database and sends a registration success message back to the terminal. The user confirms this message and completes the account registration.

[0040] Afterward, the user logs in using the email address and password they entered during registration. The device sends the entered login information to the server, which authenticates the user. If authentication is successful, the user's information is sent to the device, and the user can access the portal.

[0041] 2. Receiving and processing inquiries

[0042] The user accesses the portal and enters an inquiry such as, "Where are the hospitals where English is spoken?" The terminal converts the data entered in the input fields into text format in real time and sends it to the server.

[0043] The server analyzes the received query data and retrieves the user's native language information from the database. Then, it sends the query content and the user's native language to a generating AI model to produce an appropriate response.

[0044] 3. Response generation and reply

[0045] The generative AI model processes information based on the inquiry and generates an appropriate response in the user's native language. For example, in response to the inquiry, "Where are the hospitals where they speak English?", the generative AI model will generate a specific response such as, "There are several hospitals in Tokyo where they speak English."

[0046] The server sends the generated response to the user's terminal. The terminal displays the received response on the screen, making it easy for the user to understand. The user reviews the displayed response and can enter further questions if necessary.

[0047] Specific example

[0048] Suppose a user becomes ill and enters the question, "What medicine is effective for a cold?" The device sends the entered data to a server, which retrieves the user's native language from its database and then sends the query to a generative AI model. The generative AI model generates a response such as, "Paracetamol and ibuchen are effective for colds," and the server sends this response back to the user. Finally, the device displays this message to the user, allowing them to instantly obtain the necessary information.

[0049] In this way, the system of the present invention solves the communication challenges that foreign residents face while living in Japan and provides an environment in which users can easily obtain necessary information in their native language.

[0050] The following describes the processing flow.

[0051] Step 1:

[0052] The user installs the application and presses the "Register" button upon first launch. The device displays a form for the user to enter their name, email address, password, and native language.

[0053] Step 2:

[0054] The user enters the required information into the registration form and presses the submit button. The device sends the entered information to the server.

[0055] Step 3:

[0056] The server saves the received user information to the database and sends a registration success message back to the terminal. The terminal then displays this message to the user.

[0057] Step 4:

[0058] The user logs into the app. The device receives the user's email address and password and sends them to the server.

[0059] Step 5:

[0060] The server authenticates the user by comparing the information with that in the database and sends the authentication result back to the terminal. If authentication is successful, the user can access the portal.

[0061] Step 6:

[0062] The user accesses the portal and enters their inquiry. For example, they might enter, "Where are the hospitals where English is spoken?" The device then sends the entered information as data to the server.

[0063] Step 7:

[0064] The server analyzes the received query data and retrieves native language information from the database based on the user ID. The analysis results and the user's native language information are then sent to the AI ​​model for generation.

[0065] Step 8:

[0066] The generative AI model generates a response based on the inquiry and translates it into the user's native language. For example, in response to the inquiry, "Where are there hospitals where they speak English?", it generates the response, "There are several hospitals in Tokyo where they speak English."

[0067] Step 9:

[0068] The server sends the generated response to the user's terminal. The terminal displays the received response on the screen and presents it to the user.

[0069] Step 10:

[0070] The user reviews the displayed response and enters further questions if necessary. The server repeats the same process, generating an appropriate response and sending it back.

[0071] (Example 1)

[0072] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0073] Foreign residents in Japan often face language barriers and a lack of information in their daily lives. This problem stems from the difficulty in quickly and accurately obtaining essential information for daily life, such as the location of medical facilities and how to obtain necessary medications. This issue becomes particularly serious in emergencies or when critical information is needed. Current multilingual systems fail to provide accurate responses and timely information, thus compromising user convenience.

[0074] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0075] In this invention, the server includes means for the user to register basic information, means for the user to input inquiry content, means for transmitting the inputted inquiry content to the server, means for the server to analyze the inquiry content and obtain the user's native language information from a database, means for the server to generate a response using a generation AI model based on the inquiry content, and means for transmitting and displaying the generated response to the user's terminal. As a result, the process from user information input to response acquisition is carried out quickly and accurately, and the user can easily obtain the necessary information in their native language.

[0076] A "user" refers to an individual or group that uses the system, and in particular to foreign residents who access the multilingual portal to obtain information.

[0077] "Basic information" refers to personal information such as the user's name, email address, password, and native language, which is necessary to register with the system.

[0078] "Inquiry content" refers to specific information such as questions and requests that users enter through the system, for example, "Where are the hospitals where English is spoken?"

[0079] A "terminal" is a device used by a user to access a system and input and retrieve information, and includes smartphones, tablets, PCs, and other similar devices.

[0080] A "server" refers to a central control unit that analyzes and processes user data and inquiries, and is responsible for data storage and response generation.

[0081] A "database" refers to a system that centrally stores and manages various types of data required by the system, such as basic user information and native language information.

[0082] A "generative AI model" refers to an algorithm or program that uses artificial intelligence technology to generate appropriate responses based on user inquiries.

[0083] "Response" refers to the information returned to the user as a result of analysis and processing by the server and the generating AI model, and is provided in the user's native language.

[0084] "Login authentication" refers to the procedure used to verify that a user is a legitimate user when they access a system using their account.

[0085] A "multilingual portal" refers to a web-based or application-based information access platform that supports multiple languages ​​and is designed to allow users to obtain the information they need in their native language.

[0086] This invention is a system that incorporates a high-performance conversational AI into a multilingual portal, thereby solving the language barriers and lack of information that foreigners residing in Japan face in their daily lives. Specific embodiments of this system are described below.

[0087] User registration and login

[0088] The user installs the application and starts the registration process by pressing the "Register" button upon first launch. The device displays a form for the user to enter basic information such as name, email address, password, and native language. After the user enters this information and presses the "Submit" button, the entered information is sent to the server. The server saves the received data to its database and sends a confirmation message back to the device, acknowledging that the save was successful. The user confirms the displayed message and completes the account registration. After that, the user logs in using the email address and password entered during registration. The device sends the entered login information to the server, which performs authentication. If authentication is successful, the server returns the user information to the device, and the user can access the portal.

[0089] Inquiry reception and processing

[0090] A user accesses the portal and enters an inquiry, such as "Where are the hospitals where English is spoken?", into the input field. The terminal converts this input data into text format in real time and sends it to the server. The server analyzes the received inquiry and retrieves the user's native language information from the database. Then, it sends the inquiry and native language information to an AI model to generate an appropriate response.

[0091] Response generation and response

[0092] The generative AI model processes the inquiry content and native language information received from the server and generates an appropriate response. For example, in response to the question, "Where are the hospitals where they speak English?", it generates a specific answer such as, "There are several hospitals in Tokyo where they speak English." The server sends the generated response to the user's terminal, which displays the received response on the screen. The user can then review the displayed response and enter further questions.

[0093] Specific example

[0094] For example, suppose a user has a cold and enters the question, "What medicine is effective for a cold?" The terminal sends this input data to the server, which analyzes the query and retrieves the user's native language information from the database. Then, it sends the query "What medicine is effective for a cold?" to a generative AI model. The generative AI model generates a response such as "Paracetamol and Ibchen are effective for colds." The server sends this response to the terminal, which displays this message to the user. The user can instantly obtain the information they need.

[0095] Example of a prompt

[0096] "I want to know about cold medicine."

[0097] "Can you recommend a hospital where they speak English?"

[0098] "Does the city hall offer foreign language support?"

[0099] In this way, the system of the present invention eliminates the language barrier and lack of information that foreign residents face in their daily lives, and provides an environment in which users can easily obtain the information they need in their native language.

[0100] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0101] Step 1:

[0102] The user installs the application and clicks the "Register" button upon first launch. An input form will appear, and the user will enter basic information such as their name, email address, password, and native language.

[0103] Input: User's basic information (name, email address, password, native language)

[0104] Output: Input basic information

[0105] Step 2:

[0106] The terminal generates a request to send the entered basic information to the server and then sends the data.

[0107] Input: Basic information entered

[0108] Output: Request to send to the server

[0109] Step 3:

[0110] The server saves the received data to the database and generates a confirmation message acknowledging the successful saving. It then sends the confirmation message back to the terminal.

[0111] Input: Basic information sent to the server

[0112] Output: Confirmation message for successful saving

[0113] Step 4:

[0114] The user confirms the message displayed on their device and completes the account registration.

[0115] Input: Confirmation message for successful saving

[0116] Output: Account registration complete

[0117] Step 5:

[0118] Users log in using the email address and password they entered during registration.

[0119] Enter: Email address, password

[0120] Output: Login Request

[0121] Step 6:

[0122] The terminal sends the entered login information to the server.

[0123] Enter: Email address, password

[0124] Output: Request to send to the server

[0125] Step 7:

[0126] The server checks the received login information against its database and performs authentication. If authentication is successful, it sends the user information back to the terminal.

[0127] Input: Login information (email address, password)

[0128] Output: User information

[0129] Step 8:

[0130] The terminal displays the received user information, and the user can then access the portal.

[0131] Input: User information

[0132] Output: Access permission to the portal

[0133] Step 9:

[0134] The user accesses the portal and enters an inquiry such as, "Where are the hospitals where English is spoken?"

[0135] Input: Inquiry details

[0136] Output: Query content in text format

[0137] Step 10:

[0138] The terminal converts the entered inquiry content into text format and sends it to the server.

[0139] Input: Inquiry content in text format

[0140] Output: Request to send to the server

[0141] Step 11:

[0142] The server analyzes the received query and retrieves the user's native language information from the database. It then sends the query content and native language information to the AI ​​model for generating queries.

[0143] Input: Inquiry content in text format, native language information

[0144] Output: Request to send to the generating AI model

[0145] Step 12:

[0146] The generative AI model generates an appropriate response based on the query content and native language information.

[0147] Input: Inquiry details, native language information

[0148] Output: Generated response

[0149] Step 13:

[0150] The server sends the generated response to the user's terminal.

[0151] Input: Generated response

[0152] Output: Request to send to the terminal

[0153] Step 14:

[0154] The terminal displays the received response on the screen so that the user can confirm it.

[0155] Input: Generated response

[0156] Output: Displayed response

[0157] Step 15:

[0158] The user reviews the displayed response and enters further questions if necessary.

[0159] Input: Displayed response

[0160] Output: Additional questions

[0161] (Application Example 1)

[0162] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0163] Foreign residents and tourists in Japan often face language barriers and a lack of information when trying to understand product information and store directions in physical stores, making smooth shopping difficult. This frequently leads to inconvenience for foreigners who cannot quickly and accurately obtain the information they need. To solve this problem, there is a need to provide multilingual systems.

[0164] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0165] In this invention, the server includes means for the user to register basic information, means for the user to input inquiry content, means for transmitting the input inquiry content to the server, means for the server to analyze the inquiry content and generate a response based on the user's native language, means for transmitting and displaying the generated response on the user's terminal, and means for providing product information and store guidance in the native language at physical stores. This makes it possible for foreigners to easily obtain product information and store guidance in their native language at physical stores.

[0166] "Foreign residents" refers to foreigners who stay in Japan for an extended period, and may also include temporary tourists.

[0167] A "multilingual portal" refers to a web portal or application that supports multiple languages ​​and provides information in the language selected by the user.

[0168] "Basic information" refers to the minimum information necessary to identify a user and register them in the system, such as the user's name, email address, password, and native language.

[0169] "Inquiry content" refers to the information or questions the user wants to know, and the data used by the system to generate a response based on this.

[0170] A "server" is a system that provides specific services on a computer network, and it handles tasks such as storing, processing, and transmitting data.

[0171] A "generative AI model" is a type of artificial intelligence that refers to an algorithm used to generate appropriate responses or information based on input data.

[0172] A "prompt sentence" refers to an introductory sentence or question that a generative AI model uses to generate an appropriate response.

[0173] "Login authentication" refers to a method used to verify a user's identity when accessing a system, and typically involves using a username and password.

[0174] A "physical store" refers to a sales facility that exists physically, a place where users can go in person to purchase products.

[0175] "Product information" refers to detailed information that users need to make a purchase decision, such as the price, description, and origin of the product.

[0176] This invention is a multilingual portal system that allows foreign residents to easily obtain product information and store guides in their native language at physical stores. Specific embodiments of this system are described below.

[0177] User registration and login

[0178] The registration process begins when the user installs the application and presses the registration button upon first launch. The device displays a form for basic information such as name, email address, password, and native language, prompting the user to enter the information. The entered data is sent to the server, which stores the received data in its database and sends a registration success message back to the device. The user confirms this message and completes the account registration. After that, the user logs in using the email address and password entered during registration. The device sends the entered login information to the server, which then authenticates the user.

[0179] Inquiry reception and processing

[0180] A user accesses the portal and enters an inquiry, such as, "What products can I buy at this store?" The terminal converts the data entered in the input fields into text format in real time and sends it to the server. The server analyzes the received inquiry data and retrieves the user's native language information from the database. Then, it sends the inquiry content and the user's native language to a generating AI model to produce an appropriate response. The generated response is sent from the server to the user's terminal and displayed on the screen.

[0181] Use at physical stores

[0182] When a user needs information such as "I want to know more about the product" or "I want to know more about the store" at a physical store, they similarly input a question into the application. For example, they might enter an inquiry like, "Where is this rice produced?" The server receives this inquiry, retrieves the product information from the database, and sends it to a generative AI model to translate it into the user's native language. The generative AI model generates an appropriate response in the user's native language based on the inquiry, producing a specific answer such as, "This rice is produced in Niigata."

[0183] Response generation and response

[0184] The generative AI model processes information based on the inquiry and generates an appropriate response in the user's native language. The server sends the generated response to the user's device and displays it on the user's screen. The user can review the displayed response and enter further questions if necessary.

[0185] Program usage examples

[0186] As a concrete example, if a user wants to know information about rice at a Japanese supermarket, they would use the app to ask "What is the price and origin of the rice?" This question is sent to a generative AI model, and the translated result is displayed in the user's native language.

[0187] Example of a prompt:

[0188] Please translate the following product information from Japanese to English: "The price of the rice is 1200 yen, and it is produced in Niigata."

[0189] Hardware and software used

[0190] The hardware used in this system is a smartphone, and the software employs Python, utilizing the Transformers library for generating AI models. On the server side, a database management system is used to store and manage user data and product information.

[0191] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0192] Step 1:

[0193] The registration process begins when the user installs the application and presses the registration button upon first launch. The terminal displays a form for entering basic information such as name, email address, password, and native language, prompting the user to fill it out. The entered data (name, email address, password, native language) is sent to the server. The server stores the received data in its database, generates a registration success message, and sends it to the terminal. The terminal then displays this message to the user.

[0194] Step 2:

[0195] The user logs in using the email address and password entered during registration. The terminal sends the entered login information (email address, password) to the server. The server compares the received login information with the data in the database and performs authentication. If authentication is successful, the server generates and sends a login success message to the user. The terminal displays this message to the user.

[0196] Step 3:

[0197] The user accesses the portal and enters their inquiry (e.g., "What products are available at this store?"). The terminal converts the data entered in the input fields into text format in real time and sends that data to the server. The server analyzes the received inquiry data (text-formatted question) and retrieves the user's native language information from the database.

[0198] Step 4:

[0199] The server sends the query content and the user's native language information to the generating AI model, which then generates a prompt (e.g., "Please translate the following information from Japanese to English"). The generating AI model then generates an appropriate response based on the input query content and prompt, creating a reply in the user's native language. For example, if the query is "Where is this rice produced?", it will generate a response such as "This rice is produced in Niigata."

[0200] Step 5:

[0201] The server sends the generated response to the user's terminal. The terminal displays the received response (the user's native language) on the screen, providing it in a format that is easy for the user to understand. The user can review this displayed information and enter further questions if necessary.

[0202] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0203] This invention is a system that incorporates a high-performance conversational AI and emotion recognition engine into a multilingual portal, thereby solving the language barriers and lack of information that foreigners residing in Japan face in their daily lives. Specific embodiments of this system are described below.

[0204] 1. User registration and login

[0205] The user installs the application and starts the registration process by pressing the "Register" button upon first launch. The device displays a form for entering basic information such as name, email address, password, and native language, prompting the user to fill it out. The entered data is sent to the server.

[0206] The server saves the received data to its database and sends a registration success message back to the terminal. The user confirms this message and completes the account registration.

[0207] Afterward, the user logs in using the email address and password they entered during registration. The device sends the entered login information to the server, which authenticates the user. If authentication is successful, the user's information is sent to the device, and the user can access the portal.

[0208] 2. Receiving and processing inquiries

[0209] The user accesses the portal and enters an inquiry such as, "Where are the hospitals where English is spoken?" The terminal converts the data entered in the input fields into text format in real time and sends it to the server.

[0210] The server analyzes the received query data and retrieves native language information from the database based on the user ID. Simultaneously, the emotion engine analyzes the user's emotional state from their input and sends that information to the server. The server then sends the analysis results and emotional state information to the AI ​​model to generate an appropriate response.

[0211] 3. Response generation and reply

[0212] The generative AI model processes information based on the inquiry and considers emotional state information when generating an appropriate response in the user's native language. For example, if the user expresses anxiety, the generative AI model will generate an empathetic response that alleviates that anxiety.

[0213] The server sends the generated response to the user's terminal. The terminal displays the received response on the screen, making it easy for the user to understand. The user reviews the displayed response and can enter further questions if necessary.

[0214] Specific example

[0215] Let's say a user enters the question, "What medicine is effective for a cold?", and the emotion engine analyzes the user's emotional state as "anxious". The device sends the entered data and emotional state to the server, which then sends it to a generative AI model. The generative AI model generates a response such as, "Paracetamol and Ipuhen are effective for colds. Don't worry, these medicines are reliable." The server sends this response back to the user, and the device displays it to the user, allowing them to receive appropriate information with peace of mind.

[0216] In this way, the system of the present invention solves the communication challenges faced by foreign residents in Japan and provides an environment in which users can easily obtain necessary information in their native language. Furthermore, by providing responses that correspond to the user's emotional state through the emotion engine, it achieves more personalized support.

[0217] The following describes the processing flow.

[0218] Step 1:

[0219] The user installs the application and presses the "Register" button upon first launch. The device displays a form for the user to enter their name, email address, password, and native language.

[0220] Step 2:

[0221] The user enters the required information into the registration form and presses the submit button. The device sends the entered information to the server.

[0222] Step 3:

[0223] The server saves the received user information to the database and sends a registration success message back to the terminal. The terminal then displays this message to the user.

[0224] Step 4:

[0225] The user logs into the app. The device receives the user's email address and password and sends them to the server.

[0226] Step 5:

[0227] The server authenticates the user by comparing the information with that in the database and sends the authentication result back to the terminal. If authentication is successful, the user can access the portal.

[0228] Step 6:

[0229] The user accesses the portal and enters their inquiry. For example, they might enter, "Where are the hospitals where English is spoken?" The device then sends the entered information as data to the server.

[0230] Step 7:

[0231] The server analyzes the received query data and retrieves native language information from the database based on the user ID. Simultaneously, the emotion engine analyzes the user's emotional state from their input and sends that information to the server.

[0232] Step 8:

[0233] The server sends the query content and emotional state information to the generating AI model and instructs it to generate an appropriate response. The generating AI model then generates a response based on the query content and emotional state.

[0234] Step 9:

[0235] The generative AI model sends the generated response back to the server. For example, if a user asks, "What medicine is effective for a cold?" and their emotional state is analyzed as "anxious," a response such as, "Paracetamol and ibuchen are effective for colds. Don't worry, these medicines are reliable," is generated.

[0236] Step 10:

[0237] The server sends the generated response to the user's terminal. The terminal displays the received response on the user's screen.

[0238] Step 11:

[0239] The user can review the displayed response and gain a sense of reassurance. The user can enter further questions if necessary. The server then performs the same process again, generating an appropriate response and sending it back.

[0240] (Example 2)

[0241] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0242] The language barrier and lack of information that foreign residents in Japan face are major obstacles in various aspects of their daily lives. Misunderstandings and stress resulting from the inability to properly communicate emotions are also problematic. Because existing systems have not adequately addressed these challenges, improvements in usability are needed.

[0243] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0244] In this invention, the server includes means for registering the user's basic information, means for inputting the user's inquiry, means for transmitting the inputted inquiry to the server, means for analyzing the inquiry and generating a response based on the user's native language, means for transmitting and displaying the generated response on the user's terminal, means for acquiring emotional state information using an emotional recognition engine that analyzes the user's emotional state from the user's input, and means for generating a response considering the emotional state information. This provides an environment in which foreign residents can overcome language barriers and lack of information, appropriately convey their emotions, and enable more comfortable communication.

[0245] A "foreign resident" refers to a person who has emigrated to Japan from another country and lives in Japan for a certain period of time or permanently.

[0246] A "multilingual portal" refers to a web system or application that provides information in multiple languages ​​and allows users to interact with it in their chosen language.

[0247] "Means for users to register basic information" refers to the interface through which users enter and submit personal information such as their name, email address, password, and native language.

[0248] "Means for users to input inquiry details" refers to an interface for users to input questions or requests in text format and send them to the system.

[0249] "Means of sending the entered inquiry content to the server" refers to a function that captures information entered by the user in real time and sends the data to the server via the network.

[0250] "Means by which the server analyzes the query content and generates a response based on the user's native language" refers to the function by which the server processes the received data and generates an appropriate response in the user's native language using a database or algorithm.

[0251] "Means of sending and displaying the generated response on the user's terminal" refers to a function that sends the response generated by the server to the user's device and displays it in a format that the user can easily view.

[0252] An "emotion recognition engine" refers to algorithms and software that analyze a user's emotional state from text data, audio data, and other sources, and extract emotional information.

[0253] "Means of generating responses while considering emotional state information" refers to a function that analyzes the user's emotional state and generates more personalized responses based on that information.

[0254] "Login authentication method" refers to the process or function that allows a user to log in to a system using registered credentials (e.g., email address and password) and verify access rights.

[0255] This invention is a system that incorporates a high-performance conversational AI and emotion recognition engine into a multilingual portal, thereby solving the language barrier and lack of information that foreigners residing in Japan face in their daily lives. Specific embodiments of this invention are described in detail below.

[0256] 1. User registration and login

[0257] The user first installs the application and then starts the registration process by pressing the "Register" button upon initial launch. The terminal displays a form prompting the user to enter basic information such as name, email address, password, and native language. Once the user enters this information and presses the "Submit" button, the terminal sends the entered data to the server.

[0258] The server stores the received user information in a database (using MySQL or PostgreSQL as the database). Upon successful registration, the server sends a success message back to the terminal, which the user confirms to complete account registration.

[0259] Afterward, the user logs in using the email address and password they entered during registration. The device sends the login information to the server, which authenticates the user by comparing it with the information stored in the database. If authentication is successful, the user's information is sent to the device, and the user can access the portal.

[0260] 2. Receiving and processing inquiries

[0261] Users access the portal and enter inquiries such as, "Where are the hospitals where English is spoken?" The terminal converts the entered data into text format in real time and sends it to the server. The server analyzes the received data and retrieves native language information from the database based on the user ID.

[0262] Simultaneously, an emotion recognition engine (e.g., IBM Watson or Microsoft Azure Sentiment Analysis) is used to analyze the user's emotional state from their input. The emotional state information is sent to a server, which then sends it along with the analysis results to a generating AI model (e.g., GPT-3 or GPT-4).

[0263] 3. Response generation and reply

[0264] The generative AI model processes information based on the inquiry and generates an appropriate response in the user's native language. In doing so, it considers emotional state information, ensuring that the response is sensitive to the user's feelings.

[0265] For example, if a user enters the question "What medicine is effective for a cold?" and the emotion recognition engine analyzes the user's emotional state as "anxious," the generative AI model will generate a response such as "Paracetamol and Ipuhen are effective for colds. Don't worry, these medicines are reliable." The server sends this response to the user's device, which then displays it to the user.

[0266] Example of a prompt:

[0267] Question from a user: "What medicine is effective for colds?"

[0268] User's emotional state: "Anxious"

[0269] Response generated: "Paracetamol and ibuchen are effective for colds. Don't worry, these medicines are reliable."

[0270] In this way, the system of the present invention solves the communication challenges faced by foreign residents in Japan and provides an environment in which users can easily obtain necessary information in their native language. Furthermore, by using an emotion recognition engine, it provides responses that correspond to the user's emotional state, realizing more personalized support.

[0271] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0272] Step 1:

[0273] The user installs the application and presses the "Register" button at the first startup. The terminal displays a form for the user to enter basic information such as name, email address, password, and mother tongue. When the user enters this information and presses the "Send" button, the terminal sends the input data to the server.

[0274] Input: Name, email address, password, mother tongue

[0275] Output: The user's basic information sent to the server

[0276] Step 2:

[0277] The server saves the user information received from the terminal in a database (MySQL, PostgreSQL). If the saving is successful, the server returns a registration success message to the terminal. The user checks this message and completes the registration of the account.

[0278] Input: The user's basic information sent from the terminal

[0279] Output: The user information saved in the database, registration success message

[0280] Step 3:

[0281] The user logs in using the email address and password entered at registration. The terminal sends the login information to the server, and the server performs authentication by comparing it with the information saved in the database. If the authentication is successful, the server returns the user's information to the terminal, and the user can access the portal.

[0282] Input: Email address, password

[0283] Output: Authentication result, access right to the portal

[0284] Step 4:

[0285] The user accesses the portal and enters inquiry content such as "Where are the hospitals where English can be spoken?". The terminal converts the input data into text format in real time and sends it to the server.

[0286] Input: Text data of inquiry content

[0287] Output: Inquiry data sent to the server

[0288] Step 5:

[0289] The server analyzes the received data and retrieves the native language information from the database based on the user ID. At the same time, it analyzes the emotional state from the user's input content using an emotion recognition engine (IBM Watson, Microsoft Azure Sentiment Analysis). The emotional state information is sent to the server.

[0290] Input: Inquiry data, user ID

[0291] Output: Native language information, emotional state information

[0292] Step 6:

[0293] The server sends the analysis results and the emotional state information to a generative AI model (GPT-3, GPT-4). The generative AI model processes the information based on the inquiry content and generates an appropriate response in the user's native language. At this time, considering the emotional state information, a response that takes into account the user's emotions is generated.

[0294] Input: Analysis results, emotional state information

[0295] Output: Generated response

[0296] Step 7:

[0297] The server sends the generated response to the terminal. The terminal displays the received response on the screen, presenting it in a way that is easy for the user to understand. The user reviews the displayed information and can enter further questions if necessary.

[0298] Input: Generated response

[0299] Output: Response displayed on the terminal

[0300] (Application Example 2)

[0301] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0302] This invention aims to provide a multilingual portal system to address the language barriers and lack of information that foreigners residing in Japan face in their daily lives. It also aims to provide more personalized support by offering responses that take into account the user's emotional state. In particular, it aims to improve the user experience by providing appropriate support tailored to the user's emotions during the shopping experience within virtual stores.

[0303] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0304] In this invention, the server includes means for the user to register basic information, means for the user to input inquiry content, means for transmitting the input inquiry content to the server, means for the server to analyze the inquiry content and generate a response based on the user's native language, means for transmitting and displaying the generated response to the user's terminal, means for using an emotion recognition engine to analyze the user's emotional state, and means for considering the user's emotional state information when generating the response. This enables personalized shopping support tailored to the user's emotions within a virtual store.

[0305] The "multilingual portal" is an information - providing system that supports multiple languages and enables users to make inquiries in multiple languages.

[0306] The "user" refers to a person who uses the system, including but not limited to foreigners residing in Japan.

[0307] The "basic information" refers to the minimum information required for using the system, such as the user's name, email address, password, and native language.

[0308] The "inquiry content" refers to the content that describes the information or questions that the user wants to know.

[0309] The "server" refers to a computer system that receives and analyzes data from users and generates and transmits necessary responses.

[0310] The "native language" refers to the language that the user mainly uses, and the system generates responses based on this language.

[0311] The "emotion recognition engine" refers to a software function for analyzing and identifying the user's emotional state from the user's input and speech.

[0312] "Response generation" refers to the process in which the server creates an appropriate response based on the inquiry content, native - language information, and emotional - state information.

[0313] The "generation AI model" refers to an algorithm or system that utilizes artificial intelligence technology to automatically generate responses based on the inquiry content.

[0314] The "virtual store" is a virtual store existing on the Internet, referring to a platform that has a mechanism enabling users to view and purchase products online.

[0315] The "shopping assistant application" refers to an application that helps users obtain information about products and assist in purchases within the virtual store.

[0316] "Personalized support" refers to support that provides individualized assistance tailored to each user's needs and emotional state.

[0317] The system of this invention is a multilingual portal and can be implemented particularly as a shopping assistant application within a virtual store. The server, terminal, and user work together. Specific embodiments are described below.

[0318] 1. User registration and login

[0319] The user first installs the application and starts the registration process by pressing the "Register" button upon initial launch. The device displays a form for entering basic information such as name, email address, password, and native language, prompting the user to fill it out. The entered data is sent to the server, which stores it in a database and sends a registration success message back to the device. The user confirms this message and completes account registration. After that, the user logs in using the email address and password entered during registration. The device sends the entered login information to the server, which authenticates the user. If authentication is successful, the user's information is sent to the device, and the user can access the portal.

[0320] 2. Receiving and processing inquiries

[0321] Users ask questions about products and services within a virtual store. For example, they might input an inquiry such as, "I'd like to know more about this camera." The terminal converts the data entered in the input field into text format in real time and sends it to the server. The server analyzes the received inquiry data and retrieves native language information from the database based on the user ID. Simultaneously, an emotion recognition engine analyzes the emotional state from the user's input and sends that information to the server. The server sends the analysis results and emotional state information to a generating AI model to generate an appropriate response.

[0322] 3. Response generation and reply

[0323] The generative AI model processes information based on the inquiry and considers emotional state information when generating an appropriate response in the user's native language. For example, suppose a user asks, "Tell me more about this camera," and the emotion recognition engine analyzes the user's emotional state as "interested." The generative AI model then generates a response such as, "This camera features the latest optical zoom technology and is particularly suitable for landscape photography." The server sends the generated response to the user's device. The device displays the received response on its screen, making it easy for the user to understand. The user can review the displayed response and enter further questions if necessary.

[0324] As a concrete example, we can use the text "I'd like to know more about this camera" as an example of a prompt to ask a question to a generative AI model. This prompt allows the user to appropriately obtain the necessary information.

[0325] Specific examples of hardware and software used include the emotion recognition engine and generative AI models "EmotionRecognizer" and "Hugging Face Transformers." By utilizing these, a system that provides personalized responses based on the user's emotional state can be realized.

[0326] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0327] Step 1:

[0328] User registration and login

[0329] Input: The user enters their name, email address, password, and native language.

[0330] Output: User registration information database saving success message

[0331] Operation: The user installs the application and enters basic information such as name, email address, password, and native language. The device sends the data to the server, which stores it in a database. After saving, a registration success message is sent back to the device. The user confirms this message and completes account registration. The user then enters their email address and password to log in. The device sends the entered information to the server, which performs authentication. If authentication is successful, user information is sent to the device, and the user can access the portal.

[0332] Step 2:

[0333] Enter your inquiry details

[0334] Input: The user enters the question in text format.

[0335] Output: The entered query content is sent from the terminal to the server.

[0336] Operation: The user enters their question within the virtual store. The terminal displays the question in text format in the input field and sends it to the server. Example: "I'd like to know more about this camera."

[0337] Step 3:

[0338] Analysis of query data

[0339] Input: Query data received by the server

[0340] Output: Native language information, emotional state information

[0341] Operation: The server analyzes the input query data and retrieves native language information from the database based on the user ID. Simultaneously, the emotion recognition engine analyzes the user's emotional state from their input and sends that information to the server.

[0342] Step 4:

[0343] Response generation

[0344] Input: Inquiry data, native language information, emotional state information

[0345] Output: Generated response text

[0346] Operation: The server sends the analysis results and emotional state information to a generating AI model to generate an appropriate response. The generating AI model considers the query content and emotional state to generate an appropriate response in the user's native language. Example: "This camera is equipped with the latest optical zoom technology and is especially suitable for landscape photography."

[0347] Step 5:

[0348] Sending and displaying generated responses

[0349] Input: Generated response text

[0350] Output: Response displayed on the user terminal

[0351] Operation: The server sends the generated response to the user's terminal. The terminal displays the received response on the screen to make it easy for the user to understand. The user reviews the displayed response and can enter further questions if necessary.

[0352] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0353] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0354] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0355] [Second Embodiment]

[0356] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0357] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0358] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0359] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0360] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0361] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0362] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0363] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0364] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0365] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0366] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0367] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0368] This invention is a system that incorporates a high-performance conversational AI into a multilingual portal, thereby solving the language barriers and lack of information that foreigners residing in Japan face in their daily lives. Specific embodiments of this system are described below.

[0369] 1. User registration and login

[0370] The user installs the application and starts the registration process by pressing the "Register" button upon first launch. The device displays a form for entering basic information such as name, email address, password, and native language, prompting the user to fill it out. The entered data is sent to the server.

[0371] The server saves the received data to its database and sends a registration success message back to the terminal. The user confirms this message and completes the account registration.

[0372] Afterward, the user logs in using the email address and password they entered during registration. The device sends the entered login information to the server, which authenticates the user. If authentication is successful, the user's information is sent to the device, and the user can access the portal.

[0373] 2. Receiving and processing inquiries

[0374] The user accesses the portal and enters an inquiry such as, "Where are the hospitals where English is spoken?" The terminal converts the data entered in the input fields into text format in real time and sends it to the server.

[0375] The server analyzes the received query data and retrieves the user's native language information from the database. Then, it sends the query content and the user's native language to a generating AI model to produce an appropriate response.

[0376] 3. Response generation and reply

[0377] The generative AI model processes information based on the inquiry and generates an appropriate response in the user's native language. For example, in response to the inquiry, "Where are the hospitals where they speak English?", the generative AI model will generate a specific response such as, "There are several hospitals in Tokyo where they speak English."

[0378] The server sends the generated response to the user's terminal. The terminal displays the received response on the screen, making it easy for the user to understand. The user reviews the displayed response and can enter further questions if necessary.

[0379] Specific example

[0380] Suppose a user becomes ill and enters the question, "What medicine is effective for a cold?" The device sends the entered data to a server, which retrieves the user's native language from its database and then sends the query to a generative AI model. The generative AI model generates a response such as, "Paracetamol and ibuchen are effective for colds," and the server sends this response back to the user. Finally, the device displays this message to the user, allowing them to instantly obtain the necessary information.

[0381] In this way, the system of the present invention solves the communication challenges that foreign residents face while living in Japan and provides an environment in which users can easily obtain necessary information in their native language.

[0382] The following describes the processing flow.

[0383] Step 1:

[0384] The user installs the application and presses the "Register" button upon first launch. The device displays a form for the user to enter their name, email address, password, and native language.

[0385] Step 2:

[0386] The user enters the required information into the registration form and presses the submit button. The device sends the entered information to the server.

[0387] Step 3:

[0388] The server saves the received user information to the database and sends a registration success message back to the terminal. The terminal then displays this message to the user.

[0389] Step 4:

[0390] The user logs into the app. The device receives the user's email address and password and sends them to the server.

[0391] Step 5:

[0392] The server authenticates the user by comparing the information with that in the database and sends the authentication result back to the terminal. If authentication is successful, the user can access the portal.

[0393] Step 6:

[0394] The user accesses the portal and enters their inquiry. For example, they might enter, "Where are the hospitals where English is spoken?" The device then sends the entered information as data to the server.

[0395] Step 7:

[0396] The server analyzes the received query data and retrieves native language information from the database based on the user ID. The analysis results and the user's native language information are then sent to the AI ​​model for generation.

[0397] Step 8:

[0398] The generative AI model generates a response based on the inquiry and translates it into the user's native language. For example, in response to the inquiry, "Where are there hospitals where they speak English?", it generates the response, "There are several hospitals in Tokyo where they speak English."

[0399] Step 9:

[0400] The server sends the generated response to the user's terminal. The terminal displays the received response on the screen and presents it to the user.

[0401] Step 10:

[0402] The user reviews the displayed response and enters further questions if necessary. The server repeats the same process, generating an appropriate response and sending it back.

[0403] (Example 1)

[0404] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0405] Foreign residents in Japan often face language barriers and a lack of information in their daily lives. This problem stems from the difficulty in quickly and accurately obtaining essential information for daily life, such as the location of medical facilities and how to obtain necessary medications. This issue becomes particularly serious in emergencies or when critical information is needed. Current multilingual systems fail to provide accurate responses and timely information, thus compromising user convenience.

[0406] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0407] In this invention, the server includes means for the user to register basic information, means for the user to input inquiry content, means for transmitting the inputted inquiry content to the server, means for the server to analyze the inquiry content and obtain the user's native language information from a database, means for the server to generate a response using a generation AI model based on the inquiry content, and means for transmitting and displaying the generated response to the user's terminal. As a result, the process from user information input to response acquisition is carried out quickly and accurately, and the user can easily obtain the necessary information in their native language.

[0408] A "user" refers to an individual or group that uses the system, and in particular to foreign residents who access the multilingual portal to obtain information.

[0409] "Basic information" refers to personal information such as the user's name, email address, password, and native language, which is necessary to register with the system.

[0410] "Inquiry content" refers to specific information such as questions and requests that users enter through the system, for example, "Where are the hospitals where English is spoken?"

[0411] A "terminal" is a device used by a user to access a system and input and retrieve information, and includes smartphones, tablets, PCs, and other similar devices.

[0412] A "server" refers to a central control unit that analyzes and processes user data and inquiries, and is responsible for data storage and response generation.

[0413] A "database" refers to a system that centrally stores and manages various types of data required by the system, such as basic user information and native language information.

[0414] A "generative AI model" refers to an algorithm or program that uses artificial intelligence technology to generate appropriate responses based on user inquiries.

[0415] "Response" refers to the information returned to the user as a result of analysis and processing by the server and the generating AI model, and is provided in the user's native language.

[0416] "Login authentication" refers to the procedure used to verify that a user is a legitimate user when they access a system using their account.

[0417] A "multilingual portal" refers to a web-based or application-based information access platform that supports multiple languages ​​and is designed to allow users to obtain the information they need in their native language.

[0418] This invention is a system that incorporates a high-performance conversational AI into a multilingual portal, thereby solving the language barriers and lack of information that foreigners residing in Japan face in their daily lives. Specific embodiments of this system are described below.

[0419] User registration and login

[0420] The user installs the application and starts the registration process by pressing the "Register" button upon first launch. The device displays a form for the user to enter basic information such as name, email address, password, and native language. After the user enters this information and presses the "Submit" button, the entered information is sent to the server. The server saves the received data to its database and sends a confirmation message back to the device, acknowledging that the save was successful. The user confirms the displayed message and completes the account registration. After that, the user logs in using the email address and password entered during registration. The device sends the entered login information to the server, which performs authentication. If authentication is successful, the server returns the user information to the device, and the user can access the portal.

[0421] Inquiry reception and processing

[0422] A user accesses the portal and enters an inquiry, such as "Where are the hospitals where English is spoken?", into the input field. The terminal converts this input data into text format in real time and sends it to the server. The server analyzes the received inquiry and retrieves the user's native language information from the database. Then, it sends the inquiry and native language information to an AI model to generate an appropriate response.

[0423] Response generation and response

[0424] The generative AI model processes the inquiry content and native language information received from the server and generates an appropriate response. For example, in response to the question, "Where are the hospitals where they speak English?", it generates a specific answer such as, "There are several hospitals in Tokyo where they speak English." The server sends the generated response to the user's terminal, which displays the received response on the screen. The user can then review the displayed response and enter further questions.

[0425] Specific example

[0426] For example, suppose a user has a cold and enters the question, "What medicine is effective for a cold?" The terminal sends this input data to the server, which analyzes the query and retrieves the user's native language information from the database. Then, it sends the query "What medicine is effective for a cold?" to a generative AI model. The generative AI model generates a response such as "Paracetamol and Ibchen are effective for colds." The server sends this response to the terminal, which displays this message to the user. The user can instantly obtain the information they need.

[0427] Example of a prompt

[0428] "I want to know about cold medicine."

[0429] "Can you recommend a hospital where they speak English?"

[0430] "Does the city hall offer foreign language support?"

[0431] In this way, the system of the present invention eliminates the language barrier and lack of information that foreign residents face in their daily lives, and provides an environment in which users can easily obtain the information they need in their native language.

[0432] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0433] Step 1:

[0434] The user installs the application and clicks the "Register" button upon first launch. An input form will appear, and the user will enter basic information such as their name, email address, password, and native language.

[0435] Input: User's basic information (name, email address, password, native language)

[0436] Output: Input basic information

[0437] Step 2:

[0438] The terminal generates a request to send the entered basic information to the server and then sends the data.

[0439] Input: Basic information entered

[0440] Output: Request to send to the server

[0441] Step 3:

[0442] The server saves the received data to the database and generates a confirmation message acknowledging the successful saving. It then sends the confirmation message back to the terminal.

[0443] Input: Basic information sent to the server

[0444] Output: Confirmation message for successful saving

[0445] Step 4:

[0446] The user confirms the message displayed on their device and completes the account registration.

[0447] Input: Confirmation message for successful saving

[0448] Output: Account registration complete

[0449] Step 5:

[0450] Users log in using the email address and password they entered during registration.

[0451] Enter: Email address, password

[0452] Output: Login Request

[0453] Step 6:

[0454] The terminal sends the entered login information to the server.

[0455] Enter: Email address, password

[0456] Output: Request to send to the server

[0457] Step 7:

[0458] The server checks the received login information against its database and performs authentication. If authentication is successful, it sends the user information back to the terminal.

[0459] Input: Login information (email address, password)

[0460] Output: User information

[0461] Step 8:

[0462] The terminal displays the received user information, and the user can then access the portal.

[0463] Input: User information

[0464] Output: Access permission to the portal

[0465] Step 9:

[0466] The user accesses the portal and enters an inquiry such as, "Where are the hospitals where English is spoken?"

[0467] Input: Inquiry details

[0468] Output: Query content in text format

[0469] Step 10:

[0470] The terminal converts the entered inquiry content into text format and sends it to the server.

[0471] Input: Inquiry content in text format

[0472] Output: Request to send to the server

[0473] Step 11:

[0474] The server analyzes the received query and retrieves the user's native language information from the database. It then sends the query content and native language information to the AI ​​model for generating queries.

[0475] Input: Inquiry content in text format, native language information

[0476] Output: Request to send to the generating AI model

[0477] Step 12:

[0478] The generative AI model generates an appropriate response based on the query content and native language information.

[0479] Input: Inquiry details, native language information

[0480] Output: Generated response

[0481] Step 13:

[0482] The server sends the generated response to the user's terminal.

[0483] Input: Generated response

[0484] Output: Request to send to the terminal

[0485] Step 14:

[0486] The terminal displays the received response on the screen so that the user can confirm it.

[0487] Input: Generated response

[0488] Output: Displayed response

[0489] Step 15:

[0490] The user reviews the displayed response and enters further questions if necessary.

[0491] Input: Displayed response

[0492] Output: Additional questions

[0493] (Application Example 1)

[0494] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0495] Foreign residents and tourists in Japan often face language barriers and a lack of information when trying to understand product information and store directions in physical stores, making smooth shopping difficult. This frequently leads to inconvenience for foreigners who cannot quickly and accurately obtain the information they need. To solve this problem, there is a need to provide multilingual systems.

[0496] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0497] In this invention, the server includes means for the user to register basic information, means for the user to input inquiry content, means for transmitting the input inquiry content to the server, means for the server to analyze the inquiry content and generate a response based on the user's native language, means for transmitting and displaying the generated response on the user's terminal, and means for providing product information and store guidance in the native language at physical stores. This makes it possible for foreigners to easily obtain product information and store guidance in their native language at physical stores.

[0498] "Foreign residents" refers to foreigners who stay in Japan for an extended period, and may also include temporary tourists.

[0499] A "multilingual portal" refers to a web portal or application that supports multiple languages ​​and provides information in the language selected by the user.

[0500] "Basic information" refers to the minimum information necessary to identify a user and register them in the system, such as the user's name, email address, password, and native language.

[0501] "Inquiry content" refers to the information or questions the user wants to know, and the data used by the system to generate a response based on this.

[0502] A "server" is a system that provides specific services on a computer network, and it handles tasks such as storing, processing, and transmitting data.

[0503] A "generative AI model" is a type of artificial intelligence that refers to an algorithm used to generate appropriate responses or information based on input data.

[0504] A "prompt sentence" refers to an introductory sentence or question that a generative AI model uses to generate an appropriate response.

[0505] "Login authentication" refers to a method used to verify a user's identity when accessing a system, and typically involves using a username and password.

[0506] A "physical store" refers to a sales facility that exists physically, a place where users can go in person to purchase products.

[0507] "Product information" refers to detailed information that users need to make a purchase decision, such as the price, description, and origin of the product.

[0508] This invention is a multilingual portal system that allows foreign residents to easily obtain product information and store guides in their native language at physical stores. Specific embodiments of this system are described below.

[0509] User registration and login

[0510] The registration process begins when the user installs the application and presses the registration button upon first launch. The device displays a form for basic information such as name, email address, password, and native language, prompting the user to enter the information. The entered data is sent to the server, which stores the received data in its database and sends a registration success message back to the device. The user confirms this message and completes the account registration. After that, the user logs in using the email address and password entered during registration. The device sends the entered login information to the server, which then authenticates the user.

[0511] Inquiry reception and processing

[0512] A user accesses the portal and enters an inquiry, such as, "What products can I buy at this store?" The terminal converts the data entered in the input fields into text format in real time and sends it to the server. The server analyzes the received inquiry data and retrieves the user's native language information from the database. Then, it sends the inquiry content and the user's native language to a generating AI model to produce an appropriate response. The generated response is sent from the server to the user's terminal and displayed on the screen.

[0513] Use at physical stores

[0514] When a user needs information such as "I want to know more about the product" or "I want to know more about the store" at a physical store, they similarly input a question into the application. For example, they might enter an inquiry like, "Where is this rice produced?" The server receives this inquiry, retrieves the product information from the database, and sends it to a generative AI model to translate it into the user's native language. The generative AI model generates an appropriate response in the user's native language based on the inquiry, producing a specific answer such as, "This rice is produced in Niigata."

[0515] Response generation and response

[0516] The generative AI model processes information based on the inquiry and generates an appropriate response in the user's native language. The server sends the generated response to the user's device and displays it on the user's screen. The user can review the displayed response and enter further questions if necessary.

[0517] Program usage examples

[0518] As a concrete example, if a user wants to know information about rice at a Japanese supermarket, they would use the app to ask "What is the price and origin of the rice?" This question is sent to a generative AI model, and the translated result is displayed in the user's native language.

[0519] Example of a prompt:

[0520] Please translate the following product information from Japanese to English: "The price of the rice is 1200 yen, and it is produced in Niigata."

[0521] Hardware and software used

[0522] The hardware used in this system is a smartphone, and the software employs Python, utilizing the Transformers library for generating AI models. On the server side, a database management system is used to store and manage user data and product information.

[0523] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0524] Step 1:

[0525] The registration process begins when the user installs the application and presses the registration button upon first launch. The terminal displays a form for entering basic information such as name, email address, password, and native language, prompting the user to fill it out. The entered data (name, email address, password, native language) is sent to the server. The server stores the received data in its database, generates a registration success message, and sends it to the terminal. The terminal then displays this message to the user.

[0526] Step 2:

[0527] The user logs in using the email address and password entered during registration. The terminal sends the entered login information (email address, password) to the server. The server compares the received login information with the data in the database and performs authentication. If authentication is successful, the server generates and sends a login success message to the user. The terminal displays this message to the user.

[0528] Step 3:

[0529] The user accesses the portal and enters their inquiry (e.g., "What products are available at this store?"). The terminal converts the data entered in the input fields into text format in real time and sends that data to the server. The server analyzes the received inquiry data (text-formatted question) and retrieves the user's native language information from the database.

[0530] Step 4:

[0531] The server sends the query content and the user's native language information to the generating AI model, which then generates a prompt (e.g., "Please translate the following information from Japanese to English"). The generating AI model then generates an appropriate response based on the input query content and prompt, creating a reply in the user's native language. For example, if the query is "Where is this rice produced?", it will generate a response such as "This rice is produced in Niigata."

[0532] Step 5:

[0533] The server sends the generated response to the user's terminal. The terminal displays the received response (the user's native language) on the screen, providing it in a format that is easy for the user to understand. The user can review this displayed information and enter further questions if necessary.

[0534] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0535] This invention is a system that incorporates a high-performance conversational AI and emotion recognition engine into a multilingual portal, thereby solving the language barriers and lack of information that foreigners residing in Japan face in their daily lives. Specific embodiments of this system are described below.

[0536] 1. User registration and login

[0537] The user installs the application and starts the registration process by pressing the "Register" button upon first launch. The device displays a form for entering basic information such as name, email address, password, and native language, prompting the user to fill it out. The entered data is sent to the server.

[0538] The server saves the received data to its database and sends a registration success message back to the terminal. The user confirms this message and completes the account registration.

[0539] Afterward, the user logs in using the email address and password they entered during registration. The device sends the entered login information to the server, which authenticates the user. If authentication is successful, the user's information is sent to the device, and the user can access the portal.

[0540] 2. Receiving and processing inquiries

[0541] The user accesses the portal and enters an inquiry such as, "Where are the hospitals where English is spoken?" The terminal converts the data entered in the input fields into text format in real time and sends it to the server.

[0542] The server analyzes the received query data and retrieves native language information from the database based on the user ID. Simultaneously, the emotion engine analyzes the user's emotional state from their input and sends that information to the server. The server then sends the analysis results and emotional state information to the AI ​​model to generate an appropriate response.

[0543] 3. Response generation and reply

[0544] The generative AI model processes information based on the inquiry and considers emotional state information when generating an appropriate response in the user's native language. For example, if the user expresses anxiety, the generative AI model will generate an empathetic response that alleviates that anxiety.

[0545] The server sends the generated response to the user's terminal. The terminal displays the received response on the screen, making it easy for the user to understand. The user reviews the displayed response and can enter further questions if necessary.

[0546] Specific example

[0547] Let's say a user enters the question, "What medicine is effective for a cold?", and the emotion engine analyzes the user's emotional state as "anxious". The device sends the entered data and emotional state to the server, which then sends it to a generative AI model. The generative AI model generates a response such as, "Paracetamol and Ipuhen are effective for colds. Don't worry, these medicines are reliable." The server sends this response back to the user, and the device displays it to the user, allowing them to receive appropriate information with peace of mind.

[0548] In this way, the system of the present invention solves the communication challenges faced by foreign residents in Japan and provides an environment in which users can easily obtain necessary information in their native language. Furthermore, by providing responses that correspond to the user's emotional state through the emotion engine, it achieves more personalized support.

[0549] The following describes the processing flow.

[0550] Step 1:

[0551] The user installs the application and presses the "Register" button upon first launch. The device displays a form for the user to enter their name, email address, password, and native language.

[0552] Step 2:

[0553] The user enters the required information into the registration form and presses the submit button. The device sends the entered information to the server.

[0554] Step 3:

[0555] The server saves the received user information to the database and sends a registration success message back to the terminal. The terminal then displays this message to the user.

[0556] Step 4:

[0557] The user logs into the app. The device receives the user's email address and password and sends them to the server.

[0558] Step 5:

[0559] The server authenticates the user by comparing the information with that in the database and sends the authentication result back to the terminal. If authentication is successful, the user can access the portal.

[0560] Step 6:

[0561] The user accesses the portal and enters their inquiry. For example, they might enter, "Where are the hospitals where English is spoken?" The device then sends the entered information as data to the server.

[0562] Step 7:

[0563] The server analyzes the received query data and retrieves native language information from the database based on the user ID. Simultaneously, the emotion engine analyzes the user's emotional state from their input and sends that information to the server.

[0564] Step 8:

[0565] The server sends the query content and emotional state information to the generating AI model and instructs it to generate an appropriate response. The generating AI model then generates a response based on the query content and emotional state.

[0566] Step 9:

[0567] The generative AI model sends the generated response back to the server. For example, if a user asks, "What medicine is effective for a cold?" and their emotional state is analyzed as "anxious," a response such as, "Paracetamol and ibuchen are effective for colds. Don't worry, these medicines are reliable," is generated.

[0568] Step 10:

[0569] The server sends the generated response to the user's terminal. The terminal displays the received response on the user's screen.

[0570] Step 11:

[0571] The user can review the displayed response and gain a sense of reassurance. The user can enter further questions if necessary. The server then performs the same process again, generating an appropriate response and sending it back.

[0572] (Example 2)

[0573] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0574] The language barrier and lack of information that foreign residents in Japan face are major obstacles in various aspects of their daily lives. Misunderstandings and stress resulting from the inability to properly communicate emotions are also problematic. Because existing systems have not adequately addressed these challenges, improvements in usability are needed.

[0575] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0576] In this invention, the server includes means for registering the user's basic information, means for inputting the user's inquiry, means for transmitting the inputted inquiry to the server, means for analyzing the inquiry and generating a response based on the user's native language, means for transmitting and displaying the generated response on the user's terminal, means for acquiring emotional state information using an emotional recognition engine that analyzes the user's emotional state from the user's input, and means for generating a response considering the emotional state information. This provides an environment in which foreign residents can overcome language barriers and lack of information, appropriately convey their emotions, and enable more comfortable communication.

[0577] A "foreign resident" refers to a person who has emigrated to Japan from another country and lives in Japan for a certain period of time or permanently.

[0578] A "multilingual portal" refers to a web system or application that provides information in multiple languages ​​and allows users to interact with it in their chosen language.

[0579] "Means for users to register basic information" refers to the interface through which users enter and submit personal information such as their name, email address, password, and native language.

[0580] "Means for users to input inquiry details" refers to an interface for users to input questions or requests in text format and send them to the system.

[0581] "Means of sending the entered inquiry content to the server" refers to a function that captures information entered by the user in real time and sends the data to the server via the network.

[0582] "Means by which the server analyzes the query content and generates a response based on the user's native language" refers to the function by which the server processes the received data and generates an appropriate response in the user's native language using a database or algorithm.

[0583] "Means of sending and displaying the generated response on the user's terminal" refers to a function that sends the response generated by the server to the user's device and displays it in a format that the user can easily view.

[0584] An "emotion recognition engine" refers to algorithms and software that analyze a user's emotional state from text data, audio data, and other sources, and extract emotional information.

[0585] "Means of generating responses while considering emotional state information" refers to a function that analyzes the user's emotional state and generates more personalized responses based on that information.

[0586] "Login authentication method" refers to the process or function that allows a user to log in to a system using registered credentials (e.g., email address and password) and verify access rights.

[0587] This invention is a system that incorporates a high-performance conversational AI and emotion recognition engine into a multilingual portal, thereby solving the language barrier and lack of information that foreigners residing in Japan face in their daily lives. Specific embodiments of this invention are described in detail below.

[0588] 1. User registration and login

[0589] The user first installs the application and then starts the registration process by pressing the "Register" button upon initial launch. The terminal displays a form prompting the user to enter basic information such as name, email address, password, and native language. Once the user enters this information and presses the "Submit" button, the terminal sends the entered data to the server.

[0590] The server stores the received user information in a database (using MySQL or PostgreSQL as the database). Upon successful registration, the server sends a success message back to the terminal, which the user confirms to complete account registration.

[0591] Afterward, the user logs in using the email address and password they entered during registration. The device sends the login information to the server, which authenticates the user by comparing it with the information stored in the database. If authentication is successful, the user's information is sent to the device, and the user can access the portal.

[0592] 2. Receiving and processing inquiries

[0593] Users access the portal and enter inquiries such as, "Where are the hospitals where English is spoken?" The terminal converts the entered data into text format in real time and sends it to the server. The server analyzes the received data and retrieves native language information from the database based on the user ID.

[0594] Simultaneously, an emotion recognition engine (e.g., IBM Watson or Microsoft Azure Sentiment Analysis) is used to analyze the user's emotional state from their input. The emotional state information is sent to a server, which then sends it along with the analysis results to a generating AI model (e.g., GPT-3 or GPT-4).

[0595] 3. Response generation and reply

[0596] The generative AI model processes information based on the inquiry and generates an appropriate response in the user's native language. In doing so, it considers emotional state information, ensuring that the response is sensitive to the user's feelings.

[0597] For example, if a user enters the question "What medicine is effective for a cold?" and the emotion recognition engine analyzes the user's emotional state as "anxious," the generative AI model will generate a response such as "Paracetamol and Ipuhen are effective for colds. Don't worry, these medicines are reliable." The server sends this response to the user's device, which then displays it to the user.

[0598] Example of a prompt:

[0599] Question from a user: "What medicine is effective for colds?"

[0600] User's emotional state: "Anxious"

[0601] Response generated: "Paracetamol and ibuchen are effective for colds. Don't worry, these medicines are reliable."

[0602] In this way, the system of the present invention solves the communication challenges faced by foreign residents in Japan and provides an environment in which users can easily obtain necessary information in their native language. Furthermore, by using an emotion recognition engine, it provides responses that correspond to the user's emotional state, realizing more personalized support.

[0603] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0604] Step 1:

[0605] The user installs the application and presses the "Register" button upon first launch. The device displays a form prompting the user to enter basic information such as name, email address, password, and native language. After the user enters this information and presses the "Submit" button, the device sends the entered data to the server.

[0606] Input: Name, email address, password, native language

[0607] Output: Basic user information sent to the server

[0608] Step 2:

[0609] The server saves the user information received from the terminal to a database (MySQL, PostgreSQL). If the saving is successful, the server sends a registration success message back to the terminal. The user confirms this message and completes the account registration.

[0610] Input: Basic user information sent from the device.

[0611] Output: User information stored in the database, registration success message

[0612] Step 3:

[0613] Users log in using the email address and password they entered during registration. The device sends the login information to the server, which authenticates the user by comparing it with the information stored in the database. If authentication is successful, the server sends the user's information back to the device, and the user can then access the portal.

[0614] Enter: Email address, password

[0615] Output: Authentication result, access rights to the portal

[0616] Step 4:

[0617] The user accesses the portal and enters an inquiry such as, "Where are the hospitals where English is spoken?" The terminal converts the entered data into text format in real time and sends it to the server.

[0618] Input: Text data of the inquiry.

[0619] Output: Query data sent to the server

[0620] Step 5:

[0621] The server analyzes the received data and retrieves native language information from the database based on the user ID. Simultaneously, it analyzes the user's emotional state from their input using an emotion recognition engine (IBM Watson, Microsoft Azure Sentiment Analysis). The emotional state information is then sent back to the server.

[0622] Input: Inquiry data, User ID

[0623] Output: Native language information, emotional state information

[0624] Step 6:

[0625] The server sends the analysis results and emotional state information to the generating AI models (GPT-3, GPT-4). The generating AI models process the information based on the query and generate an appropriate response in the user's native language. In doing so, they take the emotional state information into consideration and generate a response that is sensitive to the user's emotions.

[0626] Input: Analysis results, emotional state information

[0627] Output: Generated response

[0628] Step 7:

[0629] The server sends the generated response to the terminal. The terminal displays the received response on the screen, presenting it in a way that is easy for the user to understand. The user reviews the displayed information and can enter further questions if necessary.

[0630] Input: Generated response

[0631] Output: Response displayed on the terminal

[0632] (Application Example 2)

[0633] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0634] This invention aims to provide a multilingual portal system to address the language barriers and lack of information that foreigners residing in Japan face in their daily lives. It also aims to provide more personalized support by offering responses that take into account the user's emotional state. In particular, it aims to improve the user experience by providing appropriate support tailored to the user's emotions during the shopping experience within virtual stores.

[0635] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0636] In this invention, the server includes means for the user to register basic information, means for the user to input inquiry content, means for transmitting the input inquiry content to the server, means for the server to analyze the inquiry content and generate a response based on the user's native language, means for transmitting and displaying the generated response to the user's terminal, means for using an emotion recognition engine to analyze the user's emotional state, and means for considering the user's emotional state information when generating the response. This enables personalized shopping support tailored to the user's emotions within a virtual store.

[0637] A "multilingual portal" is an information provision system that supports multiple languages, allowing users to make inquiries in multiple languages.

[0638] "User" refers to anyone who uses the system, and is not limited to, but includes, in particular, foreigners residing in Japan.

[0639] "Basic information" refers to the minimum information necessary to use the system, such as the user's name, email address, password, and native language.

[0640] "Inquiry content" refers to the information or questions that the user wants to know.

[0641] A "server" refers to a computer system that receives and analyzes data from users, generates the necessary responses, and sends them back.

[0642] "Native language" refers to the language the user primarily uses, and the system generates responses based on this language.

[0643] An "emotion recognition engine" refers to a software function that analyzes and identifies a user's emotional state based on their input and statements.

[0644] "Response generation" refers to the process by which a server creates an appropriate response based on the query content, native language information, and sentiment status information.

[0645] A "generative AI model" refers to an algorithm or system that uses artificial intelligence technology to automatically generate responses based on the content of an inquiry.

[0646] A "virtual store" refers to a virtual store that exists on the internet, and is a platform that allows users to browse and purchase products online.

[0647] A "shopping assistant application" refers to an application that allows users to obtain information about products and receive assistance with purchasing within a virtual store.

[0648] "Personalized support" refers to support that provides individualized assistance tailored to each user's needs and emotional state.

[0649] The system of this invention is a multilingual portal and can be implemented particularly as a shopping assistant application within a virtual store. The server, terminal, and user work together. Specific embodiments are described below.

[0650] 1. User registration and login

[0651] The user first installs the application and starts the registration process by pressing the "Register" button upon initial launch. The device displays a form for entering basic information such as name, email address, password, and native language, prompting the user to fill it out. The entered data is sent to the server, which stores it in a database and sends a registration success message back to the device. The user confirms this message and completes account registration. After that, the user logs in using the email address and password entered during registration. The device sends the entered login information to the server, which authenticates the user. If authentication is successful, the user's information is sent to the device, and the user can access the portal.

[0652] 2. Receiving and processing inquiries

[0653] Users ask questions about products and services within a virtual store. For example, they might input an inquiry such as, "I'd like to know more about this camera." The terminal converts the data entered in the input field into text format in real time and sends it to the server. The server analyzes the received inquiry data and retrieves native language information from the database based on the user ID. Simultaneously, an emotion recognition engine analyzes the emotional state from the user's input and sends that information to the server. The server sends the analysis results and emotional state information to a generating AI model to generate an appropriate response.

[0654] 3. Response generation and reply

[0655] The generative AI model processes information based on the inquiry and considers emotional state information when generating an appropriate response in the user's native language. For example, suppose a user asks, "Tell me more about this camera," and the emotion recognition engine analyzes the user's emotional state as "interested." The generative AI model then generates a response such as, "This camera features the latest optical zoom technology and is particularly suitable for landscape photography." The server sends the generated response to the user's device. The device displays the received response on its screen, making it easy for the user to understand. The user can review the displayed response and enter further questions if necessary.

[0656] As a concrete example, we can use the text "I'd like to know more about this camera" as an example of a prompt to ask a question to a generative AI model. This prompt allows the user to appropriately obtain the necessary information.

[0657] Specific examples of hardware and software used include the emotion recognition engine and generative AI models "EmotionRecognizer" and "Hugging Face Transformers." By utilizing these, a system that provides personalized responses based on the user's emotional state can be realized.

[0658] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0659] Step 1:

[0660] User registration and login

[0661] Input: The user enters their name, email address, password, and native language.

[0662] Output: User registration information database saving success message

[0663] Operation: The user installs the application and enters basic information such as name, email address, password, and native language. The device sends the data to the server, which stores it in a database. After saving, a registration success message is sent back to the device. The user confirms this message and completes account registration. The user then enters their email address and password to log in. The device sends the entered information to the server, which performs authentication. If authentication is successful, user information is sent to the device, and the user can access the portal.

[0664] Step 2:

[0665] Enter your inquiry details

[0666] Input: The user enters the question in text format.

[0667] Output: The entered query content is sent from the terminal to the server.

[0668] Operation: The user enters their question within the virtual store. The terminal displays the question in text format in the input field and sends it to the server. Example: "I'd like to know more about this camera."

[0669] Step 3:

[0670] Analysis of query data

[0671] Input: Query data received by the server

[0672] Output: Native language information, emotional state information

[0673] Operation: The server analyzes the input query data and retrieves native language information from the database based on the user ID. Simultaneously, the emotion recognition engine analyzes the user's emotional state from their input and sends that information to the server.

[0674] Step 4:

[0675] Response generation

[0676] Input: Inquiry data, native language information, emotional state information

[0677] Output: Generated response text

[0678] Operation: The server sends the analysis results and emotional state information to a generating AI model to generate an appropriate response. The generating AI model considers the query content and emotional state to generate an appropriate response in the user's native language. Example: "This camera is equipped with the latest optical zoom technology and is especially suitable for landscape photography."

[0679] Step 5:

[0680] Sending and displaying generated responses

[0681] Input: Generated response text

[0682] Output: Response displayed on the user terminal

[0683] Operation: The server sends the generated response to the user's terminal. The terminal displays the received response on the screen to make it easy for the user to understand. The user reviews the displayed response and can enter further questions if necessary.

[0684] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0685] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0686] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0687] [Third Embodiment]

[0688] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0689] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0690] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0691] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0692] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0693] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0694] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0695] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0696] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0697] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0698] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0699] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0700] This invention is a system that incorporates a high-performance conversational AI into a multilingual portal, thereby solving the language barriers and lack of information that foreigners residing in Japan face in their daily lives. Specific embodiments of this system are described below.

[0701] 1. User registration and login

[0702] The user installs the application and starts the registration process by pressing the "Register" button upon first launch. The device displays a form for entering basic information such as name, email address, password, and native language, prompting the user to fill it out. The entered data is sent to the server.

[0703] The server saves the received data to its database and sends a registration success message back to the terminal. The user confirms this message and completes the account registration.

[0704] Afterward, the user logs in using the email address and password they entered during registration. The device sends the entered login information to the server, which authenticates the user. If authentication is successful, the user's information is sent to the device, and the user can access the portal.

[0705] 2. Receiving and processing inquiries

[0706] The user accesses the portal and enters an inquiry such as, "Where are the hospitals where English is spoken?" The terminal converts the data entered in the input fields into text format in real time and sends it to the server.

[0707] The server analyzes the received query data and retrieves the user's native language information from the database. Then, it sends the query content and the user's native language to a generating AI model to produce an appropriate response.

[0708] 3. Response generation and reply

[0709] The generative AI model processes information based on the inquiry and generates an appropriate response in the user's native language. For example, in response to the inquiry, "Where are the hospitals where they speak English?", the generative AI model will generate a specific response such as, "There are several hospitals in Tokyo where they speak English."

[0710] The server sends the generated response to the user's terminal. The terminal displays the received response on the screen, making it easy for the user to understand. The user reviews the displayed response and can enter further questions if necessary.

[0711] Specific example

[0712] Suppose a user becomes ill and enters the question, "What medicine is effective for a cold?" The device sends the entered data to a server, which retrieves the user's native language from its database and then sends the query to a generative AI model. The generative AI model generates a response such as, "Paracetamol and ibuchen are effective for colds," and the server sends this response back to the user. Finally, the device displays this message to the user, allowing them to instantly obtain the necessary information.

[0713] In this way, the system of the present invention solves the communication challenges that foreign residents face while living in Japan and provides an environment in which users can easily obtain necessary information in their native language.

[0714] The following describes the processing flow.

[0715] Step 1:

[0716] The user installs the application and presses the "Register" button upon first launch. The device displays a form for the user to enter their name, email address, password, and native language.

[0717] Step 2:

[0718] The user enters the required information into the registration form and presses the submit button. The device sends the entered information to the server.

[0719] Step 3:

[0720] The server saves the received user information to the database and sends a registration success message back to the terminal. The terminal then displays this message to the user.

[0721] Step 4:

[0722] The user logs into the app. The device receives the user's email address and password and sends them to the server.

[0723] Step 5:

[0724] The server authenticates the user by comparing the information with that in the database and sends the authentication result back to the terminal. If authentication is successful, the user can access the portal.

[0725] Step 6:

[0726] The user accesses the portal and enters their inquiry. For example, they might enter, "Where are the hospitals where English is spoken?" The device then sends the entered information as data to the server.

[0727] Step 7:

[0728] The server analyzes the received query data and retrieves native language information from the database based on the user ID. The analysis results and the user's native language information are then sent to the AI ​​model for generation.

[0729] Step 8:

[0730] The generative AI model generates a response based on the inquiry and translates it into the user's native language. For example, in response to the inquiry, "Where are there hospitals where they speak English?", it generates the response, "There are several hospitals in Tokyo where they speak English."

[0731] Step 9:

[0732] The server sends the generated response to the user's terminal. The terminal displays the received response on the screen and presents it to the user.

[0733] Step 10:

[0734] The user reviews the displayed response and enters further questions if necessary. The server repeats the same process, generating an appropriate response and sending it back.

[0735] (Example 1)

[0736] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0737] Foreign residents in Japan often face language barriers and a lack of information in their daily lives. This problem stems from the difficulty in quickly and accurately obtaining essential information for daily life, such as the location of medical facilities and how to obtain necessary medications. This issue becomes particularly serious in emergencies or when critical information is needed. Current multilingual systems fail to provide accurate responses and timely information, thus compromising user convenience.

[0738] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0739] In this invention, the server includes means for the user to register basic information, means for the user to input inquiry content, means for transmitting the inputted inquiry content to the server, means for the server to analyze the inquiry content and obtain the user's native language information from a database, means for the server to generate a response using a generation AI model based on the inquiry content, and means for transmitting and displaying the generated response to the user's terminal. As a result, the process from user information input to response acquisition is carried out quickly and accurately, and the user can easily obtain the necessary information in their native language.

[0740] A "user" refers to an individual or group that uses the system, and in particular to foreign residents who access the multilingual portal to obtain information.

[0741] "Basic information" refers to personal information such as the user's name, email address, password, and native language, which is necessary to register with the system.

[0742] "Inquiry content" refers to specific information such as questions and requests that users enter through the system, for example, "Where are the hospitals where English is spoken?"

[0743] A "terminal" is a device used by a user to access a system and input and retrieve information, and includes smartphones, tablets, PCs, and other similar devices.

[0744] A "server" refers to a central control unit that analyzes and processes user data and inquiries, and is responsible for data storage and response generation.

[0745] A "database" refers to a system that centrally stores and manages various types of data required by the system, such as basic user information and native language information.

[0746] A "generative AI model" refers to an algorithm or program that uses artificial intelligence technology to generate appropriate responses based on user inquiries.

[0747] "Response" refers to the information returned to the user as a result of analysis and processing by the server and the generating AI model, and is provided in the user's native language.

[0748] "Login authentication" refers to the procedure used to verify that a user is a legitimate user when they access a system using their account.

[0749] A "multilingual portal" refers to a web-based or application-based information access platform that supports multiple languages ​​and is designed to allow users to obtain the information they need in their native language.

[0750] This invention is a system that incorporates a high-performance conversational AI into a multilingual portal, thereby solving the language barriers and lack of information that foreigners residing in Japan face in their daily lives. Specific embodiments of this system are described below.

[0751] User registration and login

[0752] The user installs the application and starts the registration process by pressing the "Register" button upon first launch. The device displays a form for the user to enter basic information such as name, email address, password, and native language. After the user enters this information and presses the "Submit" button, the entered information is sent to the server. The server saves the received data to its database and sends a confirmation message back to the device, acknowledging that the save was successful. The user confirms the displayed message and completes the account registration. After that, the user logs in using the email address and password entered during registration. The device sends the entered login information to the server, which performs authentication. If authentication is successful, the server returns the user information to the device, and the user can access the portal.

[0753] Inquiry reception and processing

[0754] A user accesses the portal and enters an inquiry, such as "Where are the hospitals where English is spoken?", into the input field. The terminal converts this input data into text format in real time and sends it to the server. The server analyzes the received inquiry and retrieves the user's native language information from the database. Then, it sends the inquiry and native language information to an AI model to generate an appropriate response.

[0755] Response generation and response

[0756] The generative AI model processes the inquiry content and native language information received from the server and generates an appropriate response. For example, in response to the question, "Where are the hospitals where they speak English?", it generates a specific answer such as, "There are several hospitals in Tokyo where they speak English." The server sends the generated response to the user's terminal, which displays the received response on the screen. The user can then review the displayed response and enter further questions.

[0757] Specific example

[0758] For example, suppose a user has a cold and enters the question, "What medicine is effective for a cold?" The terminal sends this input data to the server, which analyzes the query and retrieves the user's native language information from the database. Then, it sends the query "What medicine is effective for a cold?" to a generative AI model. The generative AI model generates a response such as "Paracetamol and Ibchen are effective for colds." The server sends this response to the terminal, which displays this message to the user. The user can instantly obtain the information they need.

[0759] Example of a prompt

[0760] "I want to know about cold medicine."

[0761] "Can you recommend a hospital where they speak English?"

[0762] "Does the city hall offer foreign language support?"

[0763] In this way, the system of the present invention eliminates the language barrier and lack of information that foreign residents face in their daily lives, and provides an environment in which users can easily obtain the information they need in their native language.

[0764] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0765] Step 1:

[0766] The user installs the application and clicks the "Register" button upon first launch. An input form will appear, and the user will enter basic information such as their name, email address, password, and native language.

[0767] Input: User's basic information (name, email address, password, native language)

[0768] Output: Input basic information

[0769] Step 2:

[0770] The terminal generates a request to send the entered basic information to the server and then sends the data.

[0771] Input: Basic information entered

[0772] Output: Request to send to the server

[0773] Step 3:

[0774] The server saves the received data to the database and generates a confirmation message acknowledging the successful saving. It then sends the confirmation message back to the terminal.

[0775] Input: Basic information sent to the server

[0776] Output: Confirmation message for successful saving

[0777] Step 4:

[0778] The user confirms the message displayed on their device and completes the account registration.

[0779] Input: Confirmation message for successful saving

[0780] Output: Account registration complete

[0781] Step 5:

[0782] Users log in using the email address and password they entered during registration.

[0783] Enter: Email address, password

[0784] Output: Login Request

[0785] Step 6:

[0786] The terminal sends the entered login information to the server.

[0787] Enter: Email address, password

[0788] Output: Request to send to the server

[0789] Step 7:

[0790] The server checks the received login information against its database and performs authentication. If authentication is successful, it sends the user information back to the terminal.

[0791] Input: Login information (email address, password)

[0792] Output: User information

[0793] Step 8:

[0794] The terminal displays the received user information, and the user can then access the portal.

[0795] Input: User information

[0796] Output: Access permission to the portal

[0797] Step 9:

[0798] The user accesses the portal and enters an inquiry such as, "Where are the hospitals where English is spoken?"

[0799] Input: Inquiry details

[0800] Output: Query content in text format

[0801] Step 10:

[0802] The terminal converts the entered inquiry content into text format and sends it to the server.

[0803] Input: Inquiry content in text format

[0804] Output: Request to send to the server

[0805] Step 11:

[0806] The server analyzes the received query and retrieves the user's native language information from the database. It then sends the query content and native language information to the AI ​​model for generating queries.

[0807] Input: Inquiry content in text format, native language information

[0808] Output: Request to send to the generating AI model

[0809] Step 12:

[0810] The generative AI model generates an appropriate response based on the query content and native language information.

[0811] Input: Inquiry details, native language information

[0812] Output: Generated response

[0813] Step 13:

[0814] The server sends the generated response to the user's terminal.

[0815] Input: Generated response

[0816] Output: Request to send to the terminal

[0817] Step 14:

[0818] The terminal displays the received response on the screen so that the user can confirm it.

[0819] Input: Generated response

[0820] Output: Displayed response

[0821] Step 15:

[0822] The user reviews the displayed response and enters further questions if necessary.

[0823] Input: Displayed response

[0824] Output: Additional questions

[0825] (Application Example 1)

[0826] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0827] Foreign residents and tourists in Japan often face language barriers and a lack of information when trying to understand product information and store directions in physical stores, making smooth shopping difficult. This frequently leads to inconvenience for foreigners who cannot quickly and accurately obtain the information they need. To solve this problem, there is a need to provide multilingual systems.

[0828] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0829] In this invention, the server includes means for the user to register basic information, means for the user to input inquiry content, means for transmitting the input inquiry content to the server, means for the server to analyze the inquiry content and generate a response based on the user's native language, means for transmitting and displaying the generated response on the user's terminal, and means for providing product information and store guidance in the native language at physical stores. This makes it possible for foreigners to easily obtain product information and store guidance in their native language at physical stores.

[0830] "Foreign residents" refers to foreigners who stay in Japan for an extended period, and may also include temporary tourists.

[0831] A "multilingual portal" refers to a web portal or application that supports multiple languages ​​and provides information in the language selected by the user.

[0832] "Basic information" refers to the minimum information necessary to identify a user and register them in the system, such as the user's name, email address, password, and native language.

[0833] "Inquiry content" refers to the information or questions the user wants to know, and the data used by the system to generate a response based on this.

[0834] A "server" is a system that provides specific services on a computer network, and it handles tasks such as storing, processing, and transmitting data.

[0835] A "generative AI model" is a type of artificial intelligence that refers to an algorithm used to generate appropriate responses or information based on input data.

[0836] A "prompt sentence" refers to an introductory sentence or question that a generative AI model uses to generate an appropriate response.

[0837] "Login authentication" refers to a method used to verify a user's identity when accessing a system, and typically involves using a username and password.

[0838] A "physical store" refers to a sales facility that exists physically, a place where users can go in person to purchase products.

[0839] "Product information" refers to detailed information that users need to make a purchase decision, such as the price, description, and origin of the product.

[0840] This invention is a multilingual portal system that allows foreign residents to easily obtain product information and store guides in their native language at physical stores. Specific embodiments of this system are described below.

[0841] User registration and login

[0842] The registration process begins when the user installs the application and presses the registration button upon first launch. The device displays a form for basic information such as name, email address, password, and native language, prompting the user to enter the information. The entered data is sent to the server, which stores the received data in its database and sends a registration success message back to the device. The user confirms this message and completes the account registration. After that, the user logs in using the email address and password entered during registration. The device sends the entered login information to the server, which then authenticates the user.

[0843] Inquiry reception and processing

[0844] A user accesses the portal and enters an inquiry, such as, "What products can I buy at this store?" The terminal converts the data entered in the input fields into text format in real time and sends it to the server. The server analyzes the received inquiry data and retrieves the user's native language information from the database. Then, it sends the inquiry content and the user's native language to a generating AI model to produce an appropriate response. The generated response is sent from the server to the user's terminal and displayed on the screen.

[0845] Use at physical stores

[0846] When a user needs information such as "I want to know more about the product" or "I want to know more about the store" at a physical store, they similarly input a question into the application. For example, they might enter an inquiry like, "Where is this rice produced?" The server receives this inquiry, retrieves the product information from the database, and sends it to a generative AI model to translate it into the user's native language. The generative AI model generates an appropriate response in the user's native language based on the inquiry, producing a specific answer such as, "This rice is produced in Niigata."

[0847] Response generation and response

[0848] The generative AI model processes information based on the inquiry and generates an appropriate response in the user's native language. The server sends the generated response to the user's device and displays it on the user's screen. The user can review the displayed response and enter further questions if necessary.

[0849] Program usage examples

[0850] As a concrete example, if a user wants to know information about rice at a Japanese supermarket, they would use the app to ask "What is the price and origin of the rice?" This question is sent to a generative AI model, and the translated result is displayed in the user's native language.

[0851] Example of a prompt:

[0852] Please translate the following product information from Japanese to English: "The price of the rice is 1200 yen, and it is produced in Niigata."

[0853] Hardware and software used

[0854] The hardware used in this system is a smartphone, and the software employs Python, utilizing the Transformers library for generating AI models. On the server side, a database management system is used to store and manage user data and product information.

[0855] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0856] Step 1:

[0857] The registration process begins when the user installs the application and presses the registration button upon first launch. The terminal displays a form for entering basic information such as name, email address, password, and native language, prompting the user to fill it out. The entered data (name, email address, password, native language) is sent to the server. The server stores the received data in its database, generates a registration success message, and sends it to the terminal. The terminal then displays this message to the user.

[0858] Step 2:

[0859] The user logs in using the email address and password entered during registration. The terminal sends the entered login information (email address, password) to the server. The server compares the received login information with the data in the database and performs authentication. If authentication is successful, the server generates and sends a login success message to the user. The terminal displays this message to the user.

[0860] Step 3:

[0861] The user accesses the portal and enters their inquiry (e.g., "What products are available at this store?"). The terminal converts the data entered in the input fields into text format in real time and sends that data to the server. The server analyzes the received inquiry data (text-formatted question) and retrieves the user's native language information from the database.

[0862] Step 4:

[0863] The server sends the query content and the user's native language information to the generating AI model, which then generates a prompt (e.g., "Please translate the following information from Japanese to English"). The generating AI model then generates an appropriate response based on the input query content and prompt, creating a reply in the user's native language. For example, if the query is "Where is this rice produced?", it will generate a response such as "This rice is produced in Niigata."

[0864] Step 5:

[0865] The server sends the generated response to the user's terminal. The terminal displays the received response (the user's native language) on the screen, providing it in a format that is easy for the user to understand. The user can review this displayed information and enter further questions if necessary.

[0866] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0867] This invention is a system that incorporates a high-performance conversational AI and emotion recognition engine into a multilingual portal, thereby solving the language barriers and lack of information that foreigners residing in Japan face in their daily lives. Specific embodiments of this system are described below.

[0868] 1. User registration and login

[0869] The user installs the application and starts the registration process by pressing the "Register" button upon first launch. The device displays a form for entering basic information such as name, email address, password, and native language, prompting the user to fill it out. The entered data is sent to the server.

[0870] The server saves the received data to its database and sends a registration success message back to the terminal. The user confirms this message and completes the account registration.

[0871] Afterward, the user logs in using the email address and password they entered during registration. The device sends the entered login information to the server, which authenticates the user. If authentication is successful, the user's information is sent to the device, and the user can access the portal.

[0872] 2. Receiving and processing inquiries

[0873] The user accesses the portal and enters an inquiry such as, "Where are the hospitals where English is spoken?" The terminal converts the data entered in the input fields into text format in real time and sends it to the server.

[0874] The server analyzes the received query data and retrieves native language information from the database based on the user ID. Simultaneously, the emotion engine analyzes the user's emotional state from their input and sends that information to the server. The server then sends the analysis results and emotional state information to the AI ​​model to generate an appropriate response.

[0875] 3. Response generation and reply

[0876] The generative AI model processes information based on the inquiry and considers emotional state information when generating an appropriate response in the user's native language. For example, if the user expresses anxiety, the generative AI model will generate an empathetic response that alleviates that anxiety.

[0877] The server sends the generated response to the user's terminal. The terminal displays the received response on the screen, making it easy for the user to understand. The user reviews the displayed response and can enter further questions if necessary.

[0878] Specific example

[0879] Let's say a user enters the question, "What medicine is effective for a cold?", and the emotion engine analyzes the user's emotional state as "anxious". The device sends the entered data and emotional state to the server, which then sends it to a generative AI model. The generative AI model generates a response such as, "Paracetamol and Ipuhen are effective for colds. Don't worry, these medicines are reliable." The server sends this response back to the user, and the device displays it to the user, allowing them to receive appropriate information with peace of mind.

[0880] In this way, the system of the present invention solves the communication challenges faced by foreign residents in Japan and provides an environment in which users can easily obtain necessary information in their native language. Furthermore, by providing responses that correspond to the user's emotional state through the emotion engine, it achieves more personalized support.

[0881] The following describes the processing flow.

[0882] Step 1:

[0883] The user installs the application and presses the "Register" button upon first launch. The device displays a form for the user to enter their name, email address, password, and native language.

[0884] Step 2:

[0885] The user enters the required information into the registration form and presses the submit button. The device sends the entered information to the server.

[0886] Step 3:

[0887] The server saves the received user information to the database and sends a registration success message back to the terminal. The terminal then displays this message to the user.

[0888] Step 4:

[0889] The user logs into the app. The device receives the user's email address and password and sends them to the server.

[0890] Step 5:

[0891] The server authenticates the user by comparing the information with that in the database and sends the authentication result back to the terminal. If authentication is successful, the user can access the portal.

[0892] Step 6:

[0893] The user accesses the portal and enters their inquiry. For example, they might enter, "Where are the hospitals where English is spoken?" The device then sends the entered information as data to the server.

[0894] Step 7:

[0895] The server analyzes the received query data and retrieves native language information from the database based on the user ID. Simultaneously, the emotion engine analyzes the user's emotional state from their input and sends that information to the server.

[0896] Step 8:

[0897] The server sends the query content and emotional state information to the generating AI model and instructs it to generate an appropriate response. The generating AI model then generates a response based on the query content and emotional state.

[0898] Step 9:

[0899] The generative AI model sends the generated response back to the server. For example, if a user asks, "What medicine is effective for a cold?" and their emotional state is analyzed as "anxious," a response such as, "Paracetamol and ibuchen are effective for colds. Don't worry, these medicines are reliable," is generated.

[0900] Step 10:

[0901] The server sends the generated response to the user's terminal. The terminal displays the received response on the user's screen.

[0902] Step 11:

[0903] The user can review the displayed response and gain a sense of reassurance. The user can enter further questions if necessary. The server then performs the same process again, generating an appropriate response and sending it back.

[0904] (Example 2)

[0905] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0906] The language barrier and lack of information that foreign residents in Japan face are major obstacles in various aspects of their daily lives. Misunderstandings and stress resulting from the inability to properly communicate emotions are also problematic. Because existing systems have not adequately addressed these challenges, improvements in usability are needed.

[0907] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0908] In this invention, the server includes means for registering the user's basic information, means for inputting the user's inquiry, means for transmitting the inputted inquiry to the server, means for analyzing the inquiry and generating a response based on the user's native language, means for transmitting and displaying the generated response on the user's terminal, means for acquiring emotional state information using an emotional recognition engine that analyzes the user's emotional state from the user's input, and means for generating a response considering the emotional state information. This provides an environment in which foreign residents can overcome language barriers and lack of information, appropriately convey their emotions, and enable more comfortable communication.

[0909] A "foreign resident" refers to a person who has emigrated to Japan from another country and lives in Japan for a certain period of time or permanently.

[0910] A "multilingual portal" refers to a web system or application that provides information in multiple languages ​​and allows users to interact with it in their chosen language.

[0911] "Means for users to register basic information" refers to the interface through which users enter and submit personal information such as their name, email address, password, and native language.

[0912] "Means for users to input inquiry details" refers to an interface for users to input questions or requests in text format and send them to the system.

[0913] "Means of sending the entered inquiry content to the server" refers to a function that captures information entered by the user in real time and sends the data to the server via the network.

[0914] "Means by which the server analyzes the query content and generates a response based on the user's native language" refers to the function by which the server processes the received data and generates an appropriate response in the user's native language using a database or algorithm.

[0915] "Means of sending and displaying the generated response on the user's terminal" refers to a function that sends the response generated by the server to the user's device and displays it in a format that the user can easily view.

[0916] An "emotion recognition engine" refers to algorithms and software that analyze a user's emotional state from text data, audio data, and other sources, and extract emotional information.

[0917] "Means of generating responses while considering emotional state information" refers to a function that analyzes the user's emotional state and generates more personalized responses based on that information.

[0918] "Login authentication method" refers to the process or function that allows a user to log in to a system using registered credentials (e.g., email address and password) and verify access rights.

[0919] This invention is a system that incorporates a high-performance conversational AI and emotion recognition engine into a multilingual portal, thereby solving the language barrier and lack of information that foreigners residing in Japan face in their daily lives. Specific embodiments of this invention are described in detail below.

[0920] 1. User registration and login

[0921] The user first installs the application and then starts the registration process by pressing the "Register" button upon initial launch. The terminal displays a form prompting the user to enter basic information such as name, email address, password, and native language. Once the user enters this information and presses the "Submit" button, the terminal sends the entered data to the server.

[0922] The server stores the received user information in a database (using MySQL or PostgreSQL as the database). Upon successful registration, the server sends a success message back to the terminal, which the user confirms to complete account registration.

[0923] Afterward, the user logs in using the email address and password they entered during registration. The device sends the login information to the server, which authenticates the user by comparing it with the information stored in the database. If authentication is successful, the user's information is sent to the device, and the user can access the portal.

[0924] 2. Receiving and processing inquiries

[0925] Users access the portal and enter inquiries such as, "Where are the hospitals where English is spoken?" The terminal converts the entered data into text format in real time and sends it to the server. The server analyzes the received data and retrieves native language information from the database based on the user ID.

[0926] Simultaneously, an emotion recognition engine (e.g., IBM Watson or Microsoft Azure Sentiment Analysis) is used to analyze the user's emotional state from their input. The emotional state information is sent to a server, which then sends it along with the analysis results to a generating AI model (e.g., GPT-3 or GPT-4).

[0927] 3. Response generation and reply

[0928] The generative AI model processes information based on the inquiry and generates an appropriate response in the user's native language. In doing so, it considers emotional state information, ensuring that the response is sensitive to the user's feelings.

[0929] For example, if a user enters the question "What medicine is effective for a cold?" and the emotion recognition engine analyzes the user's emotional state as "anxious," the generative AI model will generate a response such as "Paracetamol and Ipuhen are effective for colds. Don't worry, these medicines are reliable." The server sends this response to the user's device, which then displays it to the user.

[0930] Example of a prompt:

[0931] Question from a user: "What medicine is effective for colds?"

[0932] User's emotional state: "Anxious"

[0933] Response generated: "Paracetamol and ibuchen are effective for colds. Don't worry, these medicines are reliable."

[0934] In this way, the system of the present invention solves the communication challenges faced by foreign residents in Japan and provides an environment in which users can easily obtain necessary information in their native language. Furthermore, by using an emotion recognition engine, it provides responses that correspond to the user's emotional state, realizing more personalized support.

[0935] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0936] Step 1:

[0937] The user installs the application and presses the "Register" button upon first launch. The device displays a form prompting the user to enter basic information such as name, email address, password, and native language. After the user enters this information and presses the "Submit" button, the device sends the entered data to the server.

[0938] Input: Name, email address, password, native language

[0939] Output: Basic user information sent to the server

[0940] Step 2:

[0941] The server saves the user information received from the terminal to a database (MySQL, PostgreSQL). If the saving is successful, the server sends a registration success message back to the terminal. The user confirms this message and completes the account registration.

[0942] Input: Basic user information sent from the device.

[0943] Output: User information stored in the database, registration success message

[0944] Step 3:

[0945] Users log in using the email address and password they entered during registration. The device sends the login information to the server, which authenticates the user by comparing it with the information stored in the database. If authentication is successful, the server sends the user's information back to the device, and the user can then access the portal.

[0946] Enter: Email address, password

[0947] Output: Authentication result, access rights to the portal

[0948] Step 4:

[0949] The user accesses the portal and enters an inquiry such as, "Where are the hospitals where English is spoken?" The terminal converts the entered data into text format in real time and sends it to the server.

[0950] Input: Text data of the inquiry.

[0951] Output: Query data sent to the server

[0952] Step 5:

[0953] The server analyzes the received data and retrieves native language information from the database based on the user ID. Simultaneously, it analyzes the user's emotional state from their input using an emotion recognition engine (IBM Watson, Microsoft Azure Sentiment Analysis). The emotional state information is then sent back to the server.

[0954] Input: Inquiry data, User ID

[0955] Output: Native language information, emotional state information

[0956] Step 6:

[0957] The server sends the analysis results and emotional state information to the generating AI models (GPT-3, GPT-4). The generating AI models process the information based on the query and generate an appropriate response in the user's native language. In doing so, they take the emotional state information into consideration and generate a response that is sensitive to the user's emotions.

[0958] Input: Analysis results, emotional state information

[0959] Output: Generated response

[0960] Step 7:

[0961] The server sends the generated response to the terminal. The terminal displays the received response on the screen, presenting it in a way that is easy for the user to understand. The user reviews the displayed information and can enter further questions if necessary.

[0962] Input: Generated response

[0963] Output: Response displayed on the terminal

[0964] (Application Example 2)

[0965] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0966] This invention aims to provide a multilingual portal system to address the language barriers and lack of information that foreigners residing in Japan face in their daily lives. It also aims to provide more personalized support by offering responses that take into account the user's emotional state. In particular, it aims to improve the user experience by providing appropriate support tailored to the user's emotions during the shopping experience within virtual stores.

[0967] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0968] In this invention, the server includes means for the user to register basic information, means for the user to input inquiry content, means for transmitting the input inquiry content to the server, means for the server to analyze the inquiry content and generate a response based on the user's native language, means for transmitting and displaying the generated response to the user's terminal, means for using an emotion recognition engine to analyze the user's emotional state, and means for considering the user's emotional state information when generating the response. This enables personalized shopping support tailored to the user's emotions within a virtual store.

[0969] A "multilingual portal" is an information provision system that supports multiple languages, allowing users to make inquiries in multiple languages.

[0970] "User" refers to anyone who uses the system, and is not limited to, but includes, in particular, foreigners residing in Japan.

[0971] "Basic information" refers to the minimum information necessary to use the system, such as the user's name, email address, password, and native language.

[0972] "Inquiry content" refers to the information or questions that the user wants to know.

[0973] A "server" refers to a computer system that receives and analyzes data from users, generates the necessary responses, and sends them back.

[0974] "Native language" refers to the language the user primarily uses, and the system generates responses based on this language.

[0975] An "emotion recognition engine" refers to a software function that analyzes and identifies a user's emotional state based on their input and statements.

[0976] "Response generation" refers to the process by which a server creates an appropriate response based on the query content, native language information, and sentiment status information.

[0977] A "generative AI model" refers to an algorithm or system that uses artificial intelligence technology to automatically generate responses based on the content of an inquiry.

[0978] A "virtual store" refers to a virtual store that exists on the internet, and is a platform that allows users to browse and purchase products online.

[0979] A "shopping assistant application" refers to an application that allows users to obtain information about products and receive assistance with purchasing within a virtual store.

[0980] "Personalized support" refers to support that provides individualized assistance tailored to each user's needs and emotional state.

[0981] The system of this invention is a multilingual portal and can be implemented particularly as a shopping assistant application within a virtual store. The server, terminal, and user work together. Specific embodiments are described below.

[0982] 1. User registration and login

[0983] The user first installs the application and starts the registration process by pressing the "Register" button upon initial launch. The device displays a form for entering basic information such as name, email address, password, and native language, prompting the user to fill it out. The entered data is sent to the server, which stores it in a database and sends a registration success message back to the device. The user confirms this message and completes account registration. After that, the user logs in using the email address and password entered during registration. The device sends the entered login information to the server, which authenticates the user. If authentication is successful, the user's information is sent to the device, and the user can access the portal.

[0984] 2. Receiving and processing inquiries

[0985] Users ask questions about products and services within a virtual store. For example, they might input an inquiry such as, "I'd like to know more about this camera." The terminal converts the data entered in the input field into text format in real time and sends it to the server. The server analyzes the received inquiry data and retrieves native language information from the database based on the user ID. Simultaneously, an emotion recognition engine analyzes the emotional state from the user's input and sends that information to the server. The server sends the analysis results and emotional state information to a generating AI model to generate an appropriate response.

[0986] 3. Response generation and reply

[0987] The generative AI model processes information based on the inquiry and considers emotional state information when generating an appropriate response in the user's native language. For example, suppose a user asks, "Tell me more about this camera," and the emotion recognition engine analyzes the user's emotional state as "interested." The generative AI model then generates a response such as, "This camera features the latest optical zoom technology and is particularly suitable for landscape photography." The server sends the generated response to the user's device. The device displays the received response on its screen, making it easy for the user to understand. The user can review the displayed response and enter further questions if necessary.

[0988] As a concrete example, we can use the text "I'd like to know more about this camera" as an example of a prompt to ask a question to a generative AI model. This prompt allows the user to appropriately obtain the necessary information.

[0989] Specific examples of hardware and software used include the emotion recognition engine and generative AI models "EmotionRecognizer" and "Hugging Face Transformers." By utilizing these, a system that provides personalized responses based on the user's emotional state can be realized.

[0990] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0991] Step 1:

[0992] User registration and login

[0993] Input: The user enters their name, email address, password, and native language.

[0994] Output: User registration information database saving success message

[0995] Operation: The user installs the application and enters basic information such as name, email address, password, and native language. The device sends the data to the server, which stores it in a database. After saving, a registration success message is sent back to the device. The user confirms this message and completes account registration. The user then enters their email address and password to log in. The device sends the entered information to the server, which performs authentication. If authentication is successful, user information is sent to the device, and the user can access the portal.

[0996] Step 2:

[0997] Enter your inquiry details

[0998] Input: The user enters the question in text format.

[0999] Output: The entered query content is sent from the terminal to the server.

[1000] Operation: The user enters their question within the virtual store. The terminal displays the question in text format in the input field and sends it to the server. Example: "I'd like to know more about this camera."

[1001] Step 3:

[1002] Analysis of query data

[1003] Input: Query data received by the server

[1004] Output: Native language information, emotional state information

[1005] Operation: The server analyzes the input query data and retrieves native language information from the database based on the user ID. Simultaneously, the emotion recognition engine analyzes the user's emotional state from their input and sends that information to the server.

[1006] Step 4:

[1007] Response generation

[1008] Input: Inquiry data, native language information, emotional state information

[1009] Output: Generated response text

[1010] Operation: The server sends the analysis results and emotional state information to a generating AI model to generate an appropriate response. The generating AI model considers the query content and emotional state to generate an appropriate response in the user's native language. Example: "This camera is equipped with the latest optical zoom technology and is especially suitable for landscape photography."

[1011] Step 5:

[1012] Sending and displaying generated responses

[1013] Input: Generated response text

[1014] Output: Response displayed on the user terminal

[1015] Operation: The server sends the generated response to the user's terminal. The terminal displays the received response on the screen to make it easy for the user to understand. The user reviews the displayed response and can enter further questions if necessary.

[1016] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1017] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1018] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1019] [Fourth Embodiment]

[1020] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1021] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1022] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1023] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1024] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1025] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1026] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1027] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1028] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1029] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1030] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1031] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1032] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1033] This invention is a system that incorporates a high-performance conversational AI into a multilingual portal, thereby solving the language barriers and lack of information that foreigners residing in Japan face in their daily lives. Specific embodiments of this system are described below.

[1034] 1. User registration and login

[1035] The user installs the application and starts the registration process by pressing the "Register" button upon first launch. The device displays a form for entering basic information such as name, email address, password, and native language, prompting the user to fill it out. The entered data is sent to the server.

[1036] The server saves the received data to its database and sends a registration success message back to the terminal. The user confirms this message and completes the account registration.

[1037] Afterward, the user logs in using the email address and password they entered during registration. The device sends the entered login information to the server, which authenticates the user. If authentication is successful, the user's information is sent to the device, and the user can access the portal.

[1038] 2. Receiving and processing inquiries

[1039] The user accesses the portal and enters an inquiry such as, "Where are the hospitals where English is spoken?" The terminal converts the data entered in the input fields into text format in real time and sends it to the server.

[1040] The server analyzes the received query data and retrieves the user's native language information from the database. Then, it sends the query content and the user's native language to a generating AI model to produce an appropriate response.

[1041] 3. Response generation and reply

[1042] The generative AI model processes information based on the inquiry and generates an appropriate response in the user's native language. For example, in response to the inquiry, "Where are the hospitals where they speak English?", the generative AI model will generate a specific response such as, "There are several hospitals in Tokyo where they speak English."

[1043] The server sends the generated response to the user's terminal. The terminal displays the received response on the screen, making it easy for the user to understand. The user reviews the displayed response and can enter further questions if necessary.

[1044] Specific example

[1045] Suppose a user becomes ill and enters the question, "What medicine is effective for a cold?" The device sends the entered data to a server, which retrieves the user's native language from its database and then sends the query to a generative AI model. The generative AI model generates a response such as, "Paracetamol and ibuchen are effective for colds," and the server sends this response back to the user. Finally, the device displays this message to the user, allowing them to instantly obtain the necessary information.

[1046] In this way, the system of the present invention solves the communication challenges that foreign residents face while living in Japan and provides an environment in which users can easily obtain necessary information in their native language.

[1047] The following describes the processing flow.

[1048] Step 1:

[1049] The user installs the application and presses the "Register" button upon first launch. The device displays a form for the user to enter their name, email address, password, and native language.

[1050] Step 2:

[1051] The user enters the required information into the registration form and presses the submit button. The device sends the entered information to the server.

[1052] Step 3:

[1053] The server saves the received user information to the database and sends a registration success message back to the terminal. The terminal then displays this message to the user.

[1054] Step 4:

[1055] The user logs into the app. The device receives the user's email address and password and sends them to the server.

[1056] Step 5:

[1057] The server authenticates the user by comparing the information with that in the database and sends the authentication result back to the terminal. If authentication is successful, the user can access the portal.

[1058] Step 6:

[1059] The user accesses the portal and enters their inquiry. For example, they might enter, "Where are the hospitals where English is spoken?" The device then sends the entered information as data to the server.

[1060] Step 7:

[1061] The server analyzes the received query data and retrieves native language information from the database based on the user ID. The analysis results and the user's native language information are then sent to the AI ​​model for generation.

[1062] Step 8:

[1063] The generative AI model generates a response based on the inquiry and translates it into the user's native language. For example, in response to the inquiry, "Where are there hospitals where they speak English?", it generates the response, "There are several hospitals in Tokyo where they speak English."

[1064] Step 9:

[1065] The server sends the generated response to the user's terminal. The terminal displays the received response on the screen and presents it to the user.

[1066] Step 10:

[1067] The user reviews the displayed response and enters further questions if necessary. The server repeats the same process, generating an appropriate response and sending it back.

[1068] (Example 1)

[1069] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1070] Foreign residents in Japan often face language barriers and a lack of information in their daily lives. This problem stems from the difficulty in quickly and accurately obtaining essential information for daily life, such as the location of medical facilities and how to obtain necessary medications. This issue becomes particularly serious in emergencies or when critical information is needed. Current multilingual systems fail to provide accurate responses and timely information, thus compromising user convenience.

[1071] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1072] In this invention, the server includes means for the user to register basic information, means for the user to input inquiry content, means for transmitting the inputted inquiry content to the server, means for the server to analyze the inquiry content and obtain the user's native language information from a database, means for the server to generate a response using a generation AI model based on the inquiry content, and means for transmitting and displaying the generated response to the user's terminal. As a result, the process from user information input to response acquisition is carried out quickly and accurately, and the user can easily obtain the necessary information in their native language.

[1073] A "user" refers to an individual or group that uses the system, and in particular to foreign residents who access the multilingual portal to obtain information.

[1074] "Basic information" refers to personal information such as the user's name, email address, password, and native language, which is necessary to register with the system.

[1075] "Inquiry content" refers to specific information such as questions and requests that users enter through the system, for example, "Where are the hospitals where English is spoken?"

[1076] A "terminal" is a device used by a user to access a system and input and retrieve information, and includes smartphones, tablets, PCs, and other similar devices.

[1077] A "server" refers to a central control unit that analyzes and processes user data and inquiries, and is responsible for data storage and response generation.

[1078] A "database" refers to a system that centrally stores and manages various types of data required by the system, such as basic user information and native language information.

[1079] A "generative AI model" refers to an algorithm or program that uses artificial intelligence technology to generate appropriate responses based on user inquiries.

[1080] "Response" refers to the information returned to the user as a result of analysis and processing by the server and the generating AI model, and is provided in the user's native language.

[1081] "Login authentication" refers to the procedure used to verify that a user is a legitimate user when they access a system using their account.

[1082] A "multilingual portal" refers to a web-based or application-based information access platform that supports multiple languages ​​and is designed to allow users to obtain the information they need in their native language.

[1083] This invention is a system that incorporates a high-performance conversational AI into a multilingual portal, thereby solving the language barriers and lack of information that foreigners residing in Japan face in their daily lives. Specific embodiments of this system are described below.

[1084] User registration and login

[1085] The user installs the application and starts the registration process by pressing the "Register" button upon first launch. The device displays a form for the user to enter basic information such as name, email address, password, and native language. After the user enters this information and presses the "Submit" button, the entered information is sent to the server. The server saves the received data to its database and sends a confirmation message back to the device, acknowledging that the save was successful. The user confirms the displayed message and completes the account registration. After that, the user logs in using the email address and password entered during registration. The device sends the entered login information to the server, which performs authentication. If authentication is successful, the server returns the user information to the device, and the user can access the portal.

[1086] Inquiry reception and processing

[1087] A user accesses the portal and enters an inquiry, such as "Where are the hospitals where English is spoken?", into the input field. The terminal converts this input data into text format in real time and sends it to the server. The server analyzes the received inquiry and retrieves the user's native language information from the database. Then, it sends the inquiry and native language information to an AI model to generate an appropriate response.

[1088] Response generation and response

[1089] The generative AI model processes the inquiry content and native language information received from the server and generates an appropriate response. For example, in response to the question, "Where are the hospitals where they speak English?", it generates a specific answer such as, "There are several hospitals in Tokyo where they speak English." The server sends the generated response to the user's terminal, which displays the received response on the screen. The user can then review the displayed response and enter further questions.

[1090] Specific example

[1091] For example, suppose a user has a cold and enters the question, "What medicine is effective for a cold?" The terminal sends this input data to the server, which analyzes the query and retrieves the user's native language information from the database. Then, it sends the query "What medicine is effective for a cold?" to a generative AI model. The generative AI model generates a response such as "Paracetamol and Ibchen are effective for colds." The server sends this response to the terminal, which displays this message to the user. The user can instantly obtain the information they need.

[1092] Example of a prompt

[1093] "I want to know about cold medicine."

[1094] "Can you recommend a hospital where they speak English?"

[1095] "Does the city hall offer foreign language support?"

[1096] In this way, the system of the present invention eliminates the language barrier and lack of information that foreign residents face in their daily lives, and provides an environment in which users can easily obtain the information they need in their native language.

[1097] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1098] Step 1:

[1099] The user installs the application and clicks the "Register" button upon first launch. An input form will appear, and the user will enter basic information such as their name, email address, password, and native language.

[1100] Input: User's basic information (name, email address, password, native language)

[1101] Output: Input basic information

[1102] Step 2:

[1103] The terminal generates a request to send the entered basic information to the server and then sends the data.

[1104] Input: Basic information entered

[1105] Output: Request to send to the server

[1106] Step 3:

[1107] The server saves the received data to the database and generates a confirmation message acknowledging the successful saving. It then sends the confirmation message back to the terminal.

[1108] Input: Basic information sent to the server

[1109] Output: Confirmation message for successful saving

[1110] Step 4:

[1111] The user confirms the message displayed on their device and completes the account registration.

[1112] Input: Confirmation message for successful saving

[1113] Output: Account registration complete

[1114] Step 5:

[1115] Users log in using the email address and password they entered during registration.

[1116] Enter: Email address, password

[1117] Output: Login Request

[1118] Step 6:

[1119] The terminal sends the entered login information to the server.

[1120] Enter: Email address, password

[1121] Output: Request to send to the server

[1122] Step 7:

[1123] The server checks the received login information against its database and performs authentication. If authentication is successful, it sends the user information back to the terminal.

[1124] Input: Login information (email address, password)

[1125] Output: User information

[1126] Step 8:

[1127] The terminal displays the received user information, and the user can then access the portal.

[1128] Input: User information

[1129] Output: Access permission to the portal

[1130] Step 9:

[1131] The user accesses the portal and enters an inquiry such as, "Where are the hospitals where English is spoken?"

[1132] Input: Inquiry details

[1133] Output: Query content in text format

[1134] Step 10:

[1135] The terminal converts the entered inquiry content into text format and sends it to the server.

[1136] Input: Inquiry content in text format

[1137] Output: Request to send to the server

[1138] Step 11:

[1139] The server analyzes the received query and retrieves the user's native language information from the database. It then sends the query content and native language information to the AI ​​model for generating queries.

[1140] Input: Inquiry content in text format, native language information

[1141] Output: Request to send to the generating AI model

[1142] Step 12:

[1143] The generative AI model generates an appropriate response based on the query content and native language information.

[1144] Input: Inquiry details, native language information

[1145] Output: Generated response

[1146] Step 13:

[1147] The server sends the generated response to the user's terminal.

[1148] Input: Generated response

[1149] Output: Request to send to the terminal

[1150] Step 14:

[1151] The terminal displays the received response on the screen so that the user can confirm it.

[1152] Input: Generated response

[1153] Output: Displayed response

[1154] Step 15:

[1155] The user reviews the displayed response and enters further questions if necessary.

[1156] Input: Displayed response

[1157] Output: Additional questions

[1158] (Application Example 1)

[1159] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1160] Foreign residents and tourists in Japan often face language barriers and a lack of information when trying to understand product information and store directions in physical stores, making smooth shopping difficult. This frequently leads to inconvenience for foreigners who cannot quickly and accurately obtain the information they need. To solve this problem, there is a need to provide multilingual systems.

[1161] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1162] In this invention, the server includes means for the user to register basic information, means for the user to input inquiry content, means for transmitting the input inquiry content to the server, means for the server to analyze the inquiry content and generate a response based on the user's native language, means for transmitting and displaying the generated response on the user's terminal, and means for providing product information and store guidance in the native language at physical stores. This makes it possible for foreigners to easily obtain product information and store guidance in their native language at physical stores.

[1163] "Foreign residents" refers to foreigners who stay in Japan for an extended period, and may also include temporary tourists.

[1164] A "multilingual portal" refers to a web portal or application that supports multiple languages ​​and provides information in the language selected by the user.

[1165] "Basic information" refers to the minimum information necessary to identify a user and register them in the system, such as the user's name, email address, password, and native language.

[1166] "Inquiry content" refers to the information or questions the user wants to know, and the data used by the system to generate a response based on this.

[1167] A "server" is a system that provides specific services on a computer network, and it handles tasks such as storing, processing, and transmitting data.

[1168] A "generative AI model" is a type of artificial intelligence that refers to an algorithm used to generate appropriate responses or information based on input data.

[1169] A "prompt sentence" refers to an introductory sentence or question that a generative AI model uses to generate an appropriate response.

[1170] "Login authentication" refers to a method used to verify a user's identity when accessing a system, and typically involves using a username and password.

[1171] A "physical store" refers to a sales facility that exists physically, a place where users can go in person to purchase products.

[1172] "Product information" refers to detailed information that users need to make a purchase decision, such as the price, description, and origin of the product.

[1173] This invention is a multilingual portal system that allows foreign residents to easily obtain product information and store guides in their native language at physical stores. Specific embodiments of this system are described below.

[1174] User registration and login

[1175] The registration process begins when the user installs the application and presses the registration button upon first launch. The device displays a form for basic information such as name, email address, password, and native language, prompting the user to enter the information. The entered data is sent to the server, which stores the received data in its database and sends a registration success message back to the device. The user confirms this message and completes the account registration. After that, the user logs in using the email address and password entered during registration. The device sends the entered login information to the server, which then authenticates the user.

[1176] Inquiry reception and processing

[1177] A user accesses the portal and enters an inquiry, such as, "What products can I buy at this store?" The terminal converts the data entered in the input fields into text format in real time and sends it to the server. The server analyzes the received inquiry data and retrieves the user's native language information from the database. Then, it sends the inquiry content and the user's native language to a generating AI model to produce an appropriate response. The generated response is sent from the server to the user's terminal and displayed on the screen.

[1178] Use at physical stores

[1179] When a user needs information such as "I want to know more about the product" or "I want to know more about the store" at a physical store, they similarly input a question into the application. For example, they might enter an inquiry like, "Where is this rice produced?" The server receives this inquiry, retrieves the product information from the database, and sends it to a generative AI model to translate it into the user's native language. The generative AI model generates an appropriate response in the user's native language based on the inquiry, producing a specific answer such as, "This rice is produced in Niigata."

[1180] Response generation and response

[1181] The generative AI model processes information based on the inquiry and generates an appropriate response in the user's native language. The server sends the generated response to the user's device and displays it on the user's screen. The user can review the displayed response and enter further questions if necessary.

[1182] Program usage examples

[1183] As a concrete example, if a user wants to know information about rice at a Japanese supermarket, they would use the app to ask "What is the price and origin of the rice?" This question is sent to a generative AI model, and the translated result is displayed in the user's native language.

[1184] Example of a prompt:

[1185] Please translate the following product information from Japanese to English: "The price of the rice is 1200 yen, and it is produced in Niigata."

[1186] Hardware and software used

[1187] The hardware used in this system is a smartphone, and the software employs Python, utilizing the Transformers library for generating AI models. On the server side, a database management system is used to store and manage user data and product information.

[1188] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1189] Step 1:

[1190] The registration process begins when the user installs the application and presses the registration button upon first launch. The terminal displays a form for entering basic information such as name, email address, password, and native language, prompting the user to fill it out. The entered data (name, email address, password, native language) is sent to the server. The server stores the received data in its database, generates a registration success message, and sends it to the terminal. The terminal then displays this message to the user.

[1191] Step 2:

[1192] The user logs in using the email address and password entered during registration. The terminal sends the entered login information (email address, password) to the server. The server compares the received login information with the data in the database and performs authentication. If authentication is successful, the server generates and sends a login success message to the user. The terminal displays this message to the user.

[1193] Step 3:

[1194] The user accesses the portal and enters their inquiry (e.g., "What products are available at this store?"). The terminal converts the data entered in the input fields into text format in real time and sends that data to the server. The server analyzes the received inquiry data (text-formatted question) and retrieves the user's native language information from the database.

[1195] Step 4:

[1196] The server sends the query content and the user's native language information to the generating AI model, which then generates a prompt (e.g., "Please translate the following information from Japanese to English"). The generating AI model then generates an appropriate response based on the input query content and prompt, creating a reply in the user's native language. For example, if the query is "Where is this rice produced?", it will generate a response such as "This rice is produced in Niigata."

[1197] Step 5:

[1198] The server sends the generated response to the user's terminal. The terminal displays the received response (the user's native language) on the screen, providing it in a format that is easy for the user to understand. The user can review this displayed information and enter further questions if necessary.

[1199] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1200] This invention is a system that incorporates a high-performance conversational AI and emotion recognition engine into a multilingual portal, thereby solving the language barriers and lack of information that foreigners residing in Japan face in their daily lives. Specific embodiments of this system are described below.

[1201] 1. User registration and login

[1202] The user installs the application and starts the registration process by pressing the "Register" button upon first launch. The device displays a form for entering basic information such as name, email address, password, and native language, prompting the user to fill it out. The entered data is sent to the server.

[1203] The server saves the received data to its database and sends a registration success message back to the terminal. The user confirms this message and completes the account registration.

[1204] Afterward, the user logs in using the email address and password they entered during registration. The device sends the entered login information to the server, which authenticates the user. If authentication is successful, the user's information is sent to the device, and the user can access the portal.

[1205] 2. Receiving and processing inquiries

[1206] The user accesses the portal and enters an inquiry such as, "Where are the hospitals where English is spoken?" The terminal converts the data entered in the input fields into text format in real time and sends it to the server.

[1207] The server analyzes the received query data and retrieves native language information from the database based on the user ID. Simultaneously, the emotion engine analyzes the user's emotional state from their input and sends that information to the server. The server then sends the analysis results and emotional state information to the AI ​​model to generate an appropriate response.

[1208] 3. Response generation and reply

[1209] The generative AI model processes information based on the inquiry and considers emotional state information when generating an appropriate response in the user's native language. For example, if the user expresses anxiety, the generative AI model will generate an empathetic response that alleviates that anxiety.

[1210] The server sends the generated response to the user's terminal. The terminal displays the received response on the screen, making it easy for the user to understand. The user reviews the displayed response and can enter further questions if necessary.

[1211] Specific example

[1212] Let's say a user enters the question, "What medicine is effective for a cold?", and the emotion engine analyzes the user's emotional state as "anxious". The device sends the entered data and emotional state to the server, which then sends it to a generative AI model. The generative AI model generates a response such as, "Paracetamol and Ipuhen are effective for colds. Don't worry, these medicines are reliable." The server sends this response back to the user, and the device displays it to the user, allowing them to receive appropriate information with peace of mind.

[1213] In this way, the system of the present invention solves the communication challenges faced by foreign residents in Japan and provides an environment in which users can easily obtain necessary information in their native language. Furthermore, by providing responses that correspond to the user's emotional state through the emotion engine, it achieves more personalized support.

[1214] The following describes the processing flow.

[1215] Step 1:

[1216] The user installs the application and presses the "Register" button upon first launch. The device displays a form for the user to enter their name, email address, password, and native language.

[1217] Step 2:

[1218] The user enters the required information into the registration form and presses the submit button. The device sends the entered information to the server.

[1219] Step 3:

[1220] The server saves the received user information to the database and sends a registration success message back to the terminal. The terminal then displays this message to the user.

[1221] Step 4:

[1222] The user logs into the app. The device receives the user's email address and password and sends them to the server.

[1223] Step 5:

[1224] The server authenticates the user by comparing the information with that in the database and sends the authentication result back to the terminal. If authentication is successful, the user can access the portal.

[1225] Step 6:

[1226] The user accesses the portal and enters their inquiry. For example, they might enter, "Where are the hospitals where English is spoken?" The device then sends the entered information as data to the server.

[1227] Step 7:

[1228] The server analyzes the received query data and retrieves native language information from the database based on the user ID. Simultaneously, the emotion engine analyzes the user's emotional state from their input and sends that information to the server.

[1229] Step 8:

[1230] The server sends the query content and emotional state information to the generating AI model and instructs it to generate an appropriate response. The generating AI model then generates a response based on the query content and emotional state.

[1231] Step 9:

[1232] The generative AI model sends the generated response back to the server. For example, if a user asks, "What medicine is effective for a cold?" and their emotional state is analyzed as "anxious," a response such as, "Paracetamol and ibuchen are effective for colds. Don't worry, these medicines are reliable," is generated.

[1233] Step 10:

[1234] The server sends the generated response to the user's terminal. The terminal displays the received response on the user's screen.

[1235] Step 11:

[1236] The user can review the displayed response and gain a sense of reassurance. The user can enter further questions if necessary. The server then performs the same process again, generating an appropriate response and sending it back.

[1237] (Example 2)

[1238] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1239] The language barrier and lack of information that foreign residents in Japan face are major obstacles in various aspects of their daily lives. Misunderstandings and stress resulting from the inability to properly communicate emotions are also problematic. Because existing systems have not adequately addressed these challenges, improvements in usability are needed.

[1240] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1241] In this invention, the server includes means for registering the user's basic information, means for inputting the user's inquiry, means for transmitting the inputted inquiry to the server, means for analyzing the inquiry and generating a response based on the user's native language, means for transmitting and displaying the generated response on the user's terminal, means for acquiring emotional state information using an emotional recognition engine that analyzes the user's emotional state from the user's input, and means for generating a response considering the emotional state information. This provides an environment in which foreign residents can overcome language barriers and lack of information, appropriately convey their emotions, and enable more comfortable communication.

[1242] A "foreign resident" refers to a person who has emigrated to Japan from another country and lives in Japan for a certain period of time or permanently.

[1243] A "multilingual portal" refers to a web system or application that provides information in multiple languages ​​and allows users to interact with it in their chosen language.

[1244] "Means for users to register basic information" refers to the interface through which users enter and submit personal information such as their name, email address, password, and native language.

[1245] "Means for users to input inquiry details" refers to an interface for users to input questions or requests in text format and send them to the system.

[1246] "Means of sending the entered inquiry content to the server" refers to a function that captures information entered by the user in real time and sends the data to the server via the network.

[1247] "Means by which the server analyzes the query content and generates a response based on the user's native language" refers to the function by which the server processes the received data and generates an appropriate response in the user's native language using a database or algorithm.

[1248] "Means of sending and displaying the generated response on the user's terminal" refers to a function that sends the response generated by the server to the user's device and displays it in a format that the user can easily view.

[1249] An "emotion recognition engine" refers to algorithms and software that analyze a user's emotional state from text data, audio data, and other sources, and extract emotional information.

[1250] "Means of generating responses while considering emotional state information" refers to a function that analyzes the user's emotional state and generates more personalized responses based on that information.

[1251] "Login authentication method" refers to the process or function that allows a user to log in to a system using registered credentials (e.g., email address and password) and verify access rights.

[1252] This invention is a system that incorporates a high-performance conversational AI and emotion recognition engine into a multilingual portal, thereby solving the language barrier and lack of information that foreigners residing in Japan face in their daily lives. Specific embodiments of this invention are described in detail below.

[1253] 1. User registration and login

[1254] The user first installs the application and then starts the registration process by pressing the "Register" button upon initial launch. The terminal displays a form prompting the user to enter basic information such as name, email address, password, and native language. Once the user enters this information and presses the "Submit" button, the terminal sends the entered data to the server.

[1255] The server stores the received user information in a database (using MySQL or PostgreSQL as the database). Upon successful registration, the server sends a success message back to the terminal, which the user confirms to complete account registration.

[1256] Afterward, the user logs in using the email address and password they entered during registration. The device sends the login information to the server, which authenticates the user by comparing it with the information stored in the database. If authentication is successful, the user's information is sent to the device, and the user can access the portal.

[1257] 2. Receiving and processing inquiries

[1258] Users access the portal and enter inquiries such as, "Where are the hospitals where English is spoken?" The terminal converts the entered data into text format in real time and sends it to the server. The server analyzes the received data and retrieves native language information from the database based on the user ID.

[1259] Simultaneously, an emotion recognition engine (e.g., IBM Watson or Microsoft Azure Sentiment Analysis) is used to analyze the user's emotional state from their input. The emotional state information is sent to a server, which then sends it along with the analysis results to a generating AI model (e.g., GPT-3 or GPT-4).

[1260] 3. Response generation and reply

[1261] The generative AI model processes information based on the inquiry and generates an appropriate response in the user's native language. In doing so, it considers emotional state information, ensuring that the response is sensitive to the user's feelings.

[1262] For example, if a user enters the question "What medicine is effective for a cold?" and the emotion recognition engine analyzes the user's emotional state as "anxious," the generative AI model will generate a response such as "Paracetamol and Ipuhen are effective for colds. Don't worry, these medicines are reliable." The server sends this response to the user's device, which then displays it to the user.

[1263] Example of a prompt:

[1264] Question from a user: "What medicine is effective for colds?"

[1265] User's emotional state: "Anxious"

[1266] Response generated: "Paracetamol and ibuchen are effective for colds. Don't worry, these medicines are reliable."

[1267] In this way, the system of the present invention solves the communication challenges faced by foreign residents in Japan and provides an environment in which users can easily obtain necessary information in their native language. Furthermore, by using an emotion recognition engine, it provides responses that correspond to the user's emotional state, realizing more personalized support.

[1268] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1269] Step 1:

[1270] The user installs the application and presses the "Register" button upon first launch. The device displays a form prompting the user to enter basic information such as name, email address, password, and native language. After the user enters this information and presses the "Submit" button, the device sends the entered data to the server.

[1271] Input: Name, email address, password, native language

[1272] Output: Basic user information sent to the server

[1273] Step 2:

[1274] The server saves the user information received from the terminal to a database (MySQL, PostgreSQL). If the saving is successful, the server sends a registration success message back to the terminal. The user confirms this message and completes the account registration.

[1275] Input: Basic user information sent from the device.

[1276] Output: User information stored in the database, registration success message

[1277] Step 3:

[1278] Users log in using the email address and password they entered during registration. The device sends the login information to the server, which authenticates the user by comparing it with the information stored in the database. If authentication is successful, the server sends the user's information back to the device, and the user can then access the portal.

[1279] Enter: Email address, password

[1280] Output: Authentication result, access rights to the portal

[1281] Step 4:

[1282] The user accesses the portal and enters an inquiry such as, "Where are the hospitals where English is spoken?" The terminal converts the entered data into text format in real time and sends it to the server.

[1283] Input: Text data of the inquiry.

[1284] Output: Query data sent to the server

[1285] Step 5:

[1286] The server analyzes the received data and retrieves native language information from the database based on the user ID. Simultaneously, it analyzes the user's emotional state from their input using an emotion recognition engine (IBM Watson, Microsoft Azure Sentiment Analysis). The emotional state information is then sent back to the server.

[1287] Input: Inquiry data, User ID

[1288] Output: Native language information, emotional state information

[1289] Step 6:

[1290] The server sends the analysis results and emotional state information to the generating AI models (GPT-3, GPT-4). The generating AI models process the information based on the query and generate an appropriate response in the user's native language. In doing so, they take the emotional state information into consideration and generate a response that is sensitive to the user's emotions.

[1291] Input: Analysis results, emotional state information

[1292] Output: Generated response

[1293] Step 7:

[1294] The server sends the generated response to the terminal. The terminal displays the received response on the screen, presenting it in a way that is easy for the user to understand. The user reviews the displayed information and can enter further questions if necessary.

[1295] Input: Generated response

[1296] Output: Response displayed on the terminal

[1297] (Application Example 2)

[1298] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1299] This invention aims to provide a multilingual portal system to address the language barriers and lack of information that foreigners residing in Japan face in their daily lives. It also aims to provide more personalized support by offering responses that take into account the user's emotional state. In particular, it aims to improve the user experience by providing appropriate support tailored to the user's emotions during the shopping experience within virtual stores.

[1300] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1301] In this invention, the server includes means for the user to register basic information, means for the user to input inquiry content, means for transmitting the input inquiry content to the server, means for the server to analyze the inquiry content and generate a response based on the user's native language, means for transmitting and displaying the generated response to the user's terminal, means for using an emotion recognition engine to analyze the user's emotional state, and means for considering the user's emotional state information when generating the response. This enables personalized shopping support tailored to the user's emotions within a virtual store.

[1302] A "multilingual portal" is an information provision system that supports multiple languages, allowing users to make inquiries in multiple languages.

[1303] "User" refers to anyone who uses the system, and is not limited to, but includes, in particular, foreigners residing in Japan.

[1304] "Basic information" refers to the minimum information necessary to use the system, such as the user's name, email address, password, and native language.

[1305] "Inquiry content" refers to the information or questions that the user wants to know.

[1306] A "server" refers to a computer system that receives and analyzes data from users, generates the necessary responses, and sends them back.

[1307] "Native language" refers to the language the user primarily uses, and the system generates responses based on this language.

[1308] An "emotion recognition engine" refers to a software function that analyzes and identifies a user's emotional state based on their input and statements.

[1309] "Response generation" refers to the process by which a server creates an appropriate response based on the query content, native language information, and sentiment status information.

[1310] A "generative AI model" refers to an algorithm or system that uses artificial intelligence technology to automatically generate responses based on the content of an inquiry.

[1311] A "virtual store" refers to a virtual store that exists on the internet, and is a platform that allows users to browse and purchase products online.

[1312] A "shopping assistant application" refers to an application that allows users to obtain information about products and receive assistance with purchasing within a virtual store.

[1313] "Personalized support" refers to support that provides individualized assistance tailored to each user's needs and emotional state.

[1314] The system of this invention is a multilingual portal and can be implemented particularly as a shopping assistant application within a virtual store. The server, terminal, and user work together. Specific embodiments are described below.

[1315] 1. User registration and login

[1316] The user first installs the application and starts the registration process by pressing the "Register" button upon initial launch. The device displays a form for entering basic information such as name, email address, password, and native language, prompting the user to fill it out. The entered data is sent to the server, which stores it in a database and sends a registration success message back to the device. The user confirms this message and completes account registration. After that, the user logs in using the email address and password entered during registration. The device sends the entered login information to the server, which authenticates the user. If authentication is successful, the user's information is sent to the device, and the user can access the portal.

[1317] 2. Receiving and processing inquiries

[1318] Users ask questions about products and services within a virtual store. For example, they might input an inquiry such as, "I'd like to know more about this camera." The terminal converts the data entered in the input field into text format in real time and sends it to the server. The server analyzes the received inquiry data and retrieves native language information from the database based on the user ID. Simultaneously, an emotion recognition engine analyzes the emotional state from the user's input and sends that information to the server. The server sends the analysis results and emotional state information to a generating AI model to generate an appropriate response.

[1319] 3. Response generation and reply

[1320] The generative AI model processes information based on the inquiry and considers emotional state information when generating an appropriate response in the user's native language. For example, suppose a user asks, "Tell me more about this camera," and the emotion recognition engine analyzes the user's emotional state as "interested." The generative AI model then generates a response such as, "This camera features the latest optical zoom technology and is particularly suitable for landscape photography." The server sends the generated response to the user's device. The device displays the received response on its screen, making it easy for the user to understand. The user can review the displayed response and enter further questions if necessary.

[1321] As a concrete example, we can use the text "I'd like to know more about this camera" as an example of a prompt to ask a question to a generative AI model. This prompt allows the user to appropriately obtain the necessary information.

[1322] Specific examples of hardware and software used include the emotion recognition engine and generative AI models "EmotionRecognizer" and "Hugging Face Transformers." By utilizing these, a system that provides personalized responses based on the user's emotional state can be realized.

[1323] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1324] Step 1:

[1325] User registration and login

[1326] Input: The user enters their name, email address, password, and native language.

[1327] Output: User registration information database saving success message

[1328] Operation: The user installs the application and enters basic information such as name, email address, password, and native language. The device sends the data to the server, which stores it in a database. After saving, a registration success message is sent back to the device. The user confirms this message and completes account registration. The user then enters their email address and password to log in. The device sends the entered information to the server, which performs authentication. If authentication is successful, user information is sent to the device, and the user can access the portal.

[1329] Step 2:

[1330] Enter your inquiry details

[1331] Input: The user enters the question in text format.

[1332] Output: The entered query content is sent from the terminal to the server.

[1333] Operation: The user enters their question within the virtual store. The terminal displays the question in text format in the input field and sends it to the server. Example: "I'd like to know more about this camera."

[1334] Step 3:

[1335] Analysis of query data

[1336] Input: Query data received by the server

[1337] Output: Native language information, emotional state information

[1338] Operation: The server analyzes the input query data and retrieves native language information from the database based on the user ID. Simultaneously, the emotion recognition engine analyzes the user's emotional state from their input and sends that information to the server.

[1339] Step 4:

[1340] Response generation

[1341] Input: Inquiry data, native language information, emotional state information

[1342] Output: Generated response text

[1343] Operation: The server sends the analysis results and emotional state information to a generating AI model to generate an appropriate response. The generating AI model considers the query content and emotional state to generate an appropriate response in the user's native language. Example: "This camera is equipped with the latest optical zoom technology and is especially suitable for landscape photography."

[1344] Step 5:

[1345] Sending and displaying generated responses

[1346] Input: Generated response text

[1347] Output: Response displayed on the user terminal

[1348] Operation: The server sends the generated response to the user's terminal. The terminal displays the received response on the screen to make it easy for the user to understand. The user reviews the displayed response and can enter further questions if necessary.

[1349] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1350] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1351] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1352] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1353] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1354] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1355] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1356] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1357] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1358] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1359] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1360] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1361] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1362] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1363] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1364] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1365] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1366] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1367] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1368] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1369] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[1370] The following is further disclosed regarding the embodiments described above.

[1371] (Claim 1)

[1372] A multilingual portal for use by foreign residents,

[1373] A means for users to register their basic information,

[1374] A means for the user to enter the inquiry details,

[1375] A means of sending the entered query content to the server,

[1376] The server analyzes the query content and generates a response based on the user's native language,

[1377] A means for sending and displaying the generated response on the user's terminal,

[1378] A system that includes this.

[1379] (Claim 2)

[1380] The system according to claim 1, wherein the server uses a generative AI model to generate a response based on the user's native language.

[1381] (Claim 3)

[1382] The system according to claim 1, wherein the portal has a means for authenticating user login, and only authenticated users are able to input inquiry content.

[1383] "Example 1"

[1384] (Claim 1)

[1385] A means for users to register their basic information,

[1386] A means for the user to enter the inquiry details,

[1387] A means of sending the entered query content to the server,

[1388] The server analyzes the query content and obtains the user's native language information from the database.

[1389] A server provides means for generating a response using a generation AI model based on the query content,

[1390] A means for sending and displaying the generated response on the user's terminal,

[1391] A system that includes this.

[1392] (Claim 2)

[1393] The system according to claim 1, wherein the server obtains the user's native language information from a database and generates a response using a generative AI model.

[1394] (Claim 3)

[1395] The system according to claim 1, which allows users to enter inquiry content only after they have completed account registration and performed login authentication.

[1396] "Application Example 1"

[1397] (Claim 1)

[1398] A multilingual portal for use by foreign residents,

[1399] A means for users to register their basic information,

[1400] A means for the user to enter the inquiry details,

[1401] A means of sending the entered query content to the server,

[1402] The server analyzes the query content and generates a response based on the user's native language,

[1403] A means for sending and displaying the generated response on the user's terminal,

[1404] A means of providing product information and store guidance in the native language at physical stores,

[1405] A system that includes this.

[1406] (Claim 2)

[1407] The system according to claim 1, wherein the server uses a generative AI model to generate a response based on the user's native language.

[1408] (Claim 3)

[1409] The system according to claim 1, wherein the portal has a means for authenticating user login, and only authenticated users are able to input inquiry content.

[1410] (Claim 4)

[1411] The system according to claim 1, wherein the generating AI model generates prompt sentences for translating product information.

[1412] "Example 2 of combining an emotion engine"

[1413] (Claim 1)

[1414] A multilingual portal for use by foreign residents,

[1415] A means for users to register their basic information,

[1416] A means for the user to enter the inquiry details,

[1417] A means of sending the entered query content to the server,

[1418] The server analyzes the query content and generates a response based on the user's native language,

[1419] A means for sending and displaying the generated response on the user's terminal,

[1420] A means for acquiring emotional state information using an emotion recognition engine that analyzes the emotional state from the user's input,

[1421] Means for generating a response considering the aforementioned emotional state information,

[1422] A system that includes this.

[1423] (Claim 2)

[1424] The system according to claim 1, wherein the server uses a generative AI model to generate a response based on the user's native language.

[1425] (Claim 3)

[1426] The system according to claim 1, wherein the portal has a means for authenticating user login, and only authenticated users are able to input inquiry content.

[1427] "Application example 2 when combining with an emotional engine"

[1428] (Claim 1)

[1429] A multilingual portal for use by foreign residents,

[1430] A means for users to register their basic information,

[1431] A means for the user to enter the inquiry details,

[1432] A means of sending the entered query content to the server,

[1433] The server analyzes the query content and generates a response based on the user's native language,

[1434] A means for sending and displaying the generated response on the user's terminal,

[1435] A method that utilizes an emotion recognition engine to analyze the user's emotional state,

[1436] A means of considering the user's emotional state information when generating a response,

[1437] A system that includes this.

[1438] (Claim 2)

[1439] The system according to claim 1, wherein the server uses a generative AI model to generate a response based on the user's native language.

[1440] (Claim 3)

[1441] The system according to claim 1, wherein the portal provides a shopping assistant application within a virtual store. [Explanation of Symbols]

[1442] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A multilingual portal for use by foreign residents, A means for users to register their basic information, A means for the user to enter the inquiry details, A means of sending the entered query content to the server, The server analyzes the query content and generates a response based on the user's native language, A means for sending and displaying the generated response on the user's terminal, A system that includes this.

2. The system according to claim 1, wherein the server uses a generative AI model to generate a response based on the user's native language.

3. The system according to claim 1, wherein the portal has a means for authenticating user login, and only authenticated users are able to input inquiry content.

Images