System

Abstract

A system is provided.SOLUTION: A system comprising: means for reading a code including information related to a seat; means for providing an application activated by reading the code; means for allowing a user to select a mode in the application; a generative AI model configured to generate detailed information of batters or pitchers based on the mode selected by the user; a speech synthesis device configured to provide information generated from the generative AI model by voice; means for answering a question input by the user in the application; and means for generating an answer to the question and providing the answer by voice and text.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In traditional baseball games, spectators have limited means of instantly obtaining detailed information about players and the game, making it difficult to understand the game or deepen their interest. Furthermore, beginners, in particular, have difficulty understanding the game's progress and the technical terms used, preventing them from fully enjoying the experience. [Means for solving the problem]

[0005] To solve this problem, the system of the present invention includes the following means: a means for reading a code containing information related to seats, and a means for providing an application that is launched by reading the code; a means for allowing a user to select a mode within the application, and a means for using a generative artificial intelligence model to generate detailed information about the batter or pitcher based on the mode selected by the user; and a system that includes a speech synthesizer that provides information generated by the generative artificial intelligence model as voice, and a means for generating answers to questions entered by the user within the application and providing them as voice and text. This allows spectators to obtain detailed information and commentary in real time, deepening their understanding and enabling even beginners to enjoy the game.

[0006] A "code containing information related to a seat" is a digital code such as a barcode or QR code that contains information assigned to a seat in a baseball stadium, such as the seat number, section, row, etc.

[0007] "Means of reading" refers to the ability of a user's device to scan digital codes such as QR codes and barcodes.

[0008] An "application" is a software program that runs on a user's terminal and provides game commentary and player information based on seating information.

[0009] The "means for the user to select a mode" is an interface within the application that allows the user to select batter mode or pitcher mode.

[0010] A "generative artificial intelligence model" is an AI model used to generate player information and match commentary, and has the ability to generate detailed information from text data.

[0011] A "speech synthesizer" is a device equipped with technology for converting text information into speech, and is used to provide information to users by voice.

[0012] "Means for answering questions entered by a user" refers to a function in an application where a user enters a question as text and an answer to that question is provided.

[0013] The "means for generating an answer" is a function for generating an appropriate answer to a user's question, and generates the answer using a generative artificial intelligence model.

[0014] "Means for providing by voice and text" refers to the ability to provide the generated answer to the user both as a text display and as an audio playback. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] MODE FOR CARRYING OUT THE INVENTION

[0037] This invention relates to a system that provides an interactive voice guide application that provides real-time game commentary and player information while watching a baseball game. Hereinafter, this will be referred to as the "talking AI baseball commentator application."

[0038] 1. Scan the QR code and launch the app

[0039] First, when a user sits down at a baseball stadium, they use their smartphone camera to scan the QR code attached to the seat. The device then interprets the QR code and launches a URL or dedicated application. The device then transmits the seat information obtained from the QR code to the server, which then uses this information to retrieve relevant game data.

[0040] 2. Mode Selection

[0041] When the application starts, the device displays a UI that prompts the user to select either "batter mode" or "pitcher mode." The user selects the mode they are interested in.

[0042] 3. Batter mode processing

[0043] 3.1 Introduction of the next batter

[0044] When the user selects batter mode, the device sends a request for information about the next batter to the server. The server collects basic information about the next batter (such as name, age, alma mater, school achievements, and current season results) and inputs this information into a generative AI model. The generative AI model then generates a detailed introduction based on this information.

[0045] The server then sends the generated introduction back to the terminal, which uses a speech synthesizer to play the information aloud, thereby providing the user with detailed information about the next batter.

[0046] As a specific example, if the user selects batter mode and the next batter is named "Sato Ichiro," the device will provide audio commentary such as, "The next batter is Sato Ichiro. He is 23 years old and graduated from the University of Tokyo. He was active in the university championships during his student days, and his batting average this season is .300."

[0047] 4. Pitcher mode processing

[0048] 4.1 Pitching commentary

[0049] When a user selects pitcher mode, the device sends a request for information about the current pitcher to the server. The server collects basic information about the current pitcher (such as name, pitch type, and speed) and inputs this information into a generative AI model. The generative AI model then generates a detailed commentary based on that information.

[0050] The server then sends the generated commentary back to the device, which uses a speech synthesizer to play the information aloud, thereby providing the user with detailed information about the current pitcher and his pitches.

[0051] As a specific example, if the user selects pitcher mode and the current pitcher is named "Takahashi Jiro," the device will provide audio commentary such as, "The current pitcher is Takahashi Jiro. He throws a straight ball with a speed of 150 km / h."

[0052] 5. User Questions and Answers

[0053] Users can enter questions into a text input field within the app. For example, if they enter a question like, "How many home runs has Ichiro Sato hit?", the device sends the question to the server. The server provides the question to a generative artificial intelligence model, which generates an appropriate answer.

[0054] The server sends the generated answer back to the device, which uses a speech synthesizer to provide the answer aloud and also displays the answer as text.

[0055] As a specific example, in response to the question "How many home runs has Ichiro Sato hit?" the device will provide an audio commentary such as "Ichiro Sato has hit 10 home runs this season."

[0056] This invention allows spectators to receive detailed information about the game and audio commentary from players in real time, deepening their understanding of the game and making it enjoyable even for beginners.

[0057] The processing flow will be explained below.

[0058] Scan the QR code and launch the app

[0059] Step 1:

[0060] The user scans the QR code in front of the seat with their smartphone.

[0061] Step 2:

[0062] The device analyzes the QR code and launches a dedicated application based on the information obtained.

[0063] Step 3:

[0064] The terminal sends the seat information read from the QR code to the server.

[0065] Step 4:

[0066] The server receives the seat information and retrieves the game data associated with the seat.

[0067] Mode Selection

[0068] Step 5:

[0069] The device displays a UI that allows the user to select "batter mode" or "pitcher mode."

[0070] Step 6:

[0071] The user selects either "batter mode" or "pitcher mode" on the UI.

[0072] Batter mode processing

[0073] Step 7:

[0074] The terminal sends a request for information on the next batter to the server.

[0075] Step 8:

[0076] The server obtains basic information about the next batter (such as name, age, alma mater, achievements during school, and current season's results) and provides it to the generative artificial intelligence model.

[0077] Step 9:

[0078] A generative AI model generates a detailed introduction based on basic information.

[0079] Step 10:

[0080] The server returns the generated introduction to the terminal.

[0081] Step 11:

[0082] The device uses a speech synthesizer to play the detailed introduction aloud.

[0083] Pitcher mode processing

[0084] Step 12:

[0085] The terminal sends a request to the server for information on the current pitcher.

[0086] Step 13:

[0087] The server obtains basic information about the current pitcher (such as name, pitch type, and pitch speed) and provides it to the generative artificial intelligence model.

[0088] Step 14:

[0089] A generative AI model generates detailed explanatory text based on basic information.

[0090] Step 15:

[0091] The server returns the generated commentary to the terminal.

[0092] Step 16:

[0093] The device uses a speech synthesizer to play the detailed description aloud.

[0094] User Questions and Answers

[0095] Step 17:

[0096] The user types a question into the app's text input field.

[0097] Step 18:

[0098] The terminal sends a question to the server.

[0099] Step 19:

[0100] The server provides the question to a generative artificial intelligence model, which generates an appropriate answer.

[0101] Step 20:

[0102] The server generates a response and sends it back to the terminal.

[0103] Step 21:

[0104] The device uses a speech synthesizer to play the generated answer aloud and simultaneously displays the answer as text.

[0105] This allows spectators to receive detailed information about the game and player commentary in real time via audio.

[0106] Example 1

[0107] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0108] Traditionally, spectators have had difficulty in instantly learning detailed information about the game and players. Beginner spectators, in particular, have difficulty understanding the flow of the game and the characteristics of the players, limiting their viewing experience. This creates a demand for a system that provides real-time game commentary and player information.

[0109] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0110] In this invention, the server includes a means for transmitting seat information to the server and acquiring game data, a means for providing information generated based on voice input, a means for providing a user interface that displays batter mode or pitcher mode when the application is launched, and a means for automatically launching a URL or dedicated application when the user scans a QR code. This allows spectators to obtain detailed game information and player commentary in real time via voice or text, deepening their understanding of the game. It also allows even beginners to enjoy watching the game.

[0111] 1. "Seat-related information" refers to information about where a particular spectator is sitting, such as the seat location or number when watching a baseball game.

[0112] 2. "Code" means a machine-readable symbol, such as a barcode or QR code, that contains information.

[0113] 3. "Application" means a software program that runs on a smartphone or other device.

[0114] 4. "Mode" means a type of setting or feature within an application that a user can select, such as batter mode or pitcher mode.

[0115] 5. A "generative artificial intelligence model" is an artificial intelligence technology that generates new data or documents based on provided information.

[0116] 6. "Speech synthesis device" means a technical device that converts text data into speech and plays it back.

[0117] 7. "Question" means a question or inquiry entered by a User within the Application.

[0118] 8. An "answer" is information or explanation provided in response to a user's question.

[0119] 9. "Server" means a computer system that processes and provides data over a network.

[0120] 10. "Match Data" means data including detailed information about the match currently being played and player performance information.

[0121] 11. "Voice input" means a means by which a user provides information to a terminal by voice.

[0122] 12. "User interface" refers to the screens and elements that allow a user to operate an application.

[0123] 13. "URL" means an address used to identify a resource on the Internet.

[0124] 14. "Purpose-built Application" means an application designed for a specific purpose or function.

[0125] This invention is a system that provides an interactive voice guide application that provides real-time game commentary and player information while watching a baseball game. Hereafter, we will refer to it as the "talking AI baseball commentator application."

[0126] The system begins when a user uses their smartphone camera to read a QR code containing seat-related information. When the user scans the QR code, the device interprets it and automatically launches the retrieved URL or dedicated application. The seat information obtained from the QR code is sent from the device to the server, which then retrieves game data based on the received seat information.

[0127] When the application starts, the device prompts the user through the user interface to select "batter mode" or "pitcher mode." The user can select the mode they are interested in. If the user selects batter mode, the device sends an information request for the next batter to the server. The server collects the next batter's basic information (name, age, alma mater, achievements during school days, performance this season, etc.) from the database and inputs it into the generative AI model.

[0128] The generative AI model used is, for example, OpenAI's GPT-4. An example of a specific prompt is, "Please provide a detailed description of the next batter." The server retrieves the introduction generated from the AI ​​model and sends it back to the device. The device then uses a speech synthesizer (e.g., Google Text-to-Speech) to convert the introduction into audio and play it back to the user.

[0129] Similarly, when the user selects pitcher mode, the device sends a request for information about the current pitcher to the server. The server collects basic information about the current pitcher (such as name, pitch type, and speed) from the database and inputs it into the generative AI model. An example of a prompt sentence is "Please explain the current pitcher in detail." The generated explanation is sent back from the server to the device, converted into speech by a speech synthesizer, and played back to the user.

[0130] Users can also enter questions into a text input field within the app. For example, if a user enters the question, "How many home runs has Ichiro Sato hit?", the device sends the question to the server. The server inputs the question into a generative AI model and generates an appropriate answer. An example of a specific prompt sentence is, "How many home runs has Ichiro Sato hit?" The generated answer is sent back from the server to the device, where it is converted into speech by a speech synthesizer and also displayed as text.

[0131] This allows spectators to hear detailed information about the match and player commentary in real time, deepening their understanding of the match. Even beginners can enjoy the system. This system can significantly improve the experience of watching a match.

[0132] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0133] Step 1: Scan the QR code

[0134] The user uses the smartphone camera to scan a QR code containing seat-related information. The device then uses the smartphone's QR code library (e.g., ZXing) to analyze the QR code and obtain seat information or an application URL. The input is the QR code image, and the output is the seat information or URL. Specifically, the user points the camera at the QR code and presses the scan button to execute this operation.

[0135] Step 2: Launching the Application

[0136] The device accesses the URL obtained from the QR code and automatically launches the talking AI baseball commentator app. If the obtained URL is an installation link for a dedicated app, the device first installs the app and then launches it. The input is the URL, and the output is the launched application. Specifically, the device's browser accesses the URL and the app installation screen is displayed.

[0137] Step 3: Submit seating information

[0138] The device sends the seating information obtained through analysis to the server. At this time, the device uses the HTTP protocol to send the seating information as a POST request to the server's API endpoint. The input is the seating information, and the output is the data sent to the server. Specifically, a piece of software on the device automatically sends the seating information to the server.

[0139] Step 4: Obtaining match data

[0140] Based on the seat information received, the server retrieves the match data related to that seat from a database (e.g., MySQL). The server uses an SQL query to search the database and extract the relevant match data. The input is the seat information and the output is the match data. Specifically, the server sends the SQL query to the database and the results are returned.

[0141] Step 5: Display the mode selection screen

[0142] The device displays buttons on the application's UI to select "batter mode" or "pitcher mode." These buttons are generated using a UI framework (e.g., Flutter). The input is the launched application, and the output is the display of the user interface. Specifically, the mode selection buttons are displayed on the device screen.

[0143] Step 6: User Mode Selection

[0144] The user taps the button for the mode they are interested in. The tap event causes the device to retain information about the selected mode and proceed to the next step. The input is the user's selection action, and the output is information about the selected mode. A specific operation is when the user presses the "batter mode" or "pitcher mode" button.

[0145] Step 7: Request information about the next batter (in batter mode)

[0146] When the user selects "batter mode," the device requests information about the next batter from the server. This is an API request using the HTTP protocol, with the input being the mode selection information and the output being a request to the server. Specifically, the device automatically sends the request.

[0147] Step 8: Gather batter information

[0148] The server collects the next batter's basic information (e.g., name, age, alma mater, achievements during school, and this season's results) from a database. To do this, it searches the database using an SQL query. The input is a batter information request, and the output is the batter's basic information. Specifically, the server executes the SQL query and retrieves the results.

[0149] Step 9: Generative AI model generates introduction text

[0150] The server inputs the collected batter information into a generative AI model (for example, OpenAI's GPT-4). The generative AI model generates a detailed introduction based on the provided information. A specific prompt is "Please provide a detailed description of the next batter." The input is basic information about the batter, and the output is the generated introduction. In concrete terms, the server supplies the batter information to the generative AI model and generates the introduction.

[0151] Step 10: Send back testimonials and play audio

[0152] The server returns the generated introduction to the device. The device converts the introduction into speech using speech synthesis technology (e.g., Google Text-to-Speech) and plays it back. The input is the generated introduction, and the output is audio data. Specifically, the device inputs the introduction into the speech synthesis engine, and the audio is played back from the speaker.

[0153] Step 11: Pitcher Information Request (Pitcher Mode)

[0154] When the user selects "Pitcher Mode," the device requests information about the current pitcher from the server. This is an API request using the HTTP protocol, with the input being the mode selection information and the output being a request to the server. Specifically, the device automatically sends the request.

[0155] Step 12: Gather pitcher information

[0156] The server collects basic information about the current pitcher (such as name, pitch type, speed, etc.) from a database. To do this, it searches the database using an SQL query. The input is a pitcher information request, and the output is the pitcher's basic information. Specifically, the server executes the SQL query and retrieves the results.

[0157] Step 13: Generating explanatory text using a generative AI model

[0158] The server inputs the collected pitcher information into a generative AI model (for example, OpenAI's GPT-4). The generative AI model generates a detailed explanatory text based on the provided information. A specific prompt is "Please provide a detailed explanation of the current pitcher." The input is basic information about the pitcher, and the output is the generated explanatory text. In concrete terms, the server supplies pitcher information to the generative AI model and generates an explanatory text.

[0159] Step 14: Returning commentary and playing audio

[0160] The server returns the generated explanatory text to the device. The device then uses speech synthesis technology (e.g., Google Text-to-Speech) to convert the explanatory text into audio and plays it back. The input is the generated explanatory text, and the output is audio data. Specifically, the device inputs the explanatory text into a speech synthesis engine, and the audio is played back from the speaker.

[0161] Step 15: User Questions and Answers

[0162] The user enters a question into a text input field within the app. For example, if the user enters the question "How many home runs has Ichiro Sato hit?", the input is the user's question, and the device sends this question to the server. The output is the question data sent to the server. In concrete terms, the user enters a question into the text field and presses the send button.

[0163] Step 16: Input to the question generation AI model

[0164] The server provides the received question to the generative AI model and generates an appropriate answer. A specific prompt sentence is, "How many home runs has Ichiro Sato hit?" The input is the user's question, and the output is the generated answer sentence. In concrete terms, the server inputs the question into the generative AI model and generates an answer.

[0165] Step 17: Return and provide answers

[0166] The server returns the generated answer to the terminal. The terminal uses speech synthesis technology to convert the answer into speech and plays it back. At the same time, it also displays the answer as text. The input is the generated answer, and the output is audio data and text data. In concrete terms, the terminal inputs the answer into the speech synthesis engine, and the audio is played back from the speaker while the text is displayed on the screen.

[0167] (Application example 1)

[0168] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0169] In traditional shopping experiences, consumers often find it difficult to obtain detailed product information instantly in physical stores. This is especially true when selecting the perfect product from the numerous products lined up on the shelves, requiring significant effort from consumers, and it is difficult to obtain detailed information without directly asking a store clerk. This often results in an unsatisfactory shopping experience for consumers.

[0170] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0171] In this invention, the server includes: means for reading a code containing information related to a seat; means for providing an application launched by reading the code; means for a user to select a mode in the application; a generative artificial intelligence model that generates detailed information about an object based on the mode selected by the user; a speech synthesizer that provides the information generated by the generative artificial intelligence model in audio; means for answering a question entered by a user in the application; means for generating an answer to the question and providing it in audio and text; means for the application to read a code placed on a product shelf or the like and provide detailed information about the object; means for generating text based on the generative artificial intelligence model for the detailed information, including the product name, price, ingredient information, user reviews, etc., based on the generative artificial intelligence model; and means for providing an audio commentary of the object using a speech synthesizer based on the generated text. This enables consumers to instantly obtain detailed information about products in a physical store, providing an efficient and satisfying shopping experience.

[0172] A "code" is a symbol, either visual or digital, that contains specific information and that can be read and interpreted by electronic equipment.

[0173] An "application" is a software program designed to provide a specific function or service that a user uses on a smartphone or other device.

[0174] A "user" is a person who operates an application or system and receives specific information or services through operations or inputs.

[0175] A "mode" refers to a specific operating state or function set of an application, which can be selected by the user to change the system's behavior or information presentation.

[0176] A "generative artificial intelligence model" is a type of artificial intelligence that has the ability to automatically generate natural language sentences based on given input data.

[0177] A "voice synthesizer" is a device that converts text data into voice and provides information to a user in voice form.

[0178] "Detailed information" refers to specific and in-depth information about the subject matter, including product name, price, ingredient information, user reviews, etc.

[0179] A "product shelf" refers to the area or facility in a physical store where products are displayed and are set up so that consumers can pick up the products.

[0180] "Subject" refers to the specific goods or services that the application provides information about.

[0181] A "question" is a text input that requests additional information or clarification that the user desires from the application related to the current mode or information.

[0182] An "answer" is text or audio containing information generated by a generative artificial intelligence model based on a user's question.

[0183] This invention relates to a voice interactive guide application that provides consumers with real-time product explanations and promotional information in a physical store. The system is activated when a consumer scans a code placed on a product shelf in the store, and provides detailed product information.

[0184] System Configuration

[0185] Hardware

[0186] Smartphone: Camera, speaker, QR code scanner.

[0187] Product shelf: The shelf where the code is installed.

[0188] software

[0189] QR Code Scanner Library: A library for reading QR codes and obtaining product information.

[0190] HTTP request library: A library for retrieving product information from the server (e.g., Requests for Python).

[0191] Generative AI model: An AI model that generates sentences based on input data (e.g., OpenAI's GPT).

[0192] Speech synthesis API: An API for converting generated text into speech (e.g., Google Text-to-Speech).

[0193] Processing Description

[0194] 1. Scan the QR code

[0195] A user uses their smartphone camera to scan a code on a shelf.

[0196] The terminal analyzes the code it reads and obtains the product ID.

[0197] 2. Obtaining product information

[0198] Based on the product ID, the terminal sends an HTTP request to the server to obtain detailed product information.

[0199] The server collects detailed information such as product name, price, ingredient information, and user reviews and sends it back to the device.

[0200] 3. Utilizing generative artificial intelligence models

[0201] The terminal inputs the acquired product information into a generative artificial intelligence model and generates a detailed product description using prompt sentences.

[0202] For example, the prompt text might look like this:

[0203] "Product name: Organic apples. Characteristics: Grown without pesticides, very sweet. Price: 500 yen. Reviews: Popular for their sweetness and deliciousness."

[0204] The generated product description text is returned to the terminal.

[0205] 4. Speech synthesis and information provision

[0206] The device sends the generated product description text to a speech synthesis API to generate voice data.

[0207] The generated voice is used to provide a voice explanation of the product to the user.

[0208] Specific examples

[0209] For example, if a user scans a QR code for organic apples, the device inputs the following prompt text into the generative AI model: "Product name: Organic apples\nFeatures: Grown without pesticides and very sweet\nPrice: 500 yen\nReviews: Rated for their sweetness and deliciousness." Based on this, the AI ​​generates a detailed product description and then converts it into audio commentary using a speech synthesis API. The user can receive a real-time description on their smartphone, such as, "These organic apples are grown without pesticides and are characterized by their sweetness. The price is 500 yen."

[0210] The system allows consumers to instantly obtain detailed product information in-store, providing an efficient and satisfying shopping experience.

[0211] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0212] Step 1:

[0213] A user uses a smartphone camera to scan a code on a product shelf. The input is the code, and the output is the parsed product ID. Specifically, the user activates the camera and performs an operation to read the code.

[0214] Step 2:

[0215] The terminal analyzes the scanned code and obtains the product ID. The input is the scanned code and the output is the product ID. This includes analyzing the code and extracting string information.

[0216] Step 3:

[0217] The terminal sends an HTTP request to the server based on the product ID to obtain detailed product information. The input is the product ID, and the output is detailed product information such as the product name, price, ingredient information, and user reviews. Specific operations include accessing an API containing detailed product information and obtaining data.

[0218] Step 4:

[0219] The server collects detailed information such as product name, price, ingredient information, and user reviews, and sends it back to the terminal. The input is a request based on the product ID, and the output is detailed product information. The server also organizes the collected data and returns it as an HTTP response.

[0220] Step 5:

[0221] The terminal inputs the acquired product information into a generative AI model and generates a detailed product description using a prompt. The input is the product information, and the output is the generated product description. Specific operations include constructing a prompt and sending it to the AI ​​model to generate a sentence.

[0222] Step 6:

[0223] The device sends the generated product description to a speech synthesis API to generate voice data. The input is the generated product description, and the output is voice data. Specific operations include converting the generated text into voice data.

[0224] Step 7:

[0225] The device uses the generated audio data to provide the user with an audio description of the product. The input is the audio data and the output is the audio description. An operation of playing the audio using the smartphone speaker is included.

[0226] By performing these steps in succession, users can instantly obtain detailed information about products by voice in a physical store.

[0227] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0228] MODE FOR CARRYING OUT THE INVENTION

[0229] This invention relates to a system that combines an emotion engine that recognizes the user's emotions with a voice-activated interactive guide application that provides real-time game commentary and player information during baseball games. Hereinafter, this system will be referred to as the "emotion-recognition AI baseball commentator app."

[0230] 1. Scan the QR code and launch the app

[0231] First, when a user sits down at a baseball stadium, they use their smartphone camera to scan the QR code attached to the seat. The device then interprets the QR code and launches a URL or dedicated application. The device then transmits the seat information obtained from the QR code to the server, which then uses this information to retrieve relevant game data.

[0232] 2. Mode Selection

[0233] When the application starts, the device displays a UI that prompts the user to select either "batter mode" or "pitcher mode." The user selects the mode they are interested in.

[0234] 3. Batter mode processing

[0235] 3.1 Introduction of the next batter

[0236] When the user selects batter mode, the device sends a request for information about the next batter to the server. The server collects basic information about the next batter (such as name, age, alma mater, school achievements, and current season results) and inputs this information into a generative AI model. The generative AI model then generates a detailed introduction based on this information.

[0237] The server then sends the generated introduction back to the device, which uses a speech synthesizer to play the information aloud, and the device also uses an emotion engine to monitor the user's reactions and adjust the tone and content of the introduction according to the user's emotions.

[0238] For example, if the user selects batter mode and the next batter is named "Ichiro Sato," the device will provide audio commentary such as, "The next batter is Ichiro Sato. He is 23 years old and graduated from the University of Tokyo. He played in the university championships as a student. His batting average this season is .300." If the emotion engine recognizes that the user is excited, it can make the tone of the commentary more energetic.

[0239] 4. Pitcher mode processing

[0240] 4.1 Pitching commentary

[0241] When a user selects pitcher mode, the device sends a request for information about the current pitcher to the server. The server collects basic information about the current pitcher (such as name, pitch type, and speed) and inputs this information into a generative AI model. The generative AI model then generates a detailed commentary based on that information.

[0242] The server then sends the generated commentary back to the device, which uses a speech synthesizer to play the information aloud, and the device also uses an emotion engine to monitor the user's reactions and adjust the tone and content of the commentary according to the user's emotions.

[0243] For example, if the user selects pitcher mode and the current pitcher is named "Takahashi Jiro," the device will provide a voice commentary such as, "The current pitcher is Takahashi Jiro. He throws a fastball with a speed of 150 km / h." If the emotion engine determines that the user is calm, the commentary can be more detailed and educational.

[0244] 5. User Questions and Answers

[0245] Users can enter questions into a text input field within the app. For example, if they enter a question like, "How many home runs has Ichiro Sato hit?", the device sends the question to the server. The server provides the question to a generative artificial intelligence model, which generates an appropriate answer.

[0246] The server sends the generated answer back to the device, which uses a speech synthesizer to provide the answer aloud and simultaneously displays the answer as text.The device also uses an emotion engine to monitor the user's reaction and provide the answer in a tone that corresponds to the user's emotion.

[0247] For example, in response to the question, "How many home runs has Ichiro Sato hit?", the device can provide a voice commentary such as, "Ichiro Sato has hit 10 home runs this season." If the emotion engine determines that the user is happy, it can respond by brightening the tone.

[0248] This method allows spectators to obtain detailed information about the game and players in real time, while receiving commentary that is tailored to their own emotions. This is expected to deepen their understanding of the game and increase their enjoyment of watching it.

[0249] The processing flow will be explained below.

[0250] Scan the QR code and launch the app

[0251] Step 1:

[0252] The user scans the QR code in front of the seat with their smartphone.

[0253] Step 2:

[0254] The device analyzes the QR code and launches a dedicated application based on the information obtained.

[0255] Step 3:

[0256] The terminal sends the seat information read from the QR code to the server.

[0257] Step 4:

[0258] The server receives the seat information and retrieves the game data associated with the seat.

[0259] Mode Selection

[0260] Step 5:

[0261] The device displays a UI that allows the user to select "batter mode" or "pitcher mode."

[0262] Step 6:

[0263] The user selects either "batter mode" or "pitcher mode" on the UI.

[0264] Batter mode processing

[0265] Step 7:

[0266] The terminal sends a request for information on the next batter to the server.

[0267] Step 8:

[0268] The server obtains basic information about the next batter (such as name, age, alma mater, achievements during school, and current season's results) and provides it to the generative artificial intelligence model.

[0269] Step 9:

[0270] A generative AI model generates a detailed introduction based on basic information.

[0271] Step 10:

[0272] The server returns the generated introduction to the terminal.

[0273] Step 11:

[0274] The device uses a speech synthesizer to play the detailed introduction aloud.

[0275] Step 12:

[0276] The device uses an emotion engine to monitor the user's reactions and adjust the tone and content of the introduction depending on the user's emotions.

[0277] Pitcher mode processing

[0278] Step 13:

[0279] The terminal sends a request to the server for information on the current pitcher.

[0280] Step 14:

[0281] The server obtains basic information about the current pitcher (such as name, pitch type, and pitch speed) and provides it to the generative artificial intelligence model.

[0282] Step 15:

[0283] A generative AI model generates detailed explanatory text based on basic information.

[0284] Step 16:

[0285] The server returns the generated commentary to the terminal.

[0286] Step 17:

[0287] The device uses a speech synthesizer to play the detailed description aloud.

[0288] Step 18:

[0289] The device uses an emotion engine to monitor the user's reactions and adjust the tone and content of the commentary according to the user's emotions.

[0290] User Questions and Answers

[0291] Step 19:

[0292] The user types a question into the app's text input field.

[0293] Step 20:

[0294] The terminal sends a question to the server.

[0295] Step 21:

[0296] The server provides the question to a generative artificial intelligence model, which generates an appropriate answer.

[0297] Step 22:

[0298] The server generates a response and sends it back to the terminal.

[0299] Step 23:

[0300] The device uses a speech synthesizer to play the generated answer aloud and simultaneously displays the answer as text.

[0301] Step 24:

[0302] The device uses an emotion engine to monitor the user's reactions and provides answers in a tone that corresponds to the user's emotions.

[0303] This allows spectators to receive detailed information about the game and player commentary in real time via audio, and they can also receive the most appropriate commentary based on their own emotions.

[0304] Example 2

[0305] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0306] In modern sports viewing, especially baseball, there is a demand for detailed game commentary and player information in real time. However, conventional systems often provide only one-way information without considering the user's emotions, and improvements are needed to increase spectator satisfaction. In particular, a challenge is to develop technology that dynamically adjusts the tone and content of information provided according to the user's emotions.

[0307] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for reading a code including seat-related information; means for providing an application launched by reading the code; means for a user to select a mode in the application; a generative AI model that generates detailed information about a batter or pitcher based on the mode selected by the user; a voice synthesizer that provides the information generated by the generative AI model as audio; an emotion engine that analyzes the user's facial expressions and tone of voice to recognize emotions; means for adjusting the tone and content of audio commentary based on the analysis results of the emotion engine; means for answering questions entered by the user in the application; and means for generating answers to the questions and providing them in audio and text. This makes it possible to provide personalized real-time game commentary and player information according to the user's emotions.

[0308] A "code containing information related to the seat" is a code containing information such as the seat position and number where the user is seated, and includes QR codes and barcodes.

[0309] "Means for reading the code" refers to a device, such as a QR code reader or barcode scanner, or an application that is used to interpret the information in the code.

[0310] "Means for providing applications" refers to applications that run on smartphones or tablets and have the ability to start automatically by loading code.

[0311] "Means for the user to select the mode" refers to a user interface (UI) that provides the user with the ability to select "batter mode" or "pitcher mode" using on-screen buttons or menus.

[0312] A "generative artificial intelligence model" refers to a generative AI based on natural language processing, for example, which generates appropriate sentences and explanations based on input information.

[0313] A "voice synthesizer" refers to a device or software that converts text data into voice data and plays it back.

[0314] An "emotion engine" refers to a device or software that analyzes a user's facial expressions and tone of voice to recognize their emotional state at that time.

[0315] A "means for answering questions" is a function that generates appropriate answers to questions entered by users and provides them in voice or text, and often uses a generative AI model.

[0316] The "means for adjusting tone and content" provides a function for dynamically changing the tone and amount of detailed information in the generated commentary or introduction based on the analysis results of the emotion engine.

[0317] This invention is implemented as an "emotion-recognition AI baseball commentator app." The following is a detailed explanation of the system, including the hardware and software used to perform each process.

[0318] Hardware and Software

[0319] This system uses the following hardware and software:

[0320] Smartphone: A mobile device held by the user that has a camera to read QR codes, runs applications, inputs voice using a microphone, and outputs voice using a speaker.

[0321] QR Code Reader: QR code analysis software that uses a smartphone camera.

[0322] Server: Retrieves match data, processes specified API endpoints, runs generative AI models, and processes user requests.

[0323] Generative AI model: An artificial intelligence used to generate detailed information and explanatory text (e.g., GPT-3).

[0324] Speech synthesizer: A TTS (Text-to-Speech) engine that converts text data into voice data.

[0325] Emotion engine: Software that analyzes the user's facial expressions and tone of voice in real time.

[0326] System Operation

[0327] Scan the QR code and launch the app

[0328] When a user sits down at a baseball stadium, they use their smartphone camera to scan the QR code installed in front of their seat. The QR code reader analyzes the code and automatically launches a URL or dedicated application based on the results. The device then sends the seat information obtained from the QR code to the server, which then retrieves the relevant game data.

[0329] Mode Selection

[0330] When the application starts, the device displays a UI that prompts the user to select either "batter mode" or "pitcher mode." The user selects the mode they are interested in by tapping a button on the screen.

[0331] Batter mode processing

[0332] When the user selects batter mode, the device sends a request for information about the next batter to the server. The server collects basic information about the next batter from the current batting order and creates a prompt to input to the generative AI model. The generative AI model generates a detailed introduction based on the sent prompt. The generated introduction is sent back to the device by the server. The device uses a speech synthesizer to play the introduction as audio.

[0333] Next, the device uses an emotion engine to analyze the user's facial expressions and tone of voice, monitoring emotions in real time. If the user is excited, the tone of the commentary can be adjusted to be more energetic. For example, if the next batter's name is "Yamada Taro," the prompt would be "Tell me about Yamada Taro. He's 25 years old and graduated from University A. He played in the university championships as a student. His batting average this season is .320." This text would be input into the generative AI model.

[0334] Pitcher mode processing

[0335] When the user selects pitcher mode, the device sends a request for information about the current pitcher to the server. The server collects the current pitcher information and creates a prompt to input to the generative AI model. The generative AI model generates a detailed commentary based on the sent prompt. The generated commentary is sent back to the device by the server. The device uses a speech synthesizer to play the commentary as audio.

[0336] The device then uses an emotion engine to analyze the user's facial expressions and tone of voice to monitor their emotions in real time. If the user is calm, the commentary can be more detailed and educational. For example, if the current pitcher's name is "Jiro Nakamura," the prompt would be "Please describe pitcher Jiro Nakamura. He throws a straight ball with a speed of 145 km / h," and the text would be input into the generative AI model.

[0337] User Questions and Answers

[0338] The user enters a question into a text input field within the app and taps the submit button. The device sends the question to the server, which provides it as a prompt to the generative AI model to generate an appropriate answer. The generated answer is then sent back to the device by the server, and the device uses a speech synthesizer to provide the answer as voice and simultaneously displays the answer as text.

[0339] The device uses an emotion engine to analyze the user's facial expressions and tone of voice, monitoring emotions in real time. Based on the analysis results of the emotion engine, it can provide answers in a tone that corresponds to the user's emotions. For example, if a question such as "Please tell me the schedule for the next game" is entered, the prompt text "Please tell me the schedule for the next game" will be input into the generative AI model.

[0340] In this way, we can provide real-time game commentary and player information based on the user's emotions. This system is expected to make baseball watching more engaging and enjoyable.

[0341] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0342] Step 1:

[0343] A user sits in a seat at a baseball stadium and scans a QR code with a smartphone camera. The input is an image of the QR code, and the output is a URL containing seat information or the launch of a dedicated application. The QR code reader analyzes the image and extracts the seat information.

[0344] Step 2:

[0345] The device sends the seating information obtained from the QR code to the server. The input here is the seating information analyzed from the QR code, and the output is the seating information sent to the server. The device sends this information to the server as an HTTP request and obtains the access key for the match data.

[0346] Step 3:

[0347] The server retrieves data on the currently playing game from the database based on the seating information. The input is the seating information received from the terminal, and the output is the game data. The server searches for related game data based on the seating information and retrieves the necessary information.

[0348] Step 4:

[0349] The device displays a UI on the application's home screen that allows the user to select "batter mode" or "pitcher mode." The input is the game data received from the server, and the output is the mode selection screen displayed on the user interface. The device dynamically renders the UI components.

[0350] Step 5:

[0351] The user taps a button on the screen to select a mode. The input is the user's tap, and the output is the selected mode ("batter mode" or "pitcher mode").

[0352] Step 6:

[0353] When the user selects "batter mode," the terminal sends a request for information about the next batter to the server. The input is the selected "batter mode," and the output is an information request to the server. The terminal sends an HTTP request to the server, requesting information about the next batter.

[0354] Step 7:

[0355] The server collects basic information about the next batter and creates a prompt to input into the generative AI model. The input is the next batter's basic information (e.g., name, age, alma mater, grades), and the output is the prompt to input into the generative AI model. The server generates a detailed introduction based on this.

[0356] Step 8:

[0357] The generative AI model generates a detailed introduction. The input is the prompt sent from the server, and the output is a detailed introduction. The generative AI model uses an NLP algorithm to generate a sentence based on the prompt.

[0358] Step 9:

[0359] The server returns the generated introductory text to the device. The input is the introductory text generated by the generative AI model, and the output is the introductory text returned to the device. The server returns the introductory text as an HTTP response.

[0360] Step 10:

[0361] The device uses a speech synthesizer to play the introduction aloud. The input is the text data of the introduction received from the server, and the output is audio data. The speech synthesizer converts the text data into audio and plays it back.

[0362] Step 11:

[0363] The device uses an emotion engine to monitor the user's reactions. The input is the user's facial expressions and tone of voice, and the output is the analysis results of the emotion engine. The device uses a camera and microphone to analyze the user's emotions in real time.

[0364] Step 12:

[0365] The device adjusts the tone and content of the introduction based on the user's emotions. The input is the analysis result of the emotion engine, and the output is the adjusted voice. Based on the analysis result, the device appropriately changes the tone and specificity of the introduction from the next time onwards.

[0366] Step 13:

[0367] When the user selects "pitcher mode," the terminal sends a request for information about the current pitcher to the server. The input is the selected "pitcher mode," and the output is the information request to the server. The terminal sends an HTTP request to the server, requesting information about the current pitcher.

[0368] Step 14:

[0369] The server collects basic information about the current pitcher and creates a prompt to input to the generative AI model. The input is the basic information about the current pitcher (e.g., name, pitch type, speed), and the output is the prompt to input to the generative AI model. The server generates a detailed explanation based on this.

[0370] Step 15:

[0371] The generative AI model generates detailed explanatory text. The input is the prompt sent from the server, and the output is the detailed explanatory text. The generative AI model uses an NLP algorithm to generate text based on the prompt.

[0372] Step 16:

[0373] The server returns the generated commentary to the device. The input is the commentary generated by the generative AI model, and the output is the commentary returned to the device. The server returns the commentary as an HTTP response.

[0374] Step 17:

[0375] The device uses a speech synthesizer to play the explanatory text aloud. The input is the text data of the explanatory text received from the server, and the output is audio data. The speech synthesizer converts the text data into audio and plays it back.

[0376] Step 18:

[0377] The device uses an emotion engine to monitor the user's reactions. The input is the user's facial expressions and tone of voice, and the output is the analysis results of the emotion engine. The device uses a camera and microphone to analyze the user's emotions in real time.

[0378] Step 19:

[0379] The device adjusts the tone and content of the commentary based on the user's emotions. The input is the analysis result of the emotion engine, and the output is the adjusted voice. Based on the analysis result, the device appropriately changes the tone and specificity of the commentary from the next time onwards.

[0380] Step 20:

[0381] The user enters a question into a text input field in the app and taps the submit button. The input is the question entered by the user, and the output is what the device sends to the server.

[0382] Step 21:

[0383] The terminal sends a question to the server. The input is the question entered by the user, and the output is the question sent to the server.

[0384] Step 22:

[0385] The server provides the generative AI model with a question as a prompt, and it generates an appropriate answer. The input is the prompt sentence of the user question, and the output is the generated answer.

[0386] Step 23:

[0387] The server sends the generated answer back to the device. The input is the answer generated by the generative AI model, and the output is the answer sent back to the device.

[0388] Step 24:

[0389] The terminal uses a speech synthesizer to provide a voice response. The input is the text data of the response received from the server, and the output is voice data. The speech synthesizer converts the text data into voice and plays it back.

[0390] Step 25:

[0391] The device simultaneously displays the answer as text on the UI. The input is the text data of the answer received from the server, and the output is the text displayed on the screen.

[0392] Step 26:

[0393] The device uses an emotion engine to monitor the user's reactions. The input is the user's facial expressions and tone of voice, and the output is the analysis results of the emotion engine. The device uses a camera and microphone to analyze the user's emotions in real time.

[0394] Step 27:

[0395] The device responds with a tone that reflects the user's emotions. The input is the analysis result of the emotion engine, and the output is an adjusted voice. Based on the analysis result, the device appropriately changes the tone and content of future responses.

[0396] (Application example 2)

[0397] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0398] The virtual shopping experience has the problem that product information is provided uniformly and does not respond to the user's individual emotions and interests, which may reduce the user's motivation to purchase and their satisfaction.

[0399] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0400] In this invention, the server includes: means for reading a code containing seat-related information; means for providing an application launched by reading the code; means for a user to select a mode in the application; a generative AI model for generating detailed information about general information based on the mode selected by the user; a speech synthesizer for providing the information generated by the generative AI model in voice; means for recognizing the user's emotion using an emotion engine and adjusting the tone and content of the information based on the recognized emotion; means for answering questions entered by the user in the application; and means for generating answers to the questions and providing them in voice and text. This makes it possible to provide information according to the user's emotional state, thereby improving the satisfaction of the shopping experience in the virtual store.

[0401] A "code containing information related to seats" is a QR code or barcode that stores information related to the location and characteristics of products in a virtual store and user behavior.

[0402] The "means for providing applications" is a mechanism for automatically launching specific programs or applications on a user's device and enabling specific functions or services.

[0403] "Means for user mode selection" means an interface or button within an application that allows a user to select an operating mode according to their interests and needs.

[0404] A "generative artificial intelligence model" is an artificial intelligence model that generates new information and explanatory text based on input data, and utilizes machine learning and natural language processing technologies.

[0405] A "speech synthesizer" refers to a device or software that converts information generated from text data or generative artificial intelligence models into speech format and provides it to users.

[0406] An "emotion engine" is artificial intelligence or software that analyzes a user's facial expressions, tone of voice, etc. to recognize their emotional state.

[0407] "Means for answering questions" refers to a mechanism or interface for generating appropriate answers to questions entered by a user and providing those answers in voice and text format.

[0408] The present invention is a system for providing a voice interactive assistant with emotion recognition capabilities to enhance the shopping experience in a virtual store. The system operates as follows.

[0409] First, while walking through the virtual store, the user scans the QR code of a product with the camera on their smartphone or smart glasses. The server identifies the product information by reading this QR code.

[0410] Next, the server acquires the product information and inputs it into a generative AI model. The generative AI model generates a detailed explanatory text based on the product information. The generated explanatory text is then converted into voice data using a voice synthesizer.

[0411] Additionally, the device uses a camera to capture the user's facial expressions in real time and recognizes their emotions using an emotion engine. The device adjusts the tone and content of the generated commentary depending on the user's emotional state. This process is powered by the dlib and Emotion Recognition libraries.

[0412] When a user asks a question about a product, the device sends the question as text to the server, which uses a generative artificial intelligence model to generate an answer corresponding to the question and provides it to the user in voice and text.

[0413] For example, if a user scans a QR code for a "smartphone case" in a virtual store, the server retrieves the corresponding product information. Based on that information, a generative AI model generates a description such as, "This smartphone case is shock-resistant and available in a variety of colors," and a speech synthesizer converts it into voice data. If the user is excited, the device can provide a description in an energetic tone, such as, "It's on sale now, so now's your chance!"

[0414] Examples of prompts include:

[0415] What product are you looking at right now?

[0416] Could you tell me more about this product?

[0417] Are there any other products you would recommend?

[0418] This system makes it possible to provide information according to the user's emotional state, thereby improving the satisfaction of the shopping experience in the virtual store.

[0419] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0420] Step 1:

[0421] The terminal reads the QR code scanned by the user using the camera on a smartphone or smart glasses. The input is the QR code, and the output is the product identification information obtained from the QR code. Specifically, the terminal analyzes the QR code from the image data captured by the camera and extracts the product identification information.

[0422] Step 2:

[0423] The server retrieves the corresponding product information from the database based on the product identification information obtained from the QR code. The input is the product identification information, and the output is detailed information such as the product name, description, and price. Specifically, it sends a request to the REST API and retrieves the product information as response data.

[0424] Step 3:

[0425] The server inputs product information into a generative artificial intelligence model to generate detailed descriptions. The input is product information, and the output is detailed descriptions. Specifically, the server feeds product information into a machine learning model to generate natural-looking text.

[0426] Step 4:

[0427] The device uses a speech synthesizer to convert the explanatory text into audio data. The input is a detailed explanatory text, and the output is audio data. Specifically, the device converts the text into audio using a text-to-speech engine (e.g., Pyttsx3).

[0428] Step 5:

[0429] The device uses a camera to capture the user's facial expressions in real time and uses an emotion engine to recognize the user's emotions. The input is real-time video data, and the output is the user's emotional state. Specifically, it uses dlib and the Emotion Recognition library to extract facial features and determine the user's emotional state.

[0430] Step 6:

[0431] The device adjusts the tone and content of the generated commentary based on the user's emotional state. The input is the user's emotional state and commentary, and the output is the adjusted commentary. Specifically, the device applies an algorithm that changes the tone and wording of the commentary depending on the results of the emotion engine.

[0432] Step 7:

[0433] The terminal provides the adjusted commentary to the user as audio data. The input is the adjusted commentary, and the output is audio output to the user. Specifically, the terminal uses the speech synthesizer again to convert the adjusted text into audio and play it through the speaker.

[0434] Step 8:

[0435] When a user asks a question about a product, the device sends the question in text format to the server. The input is the user's question, and the output is the question data in text format. Specifically, the device captures the question entered in the text input field within the app and sends it to the server.

[0436] Step 9:

[0437] The server uses a generative artificial intelligence model to generate an answer corresponding to the question and provides the answer to the user in voice and text. The input is a text question, and the output is voice and text data of the answer. Specifically, the server provides the question to a machine learning model, generates an appropriate answer, and converts the answer into voice data using a speech synthesizer.

[0438] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0439] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0440] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0441] [Second embodiment]

[0442] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0443] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0444] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0445] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0446] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0447] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0448] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0449] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0450] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0451] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0452] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0453] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0454] MODE FOR CARRYING OUT THE INVENTION

[0455] This invention relates to a system that provides an interactive voice guide application that provides real-time game commentary and player information while watching a baseball game. Hereinafter, this will be referred to as the "talking AI baseball commentator application."

[0456] 1. Scan the QR code and launch the app

[0457] First, when a user sits down at a baseball stadium, they use their smartphone camera to scan the QR code attached to the seat. The device then interprets the QR code and launches a URL or dedicated application. The device then transmits the seat information obtained from the QR code to the server, which then uses this information to retrieve relevant game data.

[0458] 2. Mode Selection

[0459] When the application starts, the device displays a UI that prompts the user to select either "batter mode" or "pitcher mode." The user selects the mode they are interested in.

[0460] 3. Batter mode processing

[0461] 3.1 Introduction of the next batter

[0462] When the user selects batter mode, the device sends a request for information about the next batter to the server. The server collects basic information about the next batter (such as name, age, alma mater, school achievements, and current season results) and inputs this information into a generative AI model. The generative AI model then generates a detailed introduction based on this information.

[0463] The server then sends the generated introduction back to the terminal, which uses a speech synthesizer to play the information aloud, thereby providing the user with detailed information about the next batter.

[0464] As a specific example, if the user selects batter mode and the next batter is named "Sato Ichiro," the device will provide audio commentary such as, "The next batter is Sato Ichiro. He is 23 years old and graduated from the University of Tokyo. He was active in the university championships during his student days, and his batting average this season is .300."

[0465] 4. Pitcher mode processing

[0466] 4.1 Pitching commentary

[0467] When a user selects pitcher mode, the device sends a request for information about the current pitcher to the server. The server collects basic information about the current pitcher (such as name, pitch type, and speed) and inputs this information into a generative AI model. The generative AI model then generates a detailed commentary based on that information.

[0468] The server then sends the generated commentary back to the device, which uses a speech synthesizer to play the information aloud, thereby providing the user with detailed information about the current pitcher and his pitches.

[0469] As a specific example, if the user selects pitcher mode and the current pitcher is named "Takahashi Jiro," the device will provide audio commentary such as, "The current pitcher is Takahashi Jiro. He throws a straight ball with a speed of 150 km / h."

[0470] 5. User Questions and Answers

[0471] Users can enter questions into a text input field within the app. For example, if they enter a question like, "How many home runs has Ichiro Sato hit?", the device sends the question to the server. The server provides the question to a generative artificial intelligence model, which generates an appropriate answer.

[0472] The server sends the generated answer back to the device, which uses a speech synthesizer to provide the answer aloud and also displays the answer as text.

[0473] As a specific example, in response to the question "How many home runs has Ichiro Sato hit?" the device will provide an audio commentary such as "Ichiro Sato has hit 10 home runs this season."

[0474] This invention allows spectators to receive detailed information about the game and audio commentary from players in real time, deepening their understanding of the game and making it enjoyable even for beginners.

[0475] The processing flow will be explained below.

[0476] Scan the QR code and launch the app

[0477] Step 1:

[0478] The user scans the QR code in front of the seat with their smartphone.

[0479] Step 2:

[0480] The device analyzes the QR code and launches a dedicated application based on the information obtained.

[0481] Step 3:

[0482] The terminal sends the seat information read from the QR code to the server.

[0483] Step 4:

[0484] The server receives the seat information and retrieves the game data associated with the seat.

[0485] Mode Selection

[0486] Step 5:

[0487] The device displays a UI that allows the user to select "batter mode" or "pitcher mode."

[0488] Step 6:

[0489] The user selects either "batter mode" or "pitcher mode" on the UI.

[0490] Batter mode processing

[0491] Step 7:

[0492] The terminal sends a request for information on the next batter to the server.

[0493] Step 8:

[0494] The server obtains basic information about the next batter (such as name, age, alma mater, achievements during school, and current season's results) and provides it to the generative artificial intelligence model.

[0495] Step 9:

[0496] A generative AI model generates a detailed introduction based on basic information.

[0497] Step 10:

[0498] The server returns the generated introduction to the terminal.

[0499] Step 11:

[0500] The device uses a speech synthesizer to play the detailed introduction aloud.

[0501] Pitcher mode processing

[0502] Step 12:

[0503] The terminal sends a request to the server for information on the current pitcher.

[0504] Step 13:

[0505] The server obtains basic information about the current pitcher (such as name, pitch type, and pitch speed) and provides it to the generative artificial intelligence model.

[0506] Step 14:

[0507] A generative AI model generates detailed explanatory text based on basic information.

[0508] Step 15:

[0509] The server returns the generated commentary to the terminal.

[0510] Step 16:

[0511] The device uses a speech synthesizer to play the detailed description aloud.

[0512] User Questions and Answers

[0513] Step 17:

[0514] The user types a question into the app's text input field.

[0515] Step 18:

[0516] The terminal sends a question to the server.

[0517] Step 19:

[0518] The server provides the question to a generative artificial intelligence model, which generates an appropriate answer.

[0519] Step 20:

[0520] The server generates a response and sends it back to the terminal.

[0521] Step 21:

[0522] The device uses a speech synthesizer to play the generated answer aloud and simultaneously displays the answer as text.

[0523] This allows spectators to receive detailed information about the game and player commentary in real time via audio.

[0524] Example 1

[0525] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0526] Traditionally, spectators have had difficulty in instantly learning detailed information about the game and players. Beginner spectators, in particular, have difficulty understanding the flow of the game and the characteristics of the players, limiting their viewing experience. This creates a demand for a system that provides real-time game commentary and player information.

[0527] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0528] In this invention, the server includes a means for transmitting seat information to the server and acquiring game data, a means for providing information generated based on voice input, a means for providing a user interface that displays batter mode or pitcher mode when the application is launched, and a means for automatically launching a URL or dedicated application when the user scans a QR code. This allows spectators to obtain detailed game information and player commentary in real time via voice or text, deepening their understanding of the game. It also allows even beginners to enjoy watching the game.

[0529] 1. "Seat-related information" refers to information about where a particular spectator is sitting, such as the seat location or number when watching a baseball game.

[0530] 2. "Code" means a machine-readable symbol, such as a barcode or QR code, that contains information.

[0531] 3. "Application" means a software program that runs on a smartphone or other device.

[0532] 4. "Mode" means a type of setting or feature within an application that a user can select, such as batter mode or pitcher mode.

[0533] 5. A "generative artificial intelligence model" is an artificial intelligence technology that generates new data or documents based on provided information.

[0534] 6. "Speech synthesis device" means a technical device that converts text data into speech and plays it back.

[0535] 7. "Question" means a question or inquiry entered by a User within the Application.

[0536] 8. An "answer" is information or explanation provided in response to a user's question.

[0537] 9. "Server" means a computer system that processes and provides data over a network.

[0538] 10. "Match Data" means data including detailed information about the match currently being played and player performance information.

[0539] 11. "Voice input" means a means by which a user provides information to a terminal by voice.

[0540] 12. "User interface" refers to the screens and elements that allow a user to operate an application.

[0541] 13. "URL" means an address used to identify a resource on the Internet.

[0542] 14. "Purpose-built Application" means an application designed for a specific purpose or function.

[0543] This invention is a system that provides an interactive voice guide application that provides real-time game commentary and player information while watching a baseball game. Hereafter, we will refer to it as the "talking AI baseball commentator application."

[0544] The system begins when a user uses their smartphone camera to read a QR code containing seat-related information. When the user scans the QR code, the device interprets it and automatically launches the retrieved URL or dedicated application. The seat information obtained from the QR code is sent from the device to the server, which then retrieves game data based on the received seat information.

[0545] When the application starts, the device prompts the user through the user interface to select "batter mode" or "pitcher mode." The user can select the mode they are interested in. If the user selects batter mode, the device sends an information request for the next batter to the server. The server collects the next batter's basic information (name, age, alma mater, achievements during school days, performance this season, etc.) from the database and inputs it into the generative AI model.

[0546] The generative AI model used is, for example, OpenAI's GPT-4. An example of a specific prompt is, "Please provide a detailed description of the next batter." The server retrieves the introduction generated from the AI ​​model and sends it back to the device. The device then uses a speech synthesizer (e.g., Google Text-to-Speech) to convert the introduction into audio and play it back to the user.

[0547] Similarly, when the user selects pitcher mode, the device sends a request for information about the current pitcher to the server. The server collects basic information about the current pitcher (such as name, pitch type, and speed) from the database and inputs it into the generative AI model. An example of a prompt sentence is "Please explain the current pitcher in detail." The generated explanation is sent back from the server to the device, converted into speech by a speech synthesizer, and played back to the user.

[0548] Users can also enter questions into a text input field within the app. For example, if a user enters the question, "How many home runs has Ichiro Sato hit?", the device sends the question to the server. The server inputs the question into a generative AI model and generates an appropriate answer. An example of a specific prompt sentence is, "How many home runs has Ichiro Sato hit?" The generated answer is sent back from the server to the device, where it is converted into speech by a speech synthesizer and also displayed as text.

[0549] This allows spectators to hear detailed information about the match and player commentary in real time, deepening their understanding of the match. Even beginners can enjoy the system. This system can significantly improve the experience of watching a match.

[0550] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0551] Step 1: Scan the QR code

[0552] The user uses the smartphone camera to scan a QR code containing seat-related information. The device then uses the smartphone's QR code library (e.g., ZXing) to analyze the QR code and obtain seat information or an application URL. The input is the QR code image, and the output is the seat information or URL. Specifically, the user points the camera at the QR code and presses the scan button to execute this operation.

[0553] Step 2: Launching the Application

[0554] The device accesses the URL obtained from the QR code and automatically launches the talking AI baseball commentator app. If the obtained URL is an installation link for a dedicated app, the device first installs the app and then launches it. The input is the URL, and the output is the launched application. Specifically, the device's browser accesses the URL and the app installation screen is displayed.

[0555] Step 3: Submit seating information

[0556] The device sends the seating information obtained through analysis to the server. At this time, the device uses the HTTP protocol to send the seating information as a POST request to the server's API endpoint. The input is the seating information, and the output is the data sent to the server. Specifically, a piece of software on the device automatically sends the seating information to the server.

[0557] Step 4: Obtaining match data

[0558] Based on the seat information received, the server retrieves the match data related to that seat from a database (e.g., MySQL). The server uses an SQL query to search the database and extract the relevant match data. The input is the seat information and the output is the match data. Specifically, the server sends the SQL query to the database and the results are returned.

[0559] Step 5: Display the mode selection screen

[0560] The device displays buttons on the application's UI to select "batter mode" or "pitcher mode." These buttons are generated using a UI framework (e.g., Flutter). The input is the launched application, and the output is the display of the user interface. Specifically, the mode selection buttons are displayed on the device screen.

[0561] Step 6: User Mode Selection

[0562] The user taps the button for the mode they are interested in. The tap event causes the device to retain information about the selected mode and proceed to the next step. The input is the user's selection action, and the output is information about the selected mode. A specific operation is when the user presses the "batter mode" or "pitcher mode" button.

[0563] Step 7: Request information about the next batter (in batter mode)

[0564] When the user selects "batter mode," the device requests information about the next batter from the server. This is an API request using the HTTP protocol, with the input being the mode selection information and the output being a request to the server. Specifically, the device automatically sends the request.

[0565] Step 8: Gather batter information

[0566] The server collects the next batter's basic information (e.g., name, age, alma mater, achievements during school, and this season's results) from a database. To do this, it searches the database using an SQL query. The input is a batter information request, and the output is the batter's basic information. Specifically, the server executes the SQL query and retrieves the results.

[0567] Step 9: Generative AI model generates introduction text

[0568] The server inputs the collected batter information into a generative AI model (for example, OpenAI's GPT-4). The generative AI model generates a detailed introduction based on the provided information. A specific prompt is "Please provide a detailed description of the next batter." The input is basic information about the batter, and the output is the generated introduction. In concrete terms, the server supplies the batter information to the generative AI model and generates the introduction.

[0569] Step 10: Send back testimonials and play audio

[0570] The server returns the generated introduction to the device. The device converts the introduction into speech using speech synthesis technology (e.g., Google Text-to-Speech) and plays it back. The input is the generated introduction, and the output is audio data. Specifically, the device inputs the introduction into the speech synthesis engine, and the audio is played back from the speaker.

[0571] Step 11: Pitcher Information Request (Pitcher Mode)

[0572] When the user selects "Pitcher Mode," the device requests information about the current pitcher from the server. This is an API request using the HTTP protocol, with the input being the mode selection information and the output being a request to the server. Specifically, the device automatically sends the request.

[0573] Step 12: Gather pitcher information

[0574] The server collects basic information about the current pitcher (such as name, pitch type, speed, etc.) from a database. To do this, it searches the database using an SQL query. The input is a pitcher information request, and the output is the pitcher's basic information. Specifically, the server executes the SQL query and retrieves the results.

[0575] Step 13: Generating explanatory text using a generative AI model

[0576] The server inputs the collected pitcher information into a generative AI model (for example, OpenAI's GPT-4). The generative AI model generates a detailed explanatory text based on the provided information. A specific prompt is "Please provide a detailed explanation of the current pitcher." The input is basic information about the pitcher, and the output is the generated explanatory text. In concrete terms, the server supplies pitcher information to the generative AI model and generates an explanatory text.

[0577] Step 14: Returning commentary and playing audio

[0578] The server returns the generated explanatory text to the device. The device then uses speech synthesis technology (e.g., Google Text-to-Speech) to convert the explanatory text into audio and plays it back. The input is the generated explanatory text, and the output is audio data. Specifically, the device inputs the explanatory text into a speech synthesis engine, and the audio is played back from the speaker.

[0579] Step 15: User Questions and Answers

[0580] The user enters a question into a text input field within the app. For example, if the user enters the question "How many home runs has Ichiro Sato hit?", the input is the user's question, and the device sends this question to the server. The output is the question data sent to the server. In concrete terms, the user enters a question into the text field and presses the send button.

[0581] Step 16: Input to the question generation AI model

[0582] The server provides the received question to the generative AI model and generates an appropriate answer. A specific prompt sentence is, "How many home runs has Ichiro Sato hit?" The input is the user's question, and the output is the generated answer sentence. In concrete terms, the server inputs the question into the generative AI model and generates an answer.

[0583] Step 17: Return and provide answers

[0584] The server returns the generated answer to the terminal. The terminal uses speech synthesis technology to convert the answer into speech and plays it back. At the same time, it also displays the answer as text. The input is the generated answer, and the output is audio data and text data. In concrete terms, the terminal inputs the answer into the speech synthesis engine, and the audio is played back from the speaker while the text is displayed on the screen.

[0585] (Application example 1)

[0586] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0587] In traditional shopping experiences, consumers often find it difficult to obtain detailed product information instantly in physical stores. This is especially true when selecting the perfect product from the numerous products lined up on the shelves, requiring significant effort from consumers, and it is difficult to obtain detailed information without directly asking a store clerk. This often results in an unsatisfactory shopping experience for consumers.

[0588] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0589] In this invention, the server includes: means for reading a code containing information related to a seat; means for providing an application launched by reading the code; means for a user to select a mode in the application; a generative artificial intelligence model that generates detailed information about an object based on the mode selected by the user; a speech synthesizer that provides the information generated by the generative artificial intelligence model in audio; means for answering a question entered by a user in the application; means for generating an answer to the question and providing it in audio and text; means for the application to read a code placed on a product shelf or the like and provide detailed information about the object; means for generating text based on the generative artificial intelligence model for the detailed information, including the product name, price, ingredient information, user reviews, etc., based on the generative artificial intelligence model; and means for providing an audio commentary of the object using a speech synthesizer based on the generated text. This enables consumers to instantly obtain detailed information about products in a physical store, providing an efficient and satisfying shopping experience.

[0590] A "code" is a symbol, either visual or digital, that contains specific information and that can be read and interpreted by electronic equipment.

[0591] An "application" is a software program designed to provide a specific function or service that a user uses on a smartphone or other device.

[0592] A "user" is a person who operates an application or system and receives specific information or services through operations or inputs.

[0593] A "mode" refers to a specific operating state or function set of an application, which can be selected by the user to change the system's behavior or information presentation.

[0594] A "generative artificial intelligence model" is a type of artificial intelligence that has the ability to automatically generate natural language sentences based on given input data.

[0595] A "voice synthesizer" is a device that converts text data into voice and provides information to a user in voice form.

[0596] "Detailed information" refers to specific and in-depth information about the subject matter, including product name, price, ingredient information, user reviews, etc.

[0597] A "product shelf" refers to the area or facility in a physical store where products are displayed and are set up so that consumers can pick up the products.

[0598] "Subject" refers to the specific goods or services that the application provides information about.

[0599] A "question" is a text input that requests additional information or clarification that the user desires from the application related to the current mode or information.

[0600] An "answer" is text or audio containing information generated by a generative artificial intelligence model based on a user's question.

[0601] This invention relates to a voice interactive guide application that provides consumers with real-time product explanations and promotional information in a physical store. The system is activated when a consumer scans a code placed on a product shelf in the store, and provides detailed product information.

[0602] System Configuration

[0603] Hardware

[0604] Smartphone: Camera, speaker, QR code scanner.

[0605] Product shelf: The shelf where the code is installed.

[0606] software

[0607] QR Code Scanner Library: A library for reading QR codes and obtaining product information.

[0608] HTTP request library: A library for retrieving product information from the server (e.g., Requests for Python).

[0609] Generative AI model: An AI model that generates sentences based on input data (e.g., OpenAI's GPT).

[0610] Speech synthesis API: An API for converting generated text into speech (e.g., Google Text-to-Speech).

[0611] Processing Description

[0612] 1. Scan the QR code

[0613] A user uses their smartphone camera to scan a code on a shelf.

[0614] The terminal analyzes the code it reads and obtains the product ID.

[0615] 2. Obtaining product information

[0616] Based on the product ID, the terminal sends an HTTP request to the server to obtain detailed product information.

[0617] The server collects detailed information such as product name, price, ingredient information, and user reviews and sends it back to the device.

[0618] 3. Utilizing generative artificial intelligence models

[0619] The terminal inputs the acquired product information into a generative artificial intelligence model and generates a detailed product description using prompt sentences.

[0620] For example, the prompt text might look like this:

[0621] "Product name: Organic apples. Characteristics: Grown without pesticides, very sweet. Price: 500 yen. Reviews: Popular for their sweetness and deliciousness."

[0622] The generated product description text is returned to the terminal.

[0623] 4. Speech synthesis and information provision

[0624] The device sends the generated product description text to a speech synthesis API to generate voice data.

[0625] The generated voice is used to provide a voice explanation of the product to the user.

[0626] Specific examples

[0627] For example, if a user scans a QR code for organic apples, the device inputs the following prompt text into the generative AI model: "Product name: Organic apples\nFeatures: Grown without pesticides and very sweet\nPrice: 500 yen\nReviews: Rated for their sweetness and deliciousness." Based on this, the AI ​​generates a detailed product description and then converts it into audio commentary using a speech synthesis API. The user can receive a real-time description on their smartphone, such as, "These organic apples are grown without pesticides and are characterized by their sweetness. The price is 500 yen."

[0628] The system allows consumers to instantly obtain detailed product information in-store, providing an efficient and satisfying shopping experience.

[0629] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0630] Step 1:

[0631] A user uses a smartphone camera to scan a code on a product shelf. The input is the code, and the output is the parsed product ID. Specifically, the user activates the camera and performs an operation to read the code.

[0632] Step 2:

[0633] The terminal analyzes the scanned code and obtains the product ID. The input is the scanned code and the output is the product ID. This includes analyzing the code and extracting string information.

[0634] Step 3:

[0635] The terminal sends an HTTP request to the server based on the product ID to obtain detailed product information. The input is the product ID, and the output is detailed product information such as the product name, price, ingredient information, and user reviews. Specific operations include accessing an API containing detailed product information and obtaining data.

[0636] Step 4:

[0637] The server collects detailed information such as product name, price, ingredient information, and user reviews, and sends it back to the terminal. The input is a request based on the product ID, and the output is detailed product information. The server also organizes the collected data and returns it as an HTTP response.

[0638] Step 5:

[0639] The terminal inputs the acquired product information into a generative AI model and generates a detailed product description using a prompt. The input is the product information, and the output is the generated product description. Specific operations include constructing a prompt and sending it to the AI ​​model to generate a sentence.

[0640] Step 6:

[0641] The device sends the generated product description to a speech synthesis API to generate voice data. The input is the generated product description, and the output is voice data. Specific operations include converting the generated text into voice data.

[0642] Step 7:

[0643] The device uses the generated audio data to provide the user with an audio description of the product. The input is the audio data and the output is the audio description. An operation of playing the audio using the smartphone speaker is included.

[0644] By performing these steps in succession, users can instantly obtain detailed information about products by voice in a physical store.

[0645] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0646] MODE FOR CARRYING OUT THE INVENTION

[0647] This invention relates to a system that combines an emotion engine that recognizes the user's emotions with a voice-activated interactive guide application that provides real-time game commentary and player information during baseball games. Hereinafter, this system will be referred to as the "emotion-recognition AI baseball commentator app."

[0648] 1. Scan the QR code and launch the app

[0649] First, when a user sits down at a baseball stadium, they use their smartphone camera to scan the QR code attached to the seat. The device then interprets the QR code and launches a URL or dedicated application. The device then transmits the seat information obtained from the QR code to the server, which then uses this information to retrieve relevant game data.

[0650] 2. Mode Selection

[0651] When the application starts, the device displays a UI that prompts the user to select either "batter mode" or "pitcher mode." The user selects the mode they are interested in.

[0652] 3. Batter mode processing

[0653] 3.1 Introduction of the next batter

[0654] When the user selects batter mode, the device sends a request for information about the next batter to the server. The server collects basic information about the next batter (such as name, age, alma mater, school achievements, and current season results) and inputs this information into a generative AI model. The generative AI model then generates a detailed introduction based on this information.

[0655] The server then sends the generated introduction back to the device, which uses a speech synthesizer to play the information aloud, and the device also uses an emotion engine to monitor the user's reactions and adjust the tone and content of the introduction according to the user's emotions.

[0656] For example, if the user selects batter mode and the next batter is named "Ichiro Sato," the device will provide audio commentary such as, "The next batter is Ichiro Sato. He is 23 years old and graduated from the University of Tokyo. He played in the university championships as a student. His batting average this season is .300." If the emotion engine recognizes that the user is excited, it can make the tone of the commentary more energetic.

[0657] 4. Pitcher mode processing

[0658] 4.1 Pitching commentary

[0659] When a user selects pitcher mode, the device sends a request for information about the current pitcher to the server. The server collects basic information about the current pitcher (such as name, pitch type, and speed) and inputs this information into a generative AI model. The generative AI model then generates a detailed commentary based on that information.

[0660] The server then sends the generated commentary back to the device, which uses a speech synthesizer to play the information aloud, and the device also uses an emotion engine to monitor the user's reactions and adjust the tone and content of the commentary according to the user's emotions.

[0661] For example, if the user selects pitcher mode and the current pitcher is named "Takahashi Jiro," the device will provide a voice commentary such as, "The current pitcher is Takahashi Jiro. He throws a fastball with a speed of 150 km / h." If the emotion engine determines that the user is calm, the commentary can be more detailed and educational.

[0662] 5. User Questions and Answers

[0663] Users can enter questions into a text input field within the app. For example, if they enter a question like, "How many home runs has Ichiro Sato hit?", the device sends the question to the server. The server provides the question to a generative artificial intelligence model, which generates an appropriate answer.

[0664] The server sends the generated answer back to the device, which uses a speech synthesizer to provide the answer aloud and simultaneously displays the answer as text.The device also uses an emotion engine to monitor the user's reaction and provide the answer in a tone that corresponds to the user's emotion.

[0665] For example, in response to the question, "How many home runs has Ichiro Sato hit?", the device can provide a voice commentary such as, "Ichiro Sato has hit 10 home runs this season." If the emotion engine determines that the user is happy, it can respond by brightening the tone.

[0666] This method allows spectators to obtain detailed information about the game and players in real time, while receiving commentary that is tailored to their own emotions. This is expected to deepen their understanding of the game and increase their enjoyment of watching it.

[0667] The processing flow will be explained below.

[0668] Scan the QR code and launch the app

[0669] Step 1:

[0670] The user scans the QR code in front of the seat with their smartphone.

[0671] Step 2:

[0672] The device analyzes the QR code and launches a dedicated application based on the information obtained.

[0673] Step 3:

[0674] The terminal sends the seat information read from the QR code to the server.

[0675] Step 4:

[0676] The server receives the seat information and retrieves the game data associated with the seat.

[0677] Mode Selection

[0678] Step 5:

[0679] The device displays a UI that allows the user to select "batter mode" or "pitcher mode."

[0680] Step 6:

[0681] The user selects either "batter mode" or "pitcher mode" on the UI.

[0682] Batter mode processing

[0683] Step 7:

[0684] The terminal sends a request for information on the next batter to the server.

[0685] Step 8:

[0686] The server obtains basic information about the next batter (such as name, age, alma mater, achievements during school, and current season's results) and provides it to the generative artificial intelligence model.

[0687] Step 9:

[0688] A generative AI model generates a detailed introduction based on basic information.

[0689] Step 10:

[0690] The server returns the generated introduction to the terminal.

[0691] Step 11:

[0692] The device uses a speech synthesizer to play the detailed introduction aloud.

[0693] Step 12:

[0694] The device uses an emotion engine to monitor the user's reactions and adjust the tone and content of the introduction depending on the user's emotions.

[0695] Pitcher mode processing

[0696] Step 13:

[0697] The terminal sends a request to the server for information on the current pitcher.

[0698] Step 14:

[0699] The server obtains basic information about the current pitcher (such as name, pitch type, and pitch speed) and provides it to the generative artificial intelligence model.

[0700] Step 15:

[0701] A generative AI model generates detailed explanatory text based on basic information.

[0702] Step 16:

[0703] The server returns the generated commentary to the terminal.

[0704] Step 17:

[0705] The device uses a speech synthesizer to play the detailed description aloud.

[0706] Step 18:

[0707] The device uses an emotion engine to monitor the user's reactions and adjust the tone and content of the commentary according to the user's emotions.

[0708] User Questions and Answers

[0709] Step 19:

[0710] The user types a question into the app's text input field.

[0711] Step 20:

[0712] The terminal sends a question to the server.

[0713] Step 21:

[0714] The server provides the question to a generative artificial intelligence model, which generates an appropriate answer.

[0715] Step 22:

[0716] The server generates a response and sends it back to the terminal.

[0717] Step 23:

[0718] The device uses a speech synthesizer to play the generated answer aloud and simultaneously displays the answer as text.

[0719] Step 24:

[0720] The device uses an emotion engine to monitor the user's reactions and provides answers in a tone that corresponds to the user's emotions.

[0721] This allows spectators to receive detailed information about the game and player commentary in real time via audio, and they can also receive the most appropriate commentary based on their own emotions.

[0722] Example 2

[0723] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0724] In modern sports viewing, especially baseball, there is a demand for detailed game commentary and player information in real time. However, conventional systems often provide only one-way information without considering the user's emotions, and improvements are needed to increase spectator satisfaction. In particular, a challenge is to develop technology that dynamically adjusts the tone and content of information provided according to the user's emotions.

[0725] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for reading a code including seat-related information; means for providing an application launched by reading the code; means for a user to select a mode in the application; a generative AI model that generates detailed information about a batter or pitcher based on the mode selected by the user; a voice synthesizer that provides the information generated by the generative AI model as audio; an emotion engine that analyzes the user's facial expressions and tone of voice to recognize emotions; means for adjusting the tone and content of audio commentary based on the analysis results of the emotion engine; means for answering questions entered by the user in the application; and means for generating answers to the questions and providing them in audio and text. This makes it possible to provide personalized real-time game commentary and player information according to the user's emotions.

[0726] A "code containing information related to the seat" is a code containing information such as the seat position and number where the user is seated, and includes QR codes and barcodes.

[0727] "Means for reading the code" refers to a device, such as a QR code reader or barcode scanner, or an application that is used to interpret the information in the code.

[0728] "Means for providing applications" refers to applications that run on smartphones or tablets and have the ability to start automatically by loading code.

[0729] "Means for the user to select the mode" refers to a user interface (UI) that provides the user with the ability to select "batter mode" or "pitcher mode" using on-screen buttons or menus.

[0730] A "generative artificial intelligence model" refers to a generative AI based on natural language processing, for example, which generates appropriate sentences and explanations based on input information.

[0731] A "voice synthesizer" refers to a device or software that converts text data into voice data and plays it back.

[0732] An "emotion engine" refers to a device or software that analyzes a user's facial expressions and tone of voice to recognize their emotional state at that time.

[0733] A "means for answering questions" is a function that generates appropriate answers to questions entered by users and provides them in voice or text, and often uses a generative AI model.

[0734] The "means for adjusting tone and content" provides a function for dynamically changing the tone and amount of detailed information in the generated commentary or introduction based on the analysis results of the emotion engine.

[0735] This invention is implemented as an "emotion-recognition AI baseball commentator app." The following is a detailed explanation of the system, including the hardware and software used to perform each process.

[0736] Hardware and Software

[0737] This system uses the following hardware and software:

[0738] Smartphone: A mobile device held by the user that has a camera to read QR codes, runs applications, inputs voice using a microphone, and outputs voice using a speaker.

[0739] QR Code Reader: QR code analysis software that uses a smartphone camera.

[0740] Server: Retrieves match data, processes specified API endpoints, runs generative AI models, and processes user requests.

[0741] Generative AI model: An artificial intelligence used to generate detailed information and explanatory text (e.g., GPT-3).

[0742] Speech synthesizer: A TTS (Text-to-Speech) engine that converts text data into voice data.

[0743] Emotion engine: Software that analyzes the user's facial expressions and tone of voice in real time.

[0744] System Operation

[0745] Scan the QR code and launch the app

[0746] When a user sits down at a baseball stadium, they use their smartphone camera to scan the QR code installed in front of their seat. The QR code reader analyzes the code and automatically launches a URL or dedicated application based on the results. The device then sends the seat information obtained from the QR code to the server, which then retrieves the relevant game data.

[0747] Mode Selection

[0748] When the application starts, the device displays a UI that prompts the user to select either "batter mode" or "pitcher mode." The user selects the mode they are interested in by tapping a button on the screen.

[0749] Batter mode processing

[0750] When the user selects batter mode, the device sends a request for information about the next batter to the server. The server collects basic information about the next batter from the current batting order and creates a prompt to input to the generative AI model. The generative AI model generates a detailed introduction based on the sent prompt. The generated introduction is sent back to the device by the server. The device uses a speech synthesizer to play the introduction as audio.

[0751] Next, the device uses an emotion engine to analyze the user's facial expressions and tone of voice, monitoring emotions in real time. If the user is excited, the tone of the commentary can be adjusted to be more energetic. For example, if the next batter's name is "Yamada Taro," the prompt would be "Tell me about Yamada Taro. He's 25 years old and graduated from University A. He played in the university championships as a student. His batting average this season is .320." This text would be input into the generative AI model.

[0752] Pitcher mode processing

[0753] When the user selects pitcher mode, the device sends a request for information about the current pitcher to the server. The server collects the current pitcher information and creates a prompt to input to the generative AI model. The generative AI model generates a detailed commentary based on the sent prompt. The generated commentary is sent back to the device by the server. The device uses a speech synthesizer to play the commentary as audio.

[0754] The device then uses an emotion engine to analyze the user's facial expressions and tone of voice to monitor their emotions in real time. If the user is calm, the commentary can be more detailed and educational. For example, if the current pitcher's name is "Jiro Nakamura," the prompt would be "Please describe pitcher Jiro Nakamura. He throws a straight ball with a speed of 145 km / h," and the text would be input into the generative AI model.

[0755] User Questions and Answers

[0756] The user enters a question into a text input field within the app and taps the submit button. The device sends the question to the server, which provides it as a prompt to the generative AI model to generate an appropriate answer. The generated answer is then sent back to the device by the server, and the device uses a speech synthesizer to provide the answer as voice and simultaneously displays the answer as text.

[0757] The device uses an emotion engine to analyze the user's facial expressions and tone of voice, monitoring emotions in real time. Based on the analysis results of the emotion engine, it can provide answers in a tone that corresponds to the user's emotions. For example, if a question such as "Please tell me the schedule for the next game" is entered, the prompt text "Please tell me the schedule for the next game" will be input into the generative AI model.

[0758] In this way, we can provide real-time game commentary and player information based on the user's emotions. This system is expected to make baseball watching more engaging and enjoyable.

[0759] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0760] Step 1:

[0761] A user sits in a seat at a baseball stadium and scans a QR code with a smartphone camera. The input is an image of the QR code, and the output is a URL containing seat information or the launch of a dedicated application. The QR code reader analyzes the image and extracts the seat information.

[0762] Step 2:

[0763] The device sends the seating information obtained from the QR code to the server. The input here is the seating information analyzed from the QR code, and the output is the seating information sent to the server. The device sends this information to the server as an HTTP request and obtains the access key for the match data.

[0764] Step 3:

[0765] The server retrieves data on the currently playing game from the database based on the seating information. The input is the seating information received from the terminal, and the output is the game data. The server searches for related game data based on the seating information and retrieves the necessary information.

[0766] Step 4:

[0767] The device displays a UI on the application's home screen that allows the user to select "batter mode" or "pitcher mode." The input is the game data received from the server, and the output is the mode selection screen displayed on the user interface. The device dynamically renders the UI components.

[0768] Step 5:

[0769] The user taps a button on the screen to select a mode. The input is the user's tap, and the output is the selected mode ("batter mode" or "pitcher mode").

[0770] Step 6:

[0771] When the user selects "batter mode," the terminal sends a request for information about the next batter to the server. The input is the selected "batter mode," and the output is an information request to the server. The terminal sends an HTTP request to the server, requesting information about the next batter.

[0772] Step 7:

[0773] The server collects basic information about the next batter and creates a prompt to input into the generative AI model. The input is the next batter's basic information (e.g., name, age, alma mater, grades), and the output is the prompt to input into the generative AI model. The server generates a detailed introduction based on this.

[0774] Step 8:

[0775] The generative AI model generates a detailed introduction. The input is the prompt sent from the server, and the output is a detailed introduction. The generative AI model uses an NLP algorithm to generate a sentence based on the prompt.

[0776] Step 9:

[0777] The server returns the generated introductory text to the device. The input is the introductory text generated by the generative AI model, and the output is the introductory text returned to the device. The server returns the introductory text as an HTTP response.

[0778] Step 10:

[0779] The device uses a speech synthesizer to play the introduction aloud. The input is the text data of the introduction received from the server, and the output is audio data. The speech synthesizer converts the text data into audio and plays it back.

[0780] Step 11:

[0781] The device uses an emotion engine to monitor the user's reactions. The input is the user's facial expressions and tone of voice, and the output is the analysis results of the emotion engine. The device uses a camera and microphone to analyze the user's emotions in real time.

[0782] Step 12:

[0783] The device adjusts the tone and content of the introduction based on the user's emotions. The input is the analysis result of the emotion engine, and the output is the adjusted voice. Based on the analysis result, the device appropriately changes the tone and specificity of the introduction from the next time onwards.

[0784] Step 13:

[0785] When the user selects "pitcher mode," the terminal sends a request for information about the current pitcher to the server. The input is the selected "pitcher mode," and the output is the information request to the server. The terminal sends an HTTP request to the server, requesting information about the current pitcher.

[0786] Step 14:

[0787] The server collects basic information about the current pitcher and creates a prompt to input to the generative AI model. The input is the basic information about the current pitcher (e.g., name, pitch type, speed), and the output is the prompt to input to the generative AI model. The server generates a detailed explanation based on this.

[0788] Step 15:

[0789] The generative AI model generates detailed explanatory text. The input is the prompt sent from the server, and the output is the detailed explanatory text. The generative AI model uses an NLP algorithm to generate text based on the prompt.

[0790] Step 16:

[0791] The server returns the generated commentary to the device. The input is the commentary generated by the generative AI model, and the output is the commentary returned to the device. The server returns the commentary as an HTTP response.

[0792] Step 17:

[0793] The device uses a speech synthesizer to play the explanatory text aloud. The input is the text data of the explanatory text received from the server, and the output is audio data. The speech synthesizer converts the text data into audio and plays it back.

[0794] Step 18:

[0795] The device uses an emotion engine to monitor the user's reactions. The input is the user's facial expressions and tone of voice, and the output is the analysis results of the emotion engine. The device uses a camera and microphone to analyze the user's emotions in real time.

[0796] Step 19:

[0797] The device adjusts the tone and content of the commentary based on the user's emotions. The input is the analysis result of the emotion engine, and the output is the adjusted voice. Based on the analysis result, the device appropriately changes the tone and specificity of the commentary from the next time onwards.

[0798] Step 20:

[0799] The user enters a question into a text input field in the app and taps the submit button. The input is the question entered by the user, and the output is what the device sends to the server.

[0800] Step 21:

[0801] The terminal sends a question to the server. The input is the question entered by the user, and the output is the question sent to the server.

[0802] Step 22:

[0803] The server provides the generative AI model with a question as a prompt, and it generates an appropriate answer. The input is the prompt sentence of the user question, and the output is the generated answer.

[0804] Step 23:

[0805] The server sends the generated answer back to the device. The input is the answer generated by the generative AI model, and the output is the answer sent back to the device.

[0806] Step 24:

[0807] The terminal uses a speech synthesizer to provide a voice response. The input is the text data of the response received from the server, and the output is voice data. The speech synthesizer converts the text data into voice and plays it back.

[0808] Step 25:

[0809] The device simultaneously displays the answer as text on the UI. The input is the text data of the answer received from the server, and the output is the text displayed on the screen.

[0810] Step 26:

[0811] The device uses an emotion engine to monitor the user's reactions. The input is the user's facial expressions and tone of voice, and the output is the analysis results of the emotion engine. The device uses a camera and microphone to analyze the user's emotions in real time.

[0812] Step 27:

[0813] The device responds with a tone that reflects the user's emotions. The input is the analysis result of the emotion engine, and the output is an adjusted voice. Based on the analysis result, the device appropriately changes the tone and content of future responses.

[0814] (Application example 2)

[0815] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0816] The virtual shopping experience has the problem that product information is provided uniformly and does not respond to the user's individual emotions and interests, which may reduce the user's motivation to purchase and their satisfaction.

[0817] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0818] In this invention, the server includes: means for reading a code containing seat-related information; means for providing an application launched by reading the code; means for a user to select a mode in the application; a generative AI model for generating detailed information about general information based on the mode selected by the user; a speech synthesizer for providing the information generated by the generative AI model in voice; means for recognizing the user's emotion using an emotion engine and adjusting the tone and content of the information based on the recognized emotion; means for answering questions entered by the user in the application; and means for generating answers to the questions and providing them in voice and text. This makes it possible to provide information according to the user's emotional state, thereby improving the satisfaction of the shopping experience in the virtual store.

[0819] A "code containing information related to seats" is a QR code or barcode that stores information related to the location and characteristics of products in a virtual store and user behavior.

[0820] The "means for providing applications" is a mechanism for automatically launching specific programs or applications on a user's device and enabling specific functions or services.

[0821] "Means for user mode selection" means an interface or button within an application that allows a user to select an operating mode according to their interests and needs.

[0822] A "generative artificial intelligence model" is an artificial intelligence model that generates new information and explanatory text based on input data, and utilizes machine learning and natural language processing technologies.

[0823] A "speech synthesizer" refers to a device or software that converts information generated from text data or generative artificial intelligence models into speech format and provides it to users.

[0824] An "emotion engine" is artificial intelligence or software that analyzes a user's facial expressions, tone of voice, etc. to recognize their emotional state.

[0825] "Means for answering questions" refers to a mechanism or interface for generating appropriate answers to questions entered by a user and providing those answers in voice and text format.

[0826] The present invention is a system for providing a voice interactive assistant with emotion recognition capabilities to enhance the shopping experience in a virtual store. The system operates as follows.

[0827] First, while walking through the virtual store, the user scans the QR code of a product with the camera on their smartphone or smart glasses. The server identifies the product information by reading this QR code.

[0828] Next, the server acquires the product information and inputs it into a generative AI model. The generative AI model generates a detailed explanatory text based on the product information. The generated explanatory text is then converted into voice data using a voice synthesizer.

[0829] Additionally, the device uses a camera to capture the user's facial expressions in real time and recognizes their emotions using an emotion engine. The device adjusts the tone and content of the generated commentary depending on the user's emotional state. This process is powered by the dlib and Emotion Recognition libraries.

[0830] When a user asks a question about a product, the device sends the question as text to the server, which uses a generative artificial intelligence model to generate an answer corresponding to the question and provides it to the user in voice and text.

[0831] For example, if a user scans a QR code for a "smartphone case" in a virtual store, the server retrieves the corresponding product information. Based on that information, a generative AI model generates a description such as, "This smartphone case is shock-resistant and available in a variety of colors," and a speech synthesizer converts it into voice data. If the user is excited, the device can provide a description in an energetic tone, such as, "It's on sale now, so now's your chance!"

[0832] Examples of prompts include:

[0833] What product are you looking at right now?

[0834] Could you tell me more about this product?

[0835] Are there any other products you would recommend?

[0836] This system makes it possible to provide information according to the user's emotional state, thereby improving the satisfaction of the shopping experience in the virtual store.

[0837] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0838] Step 1:

[0839] The terminal reads the QR code scanned by the user using the camera on a smartphone or smart glasses. The input is the QR code, and the output is the product identification information obtained from the QR code. Specifically, the terminal analyzes the QR code from the image data captured by the camera and extracts the product identification information.

[0840] Step 2:

[0841] The server retrieves the corresponding product information from the database based on the product identification information obtained from the QR code. The input is the product identification information, and the output is detailed information such as the product name, description, and price. Specifically, it sends a request to the REST API and retrieves the product information as response data.

[0842] Step 3:

[0843] The server inputs product information into a generative artificial intelligence model to generate detailed descriptions. The input is product information, and the output is detailed descriptions. Specifically, the server feeds product information into a machine learning model to generate natural-looking text.

[0844] Step 4:

[0845] The device uses a speech synthesizer to convert the explanatory text into audio data. The input is a detailed explanatory text, and the output is audio data. Specifically, the device converts the text into audio using a text-to-speech engine (e.g., Pyttsx3).

[0846] Step 5:

[0847] The device uses a camera to capture the user's facial expressions in real time and uses an emotion engine to recognize the user's emotions. The input is real-time video data, and the output is the user's emotional state. Specifically, it uses dlib and the Emotion Recognition library to extract facial features and determine the user's emotional state.

[0848] Step 6:

[0849] The device adjusts the tone and content of the generated commentary based on the user's emotional state. The input is the user's emotional state and commentary, and the output is the adjusted commentary. Specifically, the device applies an algorithm that changes the tone and wording of the commentary depending on the results of the emotion engine.

[0850] Step 7:

[0851] The terminal provides the adjusted commentary to the user as audio data. The input is the adjusted commentary, and the output is audio output to the user. Specifically, the terminal uses the speech synthesizer again to convert the adjusted text into audio and play it through the speaker.

[0852] Step 8:

[0853] When a user asks a question about a product, the device sends the question in text format to the server. The input is the user's question, and the output is the question data in text format. Specifically, the device captures the question entered in the text input field within the app and sends it to the server.

[0854] Step 9:

[0855] The server uses a generative artificial intelligence model to generate an answer corresponding to the question and provides the answer to the user in voice and text. The input is a text question, and the output is voice and text data of the answer. Specifically, the server provides the question to a machine learning model, generates an appropriate answer, and converts the answer into voice data using a speech synthesizer.

[0856] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0857] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0858] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0859] [Third embodiment]

[0860] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0861] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0862] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0863] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0864] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0865] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0866] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0867] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0868] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0869] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0870] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0871] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0872] MODE FOR CARRYING OUT THE INVENTION

[0873] This invention relates to a system that provides an interactive voice guide application that provides real-time game commentary and player information while watching a baseball game. Hereinafter, this will be referred to as the "talking AI baseball commentator application."

[0874] 1. Scan the QR code and launch the app

[0875] First, when a user sits down at a baseball stadium, they use their smartphone camera to scan the QR code attached to the seat. The device then interprets the QR code and launches a URL or dedicated application. The device then transmits the seat information obtained from the QR code to the server, which then uses this information to retrieve relevant game data.

[0876] 2. Mode Selection

[0877] When the application starts, the device displays a UI that prompts the user to select either "batter mode" or "pitcher mode." The user selects the mode they are interested in.

[0878] 3. Batter mode processing

[0879] 3.1 Introduction of the next batter

[0880] When the user selects batter mode, the device sends a request for information about the next batter to the server. The server collects basic information about the next batter (such as name, age, alma mater, school achievements, and current season results) and inputs this information into a generative AI model. The generative AI model then generates a detailed introduction based on this information.

[0881] The server then sends the generated introduction back to the terminal, which uses a speech synthesizer to play the information aloud, thereby providing the user with detailed information about the next batter.

[0882] As a specific example, if the user selects batter mode and the next batter is named "Sato Ichiro," the device will provide audio commentary such as, "The next batter is Sato Ichiro. He is 23 years old and graduated from the University of Tokyo. He was active in the university championships during his student days, and his batting average this season is .300."

[0883] 4. Pitcher mode processing

[0884] 4.1 Pitching commentary

[0885] When a user selects pitcher mode, the device sends a request for information about the current pitcher to the server. The server collects basic information about the current pitcher (such as name, pitch type, and speed) and inputs this information into a generative AI model. The generative AI model then generates a detailed commentary based on that information.

[0886] The server then sends the generated commentary back to the device, which uses a speech synthesizer to play the information aloud, thereby providing the user with detailed information about the current pitcher and his pitches.

[0887] As a specific example, if the user selects pitcher mode and the current pitcher is named "Takahashi Jiro," the device will provide audio commentary such as, "The current pitcher is Takahashi Jiro. He throws a straight ball with a speed of 150 km / h."

[0888] 5. User Questions and Answers

[0889] Users can enter questions into a text input field within the app. For example, if they enter a question like, "How many home runs has Ichiro Sato hit?", the device sends the question to the server. The server provides the question to a generative artificial intelligence model, which generates an appropriate answer.

[0890] The server sends the generated answer back to the device, which uses a speech synthesizer to provide the answer aloud and also displays the answer as text.

[0891] As a specific example, in response to the question "How many home runs has Ichiro Sato hit?" the device will provide an audio commentary such as "Ichiro Sato has hit 10 home runs this season."

[0892] This invention allows spectators to receive detailed information about the game and audio commentary from players in real time, deepening their understanding of the game and making it enjoyable even for beginners.

[0893] The processing flow will be explained below.

[0894] Scan the QR code and launch the app

[0895] Step 1:

[0896] The user scans the QR code in front of the seat with their smartphone.

[0897] Step 2:

[0898] The device analyzes the QR code and launches a dedicated application based on the information obtained.

[0899] Step 3:

[0900] The terminal sends the seat information read from the QR code to the server.

[0901] Step 4:

[0902] The server receives the seat information and retrieves the game data associated with the seat.

[0903] Mode Selection

[0904] Step 5:

[0905] The device displays a UI that allows the user to select "batter mode" or "pitcher mode."

[0906] Step 6:

[0907] The user selects either "batter mode" or "pitcher mode" on the UI.

[0908] Batter mode processing

[0909] Step 7:

[0910] The terminal sends a request for information on the next batter to the server.

[0911] Step 8:

[0912] The server obtains basic information about the next batter (such as name, age, alma mater, achievements during school, and current season's results) and provides it to the generative artificial intelligence model.

[0913] Step 9:

[0914] A generative AI model generates a detailed introduction based on basic information.

[0915] Step 10:

[0916] The server returns the generated introduction to the terminal.

[0917] Step 11:

[0918] The device uses a speech synthesizer to play the detailed introduction aloud.

[0919] Pitcher mode processing

[0920] Step 12:

[0921] The terminal sends a request to the server for information on the current pitcher.

[0922] Step 13:

[0923] The server obtains basic information about the current pitcher (such as name, pitch type, and pitch speed) and provides it to the generative artificial intelligence model.

[0924] Step 14:

[0925] A generative AI model generates detailed explanatory text based on basic information.

[0926] Step 15:

[0927] The server returns the generated commentary to the terminal.

[0928] Step 16:

[0929] The device uses a speech synthesizer to play the detailed description aloud.

[0930] User Questions and Answers

[0931] Step 17:

[0932] The user types a question into the app's text input field.

[0933] Step 18:

[0934] The terminal sends a question to the server.

[0935] Step 19:

[0936] The server provides the question to a generative artificial intelligence model, which generates an appropriate answer.

[0937] Step 20:

[0938] The server generates a response and sends it back to the terminal.

[0939] Step 21:

[0940] The device uses a speech synthesizer to play the generated answer aloud and simultaneously displays the answer as text.

[0941] This allows spectators to receive detailed information about the game and player commentary in real time via audio.

[0942] Example 1

[0943] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0944] Traditionally, spectators have had difficulty in instantly learning detailed information about the game and players. Beginner spectators, in particular, have difficulty understanding the flow of the game and the characteristics of the players, limiting their viewing experience. This creates a demand for a system that provides real-time game commentary and player information.

[0945] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0946] In this invention, the server includes a means for transmitting seat information to the server and acquiring game data, a means for providing information generated based on voice input, a means for providing a user interface that displays batter mode or pitcher mode when the application is launched, and a means for automatically launching a URL or dedicated application when the user scans a QR code. This allows spectators to obtain detailed game information and player commentary in real time via voice or text, deepening their understanding of the game. It also allows even beginners to enjoy watching the game.

[0947] 1. "Seat-related information" refers to information about where a particular spectator is sitting, such as the seat location or number when watching a baseball game.

[0948] 2. "Code" means a machine-readable symbol, such as a barcode or QR code, that contains information.

[0949] 3. "Application" means a software program that runs on a smartphone or other device.

[0950] 4. "Mode" means a type of setting or feature within an application that a user can select, such as batter mode or pitcher mode.

[0951] 5. A "generative artificial intelligence model" is an artificial intelligence technology that generates new data or documents based on provided information.

[0952] 6. "Speech synthesis device" means a technical device that converts text data into speech and plays it back.

[0953] 7. "Question" means a question or inquiry entered by a User within the Application.

[0954] 8. An "answer" is information or explanation provided in response to a user's question.

[0955] 9. "Server" means a computer system that processes and provides data over a network.

[0956] 10. "Match Data" means data including detailed information about the match currently being played and player performance information.

[0957] 11. "Voice input" means a means by which a user provides information to a terminal by voice.

[0958] 12. "User interface" refers to the screens and elements that allow a user to operate an application.

[0959] 13. "URL" means an address used to identify a resource on the Internet.

[0960] 14. "Purpose-built Application" means an application designed for a specific purpose or function.

[0961] This invention is a system that provides an interactive voice guide application that provides real-time game commentary and player information while watching a baseball game. Hereafter, we will refer to it as the "talking AI baseball commentator application."

[0962] The system begins when a user uses their smartphone camera to read a QR code containing seat-related information. When the user scans the QR code, the device interprets it and automatically launches the retrieved URL or dedicated application. The seat information obtained from the QR code is sent from the device to the server, which then retrieves game data based on the received seat information.

[0963] When the application starts, the device prompts the user through the user interface to select "batter mode" or "pitcher mode." The user can select the mode they are interested in. If the user selects batter mode, the device sends an information request for the next batter to the server. The server collects the next batter's basic information (name, age, alma mater, achievements during school days, performance this season, etc.) from the database and inputs it into the generative AI model.

[0964] The generative AI model used is, for example, OpenAI's GPT-4. An example of a specific prompt is, "Please provide a detailed description of the next batter." The server retrieves the introduction generated from the AI ​​model and sends it back to the device. The device then uses a speech synthesizer (e.g., Google Text-to-Speech) to convert the introduction into audio and play it back to the user.

[0965] Similarly, when the user selects pitcher mode, the device sends a request for information about the current pitcher to the server. The server collects basic information about the current pitcher (such as name, pitch type, and speed) from the database and inputs it into the generative AI model. An example of a prompt sentence is "Please explain the current pitcher in detail." The generated explanation is sent back from the server to the device, converted into speech by a speech synthesizer, and played back to the user.

[0966] Users can also enter questions into a text input field within the app. For example, if a user enters the question, "How many home runs has Ichiro Sato hit?", the device sends the question to the server. The server inputs the question into a generative AI model and generates an appropriate answer. An example of a specific prompt sentence is, "How many home runs has Ichiro Sato hit?" The generated answer is sent back from the server to the device, where it is converted into speech by a speech synthesizer and also displayed as text.

[0967] This allows spectators to hear detailed information about the match and player commentary in real time, deepening their understanding of the match. Even beginners can enjoy the system. This system can significantly improve the experience of watching a match.

[0968] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0969] Step 1: Scan the QR code

[0970] The user uses the smartphone camera to scan a QR code containing seat-related information. The device then uses the smartphone's QR code library (e.g., ZXing) to analyze the QR code and obtain seat information or an application URL. The input is the QR code image, and the output is the seat information or URL. Specifically, the user points the camera at the QR code and presses the scan button to execute this operation.

[0971] Step 2: Launching the Application

[0972] The device accesses the URL obtained from the QR code and automatically launches the talking AI baseball commentator app. If the obtained URL is an installation link for a dedicated app, the device first installs the app and then launches it. The input is the URL, and the output is the launched application. Specifically, the device's browser accesses the URL and the app installation screen is displayed.

[0973] Step 3: Submit seating information

[0974] The device sends the seating information obtained through analysis to the server. At this time, the device uses the HTTP protocol to send the seating information as a POST request to the server's API endpoint. The input is the seating information, and the output is the data sent to the server. Specifically, a piece of software on the device automatically sends the seating information to the server.

[0975] Step 4: Obtaining match data

[0976] Based on the seat information received, the server retrieves the match data related to that seat from a database (e.g., MySQL). The server uses an SQL query to search the database and extract the relevant match data. The input is the seat information and the output is the match data. Specifically, the server sends the SQL query to the database and the results are returned.

[0977] Step 5: Display the mode selection screen

[0978] The device displays buttons on the application's UI to select "batter mode" or "pitcher mode." These buttons are generated using a UI framework (e.g., Flutter). The input is the launched application, and the output is the display of the user interface. Specifically, the mode selection buttons are displayed on the device screen.

[0979] Step 6: User Mode Selection

[0980] The user taps the button for the mode they are interested in. The tap event causes the device to retain information about the selected mode and proceed to the next step. The input is the user's selection action, and the output is information about the selected mode. A specific operation is when the user presses the "batter mode" or "pitcher mode" button.

[0981] Step 7: Request information about the next batter (in batter mode)

[0982] When the user selects "batter mode," the device requests information about the next batter from the server. This is an API request using the HTTP protocol, with the input being the mode selection information and the output being a request to the server. Specifically, the device automatically sends the request.

[0983] Step 8: Gather batter information

[0984] The server collects the next batter's basic information (e.g., name, age, alma mater, achievements during school, and this season's results) from a database. To do this, it searches the database using an SQL query. The input is a batter information request, and the output is the batter's basic information. Specifically, the server executes the SQL query and retrieves the results.

[0985] Step 9: Generative AI model generates introduction text

[0986] The server inputs the collected batter information into a generative AI model (for example, OpenAI's GPT-4). The generative AI model generates a detailed introduction based on the provided information. A specific prompt is "Please provide a detailed description of the next batter." The input is basic information about the batter, and the output is the generated introduction. In concrete terms, the server supplies the batter information to the generative AI model and generates the introduction.

[0987] Step 10: Send back testimonials and play audio

[0988] The server returns the generated introduction to the device. The device converts the introduction into speech using speech synthesis technology (e.g., Google Text-to-Speech) and plays it back. The input is the generated introduction, and the output is audio data. Specifically, the device inputs the introduction into the speech synthesis engine, and the audio is played back from the speaker.

[0989] Step 11: Pitcher Information Request (Pitcher Mode)

[0990] When the user selects "Pitcher Mode," the device requests information about the current pitcher from the server. This is an API request using the HTTP protocol, with the input being the mode selection information and the output being a request to the server. Specifically, the device automatically sends the request.

[0991] Step 12: Gather pitcher information

[0992] The server collects basic information about the current pitcher (such as name, pitch type, speed, etc.) from a database. To do this, it searches the database using an SQL query. The input is a pitcher information request, and the output is the pitcher's basic information. Specifically, the server executes the SQL query and retrieves the results.

[0993] Step 13: Generating explanatory text using a generative AI model

[0994] The server inputs the collected pitcher information into a generative AI model (for example, OpenAI's GPT-4). The generative AI model generates a detailed explanatory text based on the provided information. A specific prompt is "Please provide a detailed explanation of the current pitcher." The input is basic information about the pitcher, and the output is the generated explanatory text. In concrete terms, the server supplies pitcher information to the generative AI model and generates an explanatory text.

[0995] Step 14: Returning commentary and playing audio

[0996] The server returns the generated explanatory text to the device. The device then uses speech synthesis technology (e.g., Google Text-to-Speech) to convert the explanatory text into audio and plays it back. The input is the generated explanatory text, and the output is audio data. Specifically, the device inputs the explanatory text into a speech synthesis engine, and the audio is played back from the speaker.

[0997] Step 15: User Questions and Answers

[0998] The user enters a question into a text input field within the app. For example, if the user enters the question "How many home runs has Ichiro Sato hit?", the input is the user's question, and the device sends this question to the server. The output is the question data sent to the server. In concrete terms, the user enters a question into the text field and presses the send button.

[0999] Step 16: Input to the question generation AI model

[1000] The server provides the received question to the generative AI model and generates an appropriate answer. A specific prompt sentence is, "How many home runs has Ichiro Sato hit?" The input is the user's question, and the output is the generated answer sentence. In concrete terms, the server inputs the question into the generative AI model and generates an answer.

[1001] Step 17: Return and provide answers

[1002] The server returns the generated answer to the terminal. The terminal uses speech synthesis technology to convert the answer into speech and plays it back. At the same time, it also displays the answer as text. The input is the generated answer, and the output is audio data and text data. In concrete terms, the terminal inputs the answer into the speech synthesis engine, and the audio is played back from the speaker while the text is displayed on the screen.

[1003] (Application example 1)

[1004] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1005] In traditional shopping experiences, consumers often find it difficult to obtain detailed product information instantly in physical stores. This is especially true when selecting the perfect product from the numerous products lined up on the shelves, requiring significant effort from consumers, and it is difficult to obtain detailed information without directly asking a store clerk. This often results in an unsatisfactory shopping experience for consumers.

[1006] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1007] In this invention, the server includes: means for reading a code containing information related to a seat; means for providing an application launched by reading the code; means for a user to select a mode in the application; a generative artificial intelligence model that generates detailed information about an object based on the mode selected by the user; a speech synthesizer that provides the information generated by the generative artificial intelligence model in audio; means for answering a question entered by a user in the application; means for generating an answer to the question and providing it in audio and text; means for the application to read a code placed on a product shelf or the like and provide detailed information about the object; means for generating text based on the generative artificial intelligence model for the detailed information, including the product name, price, ingredient information, user reviews, etc., based on the generative artificial intelligence model; and means for providing an audio commentary of the object using a speech synthesizer based on the generated text. This enables consumers to instantly obtain detailed information about products in a physical store, providing an efficient and satisfying shopping experience.

[1008] A "code" is a symbol, either visual or digital, that contains specific information and that can be read and interpreted by electronic equipment.

[1009] An "application" is a software program designed to provide a specific function or service that a user uses on a smartphone or other device.

[1010] A "user" is a person who operates an application or system and receives specific information or services through operations or inputs.

[1011] A "mode" refers to a specific operating state or function set of an application, which can be selected by the user to change the system's behavior or information presentation.

[1012] A "generative artificial intelligence model" is a type of artificial intelligence that has the ability to automatically generate natural language sentences based on given input data.

[1013] A "voice synthesizer" is a device that converts text data into voice and provides information to a user in voice form.

[1014] "Detailed information" refers to specific and in-depth information about the subject matter, including product name, price, ingredient information, user reviews, etc.

[1015] A "product shelf" refers to the area or facility in a physical store where products are displayed and are set up so that consumers can pick up the products.

[1016] "Subject" refers to the specific goods or services that the application provides information about.

[1017] A "question" is a text input that requests additional information or clarification that the user desires from the application related to the current mode or information.

[1018] An "answer" is text or audio containing information generated by a generative artificial intelligence model based on a user's question.

[1019] This invention relates to a voice interactive guide application that provides consumers with real-time product explanations and promotional information in a physical store. The system is activated when a consumer scans a code placed on a product shelf in the store, and provides detailed product information.

[1020] System Configuration

[1021] Hardware

[1022] Smartphone: Camera, speaker, QR code scanner.

[1023] Product shelf: The shelf where the code is installed.

[1024] software

[1025] QR Code Scanner Library: A library for reading QR codes and obtaining product information.

[1026] HTTP request library: A library for retrieving product information from the server (e.g., Requests for Python).

[1027] Generative AI model: An AI model that generates sentences based on input data (e.g., OpenAI's GPT).

[1028] Speech synthesis API: An API for converting generated text into speech (e.g., Google Text-to-Speech).

[1029] Processing Description

[1030] 1. Scan the QR code

[1031] A user uses their smartphone camera to scan a code on a shelf.

[1032] The terminal analyzes the code it reads and obtains the product ID.

[1033] 2. Obtaining product information

[1034] Based on the product ID, the terminal sends an HTTP request to the server to obtain detailed product information.

[1035] The server collects detailed information such as product name, price, ingredient information, and user reviews and sends it back to the device.

[1036] 3. Utilizing generative artificial intelligence models

[1037] The terminal inputs the acquired product information into a generative artificial intelligence model and generates a detailed product description using prompt sentences.

[1038] For example, the prompt text might look like this:

[1039] "Product name: Organic apples. Characteristics: Grown without pesticides, very sweet. Price: 500 yen. Reviews: Popular for their sweetness and deliciousness."

[1040] The generated product description text is returned to the terminal.

[1041] 4. Speech synthesis and information provision

[1042] The device sends the generated product description text to a speech synthesis API to generate voice data.

[1043] The generated voice is used to provide a voice explanation of the product to the user.

[1044] Specific examples

[1045] For example, if a user scans a QR code for organic apples, the device inputs the following prompt text into the generative AI model: "Product name: Organic apples\nFeatures: Grown without pesticides and very sweet\nPrice: 500 yen\nReviews: Rated for their sweetness and deliciousness." Based on this, the AI ​​generates a detailed product description and then converts it into audio commentary using a speech synthesis API. The user can receive a real-time description on their smartphone, such as, "These organic apples are grown without pesticides and are characterized by their sweetness. The price is 500 yen."

[1046] The system allows consumers to instantly obtain detailed product information in-store, providing an efficient and satisfying shopping experience.

[1047] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1048] Step 1:

[1049] A user uses a smartphone camera to scan a code on a product shelf. The input is the code, and the output is the parsed product ID. Specifically, the user activates the camera and performs an operation to read the code.

[1050] Step 2:

[1051] The terminal analyzes the scanned code and obtains the product ID. The input is the scanned code and the output is the product ID. This includes analyzing the code and extracting string information.

[1052] Step 3:

[1053] The terminal sends an HTTP request to the server based on the product ID to obtain detailed product information. The input is the product ID, and the output is detailed product information such as the product name, price, ingredient information, and user reviews. Specific operations include accessing an API containing detailed product information and obtaining data.

[1054] Step 4:

[1055] The server collects detailed information such as product name, price, ingredient information, and user reviews, and sends it back to the terminal. The input is a request based on the product ID, and the output is detailed product information. The server also organizes the collected data and returns it as an HTTP response.

[1056] Step 5:

[1057] The terminal inputs the acquired product information into a generative AI model and generates a detailed product description using a prompt. The input is the product information, and the output is the generated product description. Specific operations include constructing a prompt and sending it to the AI ​​model to generate a sentence.

[1058] Step 6:

[1059] The device sends the generated product description to a speech synthesis API to generate voice data. The input is the generated product description, and the output is voice data. Specific operations include converting the generated text into voice data.

[1060] Step 7:

[1061] The device uses the generated audio data to provide the user with an audio description of the product. The input is the audio data and the output is the audio description. An operation of playing the audio using the smartphone speaker is included.

[1062] By performing these steps in succession, users can instantly obtain detailed information about products by voice in a physical store.

[1063] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1064] MODE FOR CARRYING OUT THE INVENTION

[1065] This invention relates to a system that combines an emotion engine that recognizes the user's emotions with a voice-activated interactive guide application that provides real-time game commentary and player information during baseball games. Hereinafter, this system will be referred to as the "emotion-recognition AI baseball commentator app."

[1066] 1. Scan the QR code and launch the app

[1067] First, when a user sits down at a baseball stadium, they use their smartphone camera to scan the QR code attached to the seat. The device then interprets the QR code and launches a URL or dedicated application. The device then transmits the seat information obtained from the QR code to the server, which then uses this information to retrieve relevant game data.

[1068] 2. Mode Selection

[1069] When the application starts, the device displays a UI that prompts the user to select either "batter mode" or "pitcher mode." The user selects the mode they are interested in.

[1070] 3. Batter mode processing

[1071] 3.1 Introduction of the next batter

[1072] When the user selects batter mode, the device sends a request for information about the next batter to the server. The server collects basic information about the next batter (such as name, age, alma mater, school achievements, and current season results) and inputs this information into a generative AI model. The generative AI model then generates a detailed introduction based on this information.

[1073] The server then sends the generated introduction back to the device, which uses a speech synthesizer to play the information aloud, and the device also uses an emotion engine to monitor the user's reactions and adjust the tone and content of the introduction according to the user's emotions.

[1074] For example, if the user selects batter mode and the next batter is named "Ichiro Sato," the device will provide audio commentary such as, "The next batter is Ichiro Sato. He is 23 years old and graduated from the University of Tokyo. He played in the university championships as a student. His batting average this season is .300." If the emotion engine recognizes that the user is excited, it can make the tone of the commentary more energetic.

[1075] 4. Pitcher mode processing

[1076] 4.1 Pitching commentary

[1077] When a user selects pitcher mode, the device sends a request for information about the current pitcher to the server. The server collects basic information about the current pitcher (such as name, pitch type, and speed) and inputs this information into a generative AI model. The generative AI model then generates a detailed commentary based on that information.

[1078] The server then sends the generated commentary back to the device, which uses a speech synthesizer to play the information aloud, and the device also uses an emotion engine to monitor the user's reactions and adjust the tone and content of the commentary according to the user's emotions.

[1079] For example, if the user selects pitcher mode and the current pitcher is named "Takahashi Jiro," the device will provide a voice commentary such as, "The current pitcher is Takahashi Jiro. He throws a fastball with a speed of 150 km / h." If the emotion engine determines that the user is calm, the commentary can be more detailed and educational.

[1080] 5. User Questions and Answers

[1081] Users can enter questions into a text input field within the app. For example, if they enter a question like, "How many home runs has Ichiro Sato hit?", the device sends the question to the server. The server provides the question to a generative artificial intelligence model, which generates an appropriate answer.

[1082] The server sends the generated answer back to the device, which uses a speech synthesizer to provide the answer aloud and simultaneously displays the answer as text.The device also uses an emotion engine to monitor the user's reaction and provide the answer in a tone that corresponds to the user's emotion.

[1083] For example, in response to the question, "How many home runs has Ichiro Sato hit?", the device can provide a voice commentary such as, "Ichiro Sato has hit 10 home runs this season." If the emotion engine determines that the user is happy, it can respond by brightening the tone.

[1084] This method allows spectators to obtain detailed information about the game and players in real time, while receiving commentary that is tailored to their own emotions. This is expected to deepen their understanding of the game and increase their enjoyment of watching it.

[1085] The processing flow will be explained below.

[1086] Scan the QR code and launch the app

[1087] Step 1:

[1088] The user scans the QR code in front of the seat with their smartphone.

[1089] Step 2:

[1090] The device analyzes the QR code and launches a dedicated application based on the information obtained.

[1091] Step 3:

[1092] The terminal sends the seat information read from the QR code to the server.

[1093] Step 4:

[1094] The server receives the seat information and retrieves the game data associated with the seat.

[1095] Mode Selection

[1096] Step 5:

[1097] The device displays a UI that allows the user to select "batter mode" or "pitcher mode."

[1098] Step 6:

[1099] The user selects either "batter mode" or "pitcher mode" on the UI.

[1100] Batter mode processing

[1101] Step 7:

[1102] The terminal sends a request for information on the next batter to the server.

[1103] Step 8:

[1104] The server obtains basic information about the next batter (such as name, age, alma mater, achievements during school, and current season's results) and provides it to the generative artificial intelligence model.

[1105] Step 9:

[1106] A generative AI model generates a detailed introduction based on basic information.

[1107] Step 10:

[1108] The server returns the generated introduction to the terminal.

[1109] Step 11:

[1110] The device uses a speech synthesizer to play the detailed introduction aloud.

[1111] Step 12:

[1112] The device uses an emotion engine to monitor the user's reactions and adjust the tone and content of the introduction depending on the user's emotions.

[1113] Pitcher mode processing

[1114] Step 13:

[1115] The terminal sends a request to the server for information on the current pitcher.

[1116] Step 14:

[1117] The server obtains basic information about the current pitcher (such as name, pitch type, and pitch speed) and provides it to the generative artificial intelligence model.

[1118] Step 15:

[1119] A generative AI model generates detailed explanatory text based on basic information.

[1120] Step 16:

[1121] The server returns the generated commentary to the terminal.

[1122] Step 17:

[1123] The device uses a speech synthesizer to play the detailed description aloud.

[1124] Step 18:

[1125] The device uses an emotion engine to monitor the user's reactions and adjust the tone and content of the commentary according to the user's emotions.

[1126] User Questions and Answers

[1127] Step 19:

[1128] The user types a question into the app's text input field.

[1129] Step 20:

[1130] The terminal sends a question to the server.

[1131] Step 21:

[1132] The server provides the question to a generative artificial intelligence model, which generates an appropriate answer.

[1133] Step 22:

[1134] The server generates a response and sends it back to the terminal.

[1135] Step 23:

[1136] The device uses a speech synthesizer to play the generated answer aloud and simultaneously displays the answer as text.

[1137] Step 24:

[1138] The device uses an emotion engine to monitor the user's reactions and provides answers in a tone that corresponds to the user's emotions.

[1139] This allows spectators to receive detailed information about the game and player commentary in real time via audio, and they can also receive the most appropriate commentary based on their own emotions.

[1140] Example 2

[1141] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1142] In modern sports viewing, especially baseball, there is a demand for detailed game commentary and player information in real time. However, conventional systems often provide only one-way information without considering the user's emotions, and improvements are needed to increase spectator satisfaction. In particular, a challenge is to develop technology that dynamically adjusts the tone and content of information provided according to the user's emotions.

[1143] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for reading a code including seat-related information; means for providing an application launched by reading the code; means for a user to select a mode in the application; a generative AI model that generates detailed information about a batter or pitcher based on the mode selected by the user; a voice synthesizer that provides the information generated by the generative AI model as audio; an emotion engine that analyzes the user's facial expressions and tone of voice to recognize emotions; means for adjusting the tone and content of audio commentary based on the analysis results of the emotion engine; means for answering questions entered by the user in the application; and means for generating answers to the questions and providing them in audio and text. This makes it possible to provide personalized real-time game commentary and player information according to the user's emotions.

[1144] A "code containing information related to the seat" is a code containing information such as the seat position and number where the user is seated, and includes QR codes and barcodes.

[1145] "Means for reading the code" refers to a device, such as a QR code reader or barcode scanner, or an application that is used to interpret the information in the code.

[1146] "Means for providing applications" refers to applications that run on smartphones or tablets and have the ability to start automatically by loading code.

[1147] "Means for the user to select the mode" refers to a user interface (UI) that provides the user with the ability to select "batter mode" or "pitcher mode" using on-screen buttons or menus.

[1148] A "generative artificial intelligence model" refers to a generative AI based on natural language processing, for example, which generates appropriate sentences and explanations based on input information.

[1149] A "voice synthesizer" refers to a device or software that converts text data into voice data and plays it back.

[1150] An "emotion engine" refers to a device or software that analyzes a user's facial expressions and tone of voice to recognize their emotional state at that time.

[1151] A "means for answering questions" is a function that generates appropriate answers to questions entered by users and provides them in voice or text, and often uses a generative AI model.

[1152] The "means for adjusting tone and content" provides a function for dynamically changing the tone and amount of detailed information in the generated commentary or introduction based on the analysis results of the emotion engine.

[1153] This invention is implemented as an "emotion-recognition AI baseball commentator app." The following is a detailed explanation of the system, including the hardware and software used to perform each process.

[1154] Hardware and Software

[1155] This system uses the following hardware and software:

[1156] Smartphone: A mobile device held by the user that has a camera to read QR codes, runs applications, inputs voice using a microphone, and outputs voice using a speaker.

[1157] QR Code Reader: QR code analysis software that uses a smartphone camera.

[1158] Server: Retrieves match data, processes specified API endpoints, runs generative AI models, and processes user requests.

[1159] Generative AI model: An artificial intelligence used to generate detailed information and explanatory text (e.g., GPT-3).

[1160] Speech synthesizer: A TTS (Text-to-Speech) engine that converts text data into voice data.

[1161] Emotion engine: Software that analyzes the user's facial expressions and tone of voice in real time.

[1162] System Operation

[1163] Scan the QR code and launch the app

[1164] When a user sits down at a baseball stadium, they use their smartphone camera to scan the QR code installed in front of their seat. The QR code reader analyzes the code and automatically launches a URL or dedicated application based on the results. The device then sends the seat information obtained from the QR code to the server, which then retrieves the relevant game data.

[1165] Mode Selection

[1166] When the application starts, the device displays a UI that prompts the user to select either "batter mode" or "pitcher mode." The user selects the mode they are interested in by tapping a button on the screen.

[1167] Batter mode processing

[1168] When the user selects batter mode, the device sends a request for information about the next batter to the server. The server collects basic information about the next batter from the current batting order and creates a prompt to input to the generative AI model. The generative AI model generates a detailed introduction based on the sent prompt. The generated introduction is sent back to the device by the server. The device uses a speech synthesizer to play the introduction as audio.

[1169] Next, the device uses an emotion engine to analyze the user's facial expressions and tone of voice, monitoring emotions in real time. If the user is excited, the tone of the commentary can be adjusted to be more energetic. For example, if the next batter's name is "Yamada Taro," the prompt would be "Tell me about Yamada Taro. He's 25 years old and graduated from University A. He played in the university championships as a student. His batting average this season is .320." This text would be input into the generative AI model.

[1170] Pitcher mode processing

[1171] When the user selects pitcher mode, the device sends a request for information about the current pitcher to the server. The server collects the current pitcher information and creates a prompt to input to the generative AI model. The generative AI model generates a detailed commentary based on the sent prompt. The generated commentary is sent back to the device by the server. The device uses a speech synthesizer to play the commentary as audio.

[1172] The device then uses an emotion engine to analyze the user's facial expressions and tone of voice to monitor their emotions in real time. If the user is calm, the commentary can be more detailed and educational. For example, if the current pitcher's name is "Jiro Nakamura," the prompt would be "Please describe pitcher Jiro Nakamura. He throws a straight ball with a speed of 145 km / h," and the text would be input into the generative AI model.

[1173] User Questions and Answers

[1174] The user enters a question into a text input field within the app and taps the submit button. The device sends the question to the server, which provides it as a prompt to the generative AI model to generate an appropriate answer. The generated answer is then sent back to the device by the server, and the device uses a speech synthesizer to provide the answer as voice and simultaneously displays the answer as text.

[1175] The device uses an emotion engine to analyze the user's facial expressions and tone of voice, monitoring emotions in real time. Based on the analysis results of the emotion engine, it can provide answers in a tone that corresponds to the user's emotions. For example, if a question such as "Please tell me the schedule for the next game" is entered, the prompt text "Please tell me the schedule for the next game" will be input into the generative AI model.

[1176] In this way, we can provide real-time game commentary and player information based on the user's emotions. This system is expected to make baseball watching more engaging and enjoyable.

[1177] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1178] Step 1:

[1179] A user sits in a seat at a baseball stadium and scans a QR code with a smartphone camera. The input is an image of the QR code, and the output is a URL containing seat information or the launch of a dedicated application. The QR code reader analyzes the image and extracts the seat information.

[1180] Step 2:

[1181] The device sends the seating information obtained from the QR code to the server. The input here is the seating information analyzed from the QR code, and the output is the seating information sent to the server. The device sends this information to the server as an HTTP request and obtains the access key for the match data.

[1182] Step 3:

[1183] The server retrieves data on the currently playing game from the database based on the seating information. The input is the seating information received from the terminal, and the output is the game data. The server searches for related game data based on the seating information and retrieves the necessary information.

[1184] Step 4:

[1185] The device displays a UI on the application's home screen that allows the user to select "batter mode" or "pitcher mode." The input is the game data received from the server, and the output is the mode selection screen displayed on the user interface. The device dynamically renders the UI components.

[1186] Step 5:

[1187] The user taps a button on the screen to select a mode. The input is the user's tap, and the output is the selected mode ("batter mode" or "pitcher mode").

[1188] Step 6:

[1189] When the user selects "batter mode," the terminal sends a request for information about the next batter to the server. The input is the selected "batter mode," and the output is an information request to the server. The terminal sends an HTTP request to the server, requesting information about the next batter.

[1190] Step 7:

[1191] The server collects basic information about the next batter and creates a prompt to input into the generative AI model. The input is the next batter's basic information (e.g., name, age, alma mater, grades), and the output is the prompt to input into the generative AI model. The server generates a detailed introduction based on this.

[1192] Step 8:

[1193] The generative AI model generates a detailed introduction. The input is the prompt sent from the server, and the output is a detailed introduction. The generative AI model uses an NLP algorithm to generate a sentence based on the prompt.

[1194] Step 9:

[1195] The server returns the generated introductory text to the device. The input is the introductory text generated by the generative AI model, and the output is the introductory text returned to the device. The server returns the introductory text as an HTTP response.

[1196] Step 10:

[1197] The device uses a speech synthesizer to play the introduction aloud. The input is the text data of the introduction received from the server, and the output is audio data. The speech synthesizer converts the text data into audio and plays it back.

[1198] Step 11:

[1199] The device uses an emotion engine to monitor the user's reactions. The input is the user's facial expressions and tone of voice, and the output is the analysis results of the emotion engine. The device uses a camera and microphone to analyze the user's emotions in real time.

[1200] Step 12:

[1201] The device adjusts the tone and content of the introduction based on the user's emotions. The input is the analysis result of the emotion engine, and the output is the adjusted voice. Based on the analysis result, the device appropriately changes the tone and specificity of the introduction from the next time onwards.

[1202] Step 13:

[1203] When the user selects "pitcher mode," the terminal sends a request for information about the current pitcher to the server. The input is the selected "pitcher mode," and the output is the information request to the server. The terminal sends an HTTP request to the server, requesting information about the current pitcher.

[1204] Step 14:

[1205] The server collects basic information about the current pitcher and creates a prompt to input to the generative AI model. The input is the basic information about the current pitcher (e.g., name, pitch type, speed), and the output is the prompt to input to the generative AI model. The server generates a detailed explanation based on this.

[1206] Step 15:

[1207] The generative AI model generates detailed explanatory text. The input is the prompt sent from the server, and the output is the detailed explanatory text. The generative AI model uses an NLP algorithm to generate text based on the prompt.

[1208] Step 16:

[1209] The server returns the generated commentary to the device. The input is the commentary generated by the generative AI model, and the output is the commentary returned to the device. The server returns the commentary as an HTTP response.

[1210] Step 17:

[1211] The device uses a speech synthesizer to play the explanatory text aloud. The input is the text data of the explanatory text received from the server, and the output is audio data. The speech synthesizer converts the text data into audio and plays it back.

[1212] Step 18:

[1213] The device uses an emotion engine to monitor the user's reactions. The input is the user's facial expressions and tone of voice, and the output is the analysis results of the emotion engine. The device uses a camera and microphone to analyze the user's emotions in real time.

[1214] Step 19:

[1215] The device adjusts the tone and content of the commentary based on the user's emotions. The input is the analysis result of the emotion engine, and the output is the adjusted voice. Based on the analysis result, the device appropriately changes the tone and specificity of the commentary from the next time onwards.

[1216] Step 20:

[1217] The user enters a question into a text input field in the app and taps the submit button. The input is the question entered by the user, and the output is what the device sends to the server.

[1218] Step 21:

[1219] The terminal sends a question to the server. The input is the question entered by the user, and the output is the question sent to the server.

[1220] Step 22:

[1221] The server provides the generative AI model with a question as a prompt, and it generates an appropriate answer. The input is the prompt sentence of the user question, and the output is the generated answer.

[1222] Step 23:

[1223] The server sends the generated answer back to the device. The input is the answer generated by the generative AI model, and the output is the answer sent back to the device.

[1224] Step 24:

[1225] The terminal uses a speech synthesizer to provide a voice response. The input is the text data of the response received from the server, and the output is voice data. The speech synthesizer converts the text data into voice and plays it back.

[1226] Step 25:

[1227] The device simultaneously displays the answer as text on the UI. The input is the text data of the answer received from the server, and the output is the text displayed on the screen.

[1228] Step 26:

[1229] The device uses an emotion engine to monitor the user's reactions. The input is the user's facial expressions and tone of voice, and the output is the analysis results of the emotion engine. The device uses a camera and microphone to analyze the user's emotions in real time.

[1230] Step 27:

[1231] The device responds with a tone that reflects the user's emotions. The input is the analysis result of the emotion engine, and the output is an adjusted voice. Based on the analysis result, the device appropriately changes the tone and content of future responses.

[1232] (Application example 2)

[1233] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1234] The virtual shopping experience has the problem that product information is provided uniformly and does not respond to the user's individual emotions and interests, which may reduce the user's motivation to purchase and their satisfaction.

[1235] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1236] In this invention, the server includes: means for reading a code containing seat-related information; means for providing an application launched by reading the code; means for a user to select a mode in the application; a generative AI model for generating detailed information about general information based on the mode selected by the user; a speech synthesizer for providing the information generated by the generative AI model in voice; means for recognizing the user's emotion using an emotion engine and adjusting the tone and content of the information based on the recognized emotion; means for answering questions entered by the user in the application; and means for generating answers to the questions and providing them in voice and text. This makes it possible to provide information according to the user's emotional state, thereby improving the satisfaction of the shopping experience in the virtual store.

[1237] A "code containing information related to seats" is a QR code or barcode that stores information related to the location and characteristics of products in a virtual store and user behavior.

[1238] The "means for providing applications" is a mechanism for automatically launching specific programs or applications on a user's device and enabling specific functions or services.

[1239] "Means for user mode selection" means an interface or button within an application that allows a user to select an operating mode according to their interests and needs.

[1240] A "generative artificial intelligence model" is an artificial intelligence model that generates new information and explanatory text based on input data, and utilizes machine learning and natural language processing technologies.

[1241] A "speech synthesizer" refers to a device or software that converts information generated from text data or generative artificial intelligence models into speech format and provides it to users.

[1242] An "emotion engine" is artificial intelligence or software that analyzes a user's facial expressions, tone of voice, etc. to recognize their emotional state.

[1243] "Means for answering questions" refers to a mechanism or interface for generating appropriate answers to questions entered by a user and providing those answers in voice and text format.

[1244] The present invention is a system for providing a voice interactive assistant with emotion recognition capabilities to enhance the shopping experience in a virtual store. The system operates as follows.

[1245] First, while walking through the virtual store, the user scans the QR code of a product with the camera on their smartphone or smart glasses. The server identifies the product information by reading this QR code.

[1246] Next, the server acquires the product information and inputs it into a generative AI model. The generative AI model generates a detailed explanatory text based on the product information. The generated explanatory text is then converted into voice data using a voice synthesizer.

[1247] Additionally, the device uses a camera to capture the user's facial expressions in real time and recognizes their emotions using an emotion engine. The device adjusts the tone and content of the generated commentary depending on the user's emotional state. This process is powered by the dlib and Emotion Recognition libraries.

[1248] When a user asks a question about a product, the device sends the question as text to the server, which uses a generative artificial intelligence model to generate an answer corresponding to the question and provides it to the user in voice and text.

[1249] For example, if a user scans a QR code for a "smartphone case" in a virtual store, the server retrieves the corresponding product information. Based on that information, a generative AI model generates a description such as, "This smartphone case is shock-resistant and available in a variety of colors," and a speech synthesizer converts it into voice data. If the user is excited, the device can provide a description in an energetic tone, such as, "It's on sale now, so now's your chance!"

[1250] Examples of prompts include:

[1251] What product are you looking at right now?

[1252] Could you tell me more about this product?

[1253] Are there any other products you would recommend?

[1254] This system makes it possible to provide information according to the user's emotional state, thereby improving the satisfaction of the shopping experience in the virtual store.

[1255] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1256] Step 1:

[1257] The terminal reads the QR code scanned by the user using the camera on a smartphone or smart glasses. The input is the QR code, and the output is the product identification information obtained from the QR code. Specifically, the terminal analyzes the QR code from the image data captured by the camera and extracts the product identification information.

[1258] Step 2:

[1259] The server retrieves the corresponding product information from the database based on the product identification information obtained from the QR code. The input is the product identification information, and the output is detailed information such as the product name, description, and price. Specifically, it sends a request to the REST API and retrieves the product information as response data.

[1260] Step 3:

[1261] The server inputs product information into a generative artificial intelligence model to generate detailed descriptions. The input is product information, and the output is detailed descriptions. Specifically, the server feeds product information into a machine learning model to generate natural-looking text.

[1262] Step 4:

[1263] The device uses a speech synthesizer to convert the explanatory text into audio data. The input is a detailed explanatory text, and the output is audio data. Specifically, the device converts the text into audio using a text-to-speech engine (e.g., Pyttsx3).

[1264] Step 5:

[1265] The device uses a camera to capture the user's facial expressions in real time and uses an emotion engine to recognize the user's emotions. The input is real-time video data, and the output is the user's emotional state. Specifically, it uses dlib and the Emotion Recognition library to extract facial features and determine the user's emotional state.

[1266] Step 6:

[1267] The device adjusts the tone and content of the generated commentary based on the user's emotional state. The input is the user's emotional state and commentary, and the output is the adjusted commentary. Specifically, the device applies an algorithm that changes the tone and wording of the commentary depending on the results of the emotion engine.

[1268] Step 7:

[1269] The terminal provides the adjusted commentary to the user as audio data. The input is the adjusted commentary, and the output is audio output to the user. Specifically, the terminal uses the speech synthesizer again to convert the adjusted text into audio and play it through the speaker.

[1270] Step 8:

[1271] When a user asks a question about a product, the device sends the question in text format to the server. The input is the user's question, and the output is the question data in text format. Specifically, the device captures the question entered in the text input field within the app and sends it to the server.

[1272] Step 9:

[1273] The server uses a generative artificial intelligence model to generate an answer corresponding to the question and provides the answer to the user in voice and text. The input is a text question, and the output is voice and text data of the answer. Specifically, the server provides the question to a machine learning model, generates an appropriate answer, and converts the answer into voice data using a speech synthesizer.

[1274] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1275] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1276] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1277] [Fourth embodiment]

[1278] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1279] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1280] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1281] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1282] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1283] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1284] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1285] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1286] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1287] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1288] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1289] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1290] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1291] MODE FOR CARRYING OUT THE INVENTION

[1292] This invention relates to a system that provides an interactive voice guide application that provides real-time game commentary and player information while watching a baseball game. Hereinafter, this will be referred to as the "talking AI baseball commentator application."

[1293] 1. Scan the QR code and launch the app

[1294] First, when a user sits down at a baseball stadium, they use their smartphone camera to scan the QR code attached to the seat. The device then interprets the QR code and launches a URL or dedicated application. The device then transmits the seat information obtained from the QR code to the server, which then uses this information to retrieve relevant game data.

[1295] 2. Mode Selection

[1296] When the application starts, the device displays a UI that prompts the user to select either "batter mode" or "pitcher mode." The user selects the mode they are interested in.

[1297] 3. Batter mode processing

[1298] 3.1 Introduction of the next batter

[1299] When the user selects batter mode, the device sends a request for information about the next batter to the server. The server collects basic information about the next batter (such as name, age, alma mater, school achievements, and current season results) and inputs this information into a generative AI model. The generative AI model then generates a detailed introduction based on this information.

[1300] The server then sends the generated introduction back to the terminal, which uses a speech synthesizer to play the information aloud, thereby providing the user with detailed information about the next batter.

[1301] As a specific example, if the user selects batter mode and the next batter is named "Sato Ichiro," the device will provide audio commentary such as, "The next batter is Sato Ichiro. He is 23 years old and graduated from the University of Tokyo. He was active in the university championships during his student days, and his batting average this season is .300."

[1302] 4. Pitcher mode processing

[1303] 4.1 Pitching commentary

[1304] When a user selects pitcher mode, the device sends a request for information about the current pitcher to the server. The server collects basic information about the current pitcher (such as name, pitch type, and speed) and inputs this information into a generative AI model. The generative AI model then generates a detailed commentary based on that information.

[1305] The server then sends the generated commentary back to the device, which uses a speech synthesizer to play the information aloud, thereby providing the user with detailed information about the current pitcher and his pitches.

[1306] As a specific example, if the user selects pitcher mode and the current pitcher is named "Takahashi Jiro," the device will provide audio commentary such as, "The current pitcher is Takahashi Jiro. He throws a straight ball with a speed of 150 km / h."

[1307] 5. User Questions and Answers

[1308] Users can enter questions into a text input field within the app. For example, if they enter a question like, "How many home runs has Ichiro Sato hit?", the device sends the question to the server. The server provides the question to a generative artificial intelligence model, which generates an appropriate answer.

[1309] The server sends the generated answer back to the device, which uses a speech synthesizer to provide the answer aloud and also displays the answer as text.

[1310] As a specific example, in response to the question "How many home runs has Ichiro Sato hit?" the device will provide an audio commentary such as "Ichiro Sato has hit 10 home runs this season."

[1311] This invention allows spectators to receive detailed information about the game and audio commentary from players in real time, deepening their understanding of the game and making it enjoyable even for beginners.

[1312] The processing flow will be explained below.

[1313] Scan the QR code and launch the app

[1314] Step 1:

[1315] The user scans the QR code in front of the seat with their smartphone.

[1316] Step 2:

[1317] The device analyzes the QR code and launches a dedicated application based on the information obtained.

[1318] Step 3:

[1319] The terminal sends the seat information read from the QR code to the server.

[1320] Step 4:

[1321] The server receives the seat information and retrieves the game data associated with the seat.

[1322] Mode Selection

[1323] Step 5:

[1324] The device displays a UI that allows the user to select "batter mode" or "pitcher mode."

[1325] Step 6:

[1326] The user selects either "batter mode" or "pitcher mode" on the UI.

[1327] Batter mode processing

[1328] Step 7:

[1329] The terminal sends a request for information on the next batter to the server.

[1330] Step 8:

[1331] The server obtains basic information about the next batter (such as name, age, alma mater, achievements during school, and current season's results) and provides it to the generative artificial intelligence model.

[1332] Step 9:

[1333] A generative AI model generates a detailed introduction based on basic information.

[1334] Step 10:

[1335] The server returns the generated introduction to the terminal.

[1336] Step 11:

[1337] The device uses a speech synthesizer to play the detailed introduction aloud.

[1338] Pitcher mode processing

[1339] Step 12:

[1340] The terminal sends a request to the server for information on the current pitcher.

[1341] Step 13:

[1342] The server obtains basic information about the current pitcher (such as name, pitch type, and pitch speed) and provides it to the generative artificial intelligence model.

[1343] Step 14:

[1344] A generative AI model generates detailed explanatory text based on basic information.

[1345] Step 15:

[1346] The server returns the generated commentary to the terminal.

[1347] Step 16:

[1348] The device uses a speech synthesizer to play the detailed description aloud.

[1349] User Questions and Answers

[1350] Step 17:

[1351] The user types a question into the app's text input field.

[1352] Step 18:

[1353] The terminal sends a question to the server.

[1354] Step 19:

[1355] The server provides the question to a generative artificial intelligence model, which generates an appropriate answer.

[1356] Step 20:

[1357] The server generates a response and sends it back to the terminal.

[1358] Step 21:

[1359] The device uses a speech synthesizer to play the generated answer aloud and simultaneously displays the answer as text.

[1360] This allows spectators to receive detailed information about the game and player commentary in real time via audio.

[1361] Example 1

[1362] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1363] Traditionally, spectators have had difficulty in instantly learning detailed information about the game and players. Beginner spectators, in particular, have difficulty understanding the flow of the game and the characteristics of the players, limiting their viewing experience. This creates a demand for a system that provides real-time game commentary and player information.

[1364] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1365] In this invention, the server includes a means for transmitting seat information to the server and acquiring game data, a means for providing information generated based on voice input, a means for providing a user interface that displays batter mode or pitcher mode when the application is launched, and a means for automatically launching a URL or dedicated application when the user scans a QR code. This allows spectators to obtain detailed game information and player commentary in real time via voice or text, deepening their understanding of the game. It also allows even beginners to enjoy watching the game.

[1366] 1. "Seat-related information" refers to information about where a particular spectator is sitting, such as the seat location or number when watching a baseball game.

[1367] 2. "Code" means a machine-readable symbol, such as a barcode or QR code, that contains information.

[1368] 3. "Application" means a software program that runs on a smartphone or other device.

[1369] 4. "Mode" means a type of setting or feature within an application that a user can select, such as batter mode or pitcher mode.

[1370] 5. A "generative artificial intelligence model" is an artificial intelligence technology that generates new data or documents based on provided information.

[1371] 6. "Speech synthesis device" means a technical device that converts text data into speech and plays it back.

[1372] 7. "Question" means a question or inquiry entered by a User within the Application.

[1373] 8. An "answer" is information or explanation provided in response to a user's question.

[1374] 9. "Server" means a computer system that processes and provides data over a network.

[1375] 10. "Match Data" means data including detailed information about the match currently being played and player performance information.

[1376] 11. "Voice input" means a means by which a user provides information to a terminal by voice.

[1377] 12. "User interface" refers to the screens and elements that allow a user to operate an application.

[1378] 13. "URL" means an address used to identify a resource on the Internet.

[1379] 14. "Purpose-built Application" means an application designed for a specific purpose or function.

[1380] This invention is a system that provides an interactive voice guide application that provides real-time game commentary and player information while watching a baseball game. Hereafter, we will refer to it as the "talking AI baseball commentator application."

[1381] The system begins when a user uses their smartphone camera to read a QR code containing seat-related information. When the user scans the QR code, the device interprets it and automatically launches the retrieved URL or dedicated application. The seat information obtained from the QR code is sent from the device to the server, which then retrieves game data based on the received seat information.

[1382] When the application starts, the device prompts the user through the user interface to select "batter mode" or "pitcher mode." The user can select the mode they are interested in. If the user selects batter mode, the device sends an information request for the next batter to the server. The server collects the next batter's basic information (name, age, alma mater, achievements during school days, performance this season, etc.) from the database and inputs it into the generative AI model.

[1383] The generative AI model used is, for example, OpenAI's GPT-4. An example of a specific prompt is, "Please provide a detailed description of the next batter." The server retrieves the introduction generated from the AI ​​model and sends it back to the device. The device then uses a speech synthesizer (e.g., Google Text-to-Speech) to convert the introduction into audio and play it back to the user.

[1384] Similarly, when the user selects pitcher mode, the device sends a request for information about the current pitcher to the server. The server collects basic information about the current pitcher (such as name, pitch type, and speed) from the database and inputs it into the generative AI model. An example of a prompt sentence is "Please explain the current pitcher in detail." The generated explanation is sent back from the server to the device, converted into speech by a speech synthesizer, and played back to the user.

[1385] Users can also enter questions into a text input field within the app. For example, if a user enters the question, "How many home runs has Ichiro Sato hit?", the device sends the question to the server. The server inputs the question into a generative AI model and generates an appropriate answer. An example of a specific prompt sentence is, "How many home runs has Ichiro Sato hit?" The generated answer is sent back from the server to the device, where it is converted into speech by a speech synthesizer and also displayed as text.

[1386] This allows spectators to hear detailed information about the match and player commentary in real time, deepening their understanding of the match. Even beginners can enjoy the system. This system can significantly improve the experience of watching a match.

[1387] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1388] Step 1: Scan the QR code

[1389] The user uses the smartphone camera to scan a QR code containing seat-related information. The device then uses the smartphone's QR code library (e.g., ZXing) to analyze the QR code and obtain seat information or an application URL. The input is the QR code image, and the output is the seat information or URL. Specifically, the user points the camera at the QR code and presses the scan button to execute this operation.

[1390] Step 2: Launching the Application

[1391] The device accesses the URL obtained from the QR code and automatically launches the talking AI baseball commentator app. If the obtained URL is an installation link for a dedicated app, the device first installs the app and then launches it. The input is the URL, and the output is the launched application. Specifically, the device's browser accesses the URL and the app installation screen is displayed.

[1392] Step 3: Submit seating information

[1393] The device sends the seating information obtained through analysis to the server. At this time, the device uses the HTTP protocol to send the seating information as a POST request to the server's API endpoint. The input is the seating information, and the output is the data sent to the server. Specifically, a piece of software on the device automatically sends the seating information to the server.

[1394] Step 4: Obtaining match data

[1395] Based on the seat information received, the server retrieves the match data related to that seat from a database (e.g., MySQL). The server uses an SQL query to search the database and extract the relevant match data. The input is the seat information and the output is the match data. Specifically, the server sends the SQL query to the database and the results are returned.

[1396] Step 5: Display the mode selection screen

[1397] The device displays buttons on the application's UI to select "batter mode" or "pitcher mode." These buttons are generated using a UI framework (e.g., Flutter). The input is the launched application, and the output is the display of the user interface. Specifically, the mode selection buttons are displayed on the device screen.

[1398] Step 6: User Mode Selection

[1399] The user taps the button for the mode they are interested in. The tap event causes the device to retain information about the selected mode and proceed to the next step. The input is the user's selection action, and the output is information about the selected mode. A specific operation is when the user presses the "batter mode" or "pitcher mode" button.

[1400] Step 7: Request information about the next batter (in batter mode)

[1401] When the user selects "batter mode," the device requests information about the next batter from the server. This is an API request using the HTTP protocol, with the input being the mode selection information and the output being a request to the server. Specifically, the device automatically sends the request.

[1402] Step 8: Gather batter information

[1403] The server collects the next batter's basic information (e.g., name, age, alma mater, achievements during school, and this season's results) from a database. To do this, it searches the database using an SQL query. The input is a batter information request, and the output is the batter's basic information. Specifically, the server executes the SQL query and retrieves the results.

[1404] Step 9: Generative AI model generates introduction text

[1405] The server inputs the collected batter information into a generative AI model (for example, OpenAI's GPT-4). The generative AI model generates a detailed introduction based on the provided information. A specific prompt is "Please provide a detailed description of the next batter." The input is basic information about the batter, and the output is the generated introduction. In concrete terms, the server supplies the batter information to the generative AI model and generates the introduction.

[1406] Step 10: Send back testimonials and play audio

[1407] The server returns the generated introduction to the device. The device converts the introduction into speech using speech synthesis technology (e.g., Google Text-to-Speech) and plays it back. The input is the generated introduction, and the output is audio data. Specifically, the device inputs the introduction into the speech synthesis engine, and the audio is played back from the speaker.

[1408] Step 11: Pitcher Information Request (Pitcher Mode)

[1409] When the user selects "Pitcher Mode," the device requests information about the current pitcher from the server. This is an API request using the HTTP protocol, with the input being the mode selection information and the output being a request to the server. Specifically, the device automatically sends the request.

[1410] Step 12: Gather pitcher information

[1411] The server collects basic information about the current pitcher (such as name, pitch type, speed, etc.) from a database. To do this, it searches the database using an SQL query. The input is a pitcher information request, and the output is the pitcher's basic information. Specifically, the server executes the SQL query and retrieves the results.

[1412] Step 13: Generating explanatory text using a generative AI model

[1413] The server inputs the collected pitcher information into a generative AI model (for example, OpenAI's GPT-4). The generative AI model generates a detailed explanatory text based on the provided information. A specific prompt is "Please provide a detailed explanation of the current pitcher." The input is basic information about the pitcher, and the output is the generated explanatory text. In concrete terms, the server supplies pitcher information to the generative AI model and generates an explanatory text.

[1414] Step 14: Returning commentary and playing audio

[1415] The server returns the generated explanatory text to the device. The device then uses speech synthesis technology (e.g., Google Text-to-Speech) to convert the explanatory text into audio and plays it back. The input is the generated explanatory text, and the output is audio data. Specifically, the device inputs the explanatory text into a speech synthesis engine, and the audio is played back from the speaker.

[1416] Step 15: User Questions and Answers

[1417] The user enters a question into a text input field within the app. For example, if the user enters the question "How many home runs has Ichiro Sato hit?", the input is the user's question, and the device sends this question to the server. The output is the question data sent to the server. In concrete terms, the user enters a question into the text field and presses the send button.

[1418] Step 16: Input to the question generation AI model

[1419] The server provides the received question to the generative AI model and generates an appropriate answer. A specific prompt sentence is, "How many home runs has Ichiro Sato hit?" The input is the user's question, and the output is the generated answer sentence. In concrete terms, the server inputs the question into the generative AI model and generates an answer.

[1420] Step 17: Return and provide answers

[1421] The server returns the generated answer to the terminal. The terminal uses speech synthesis technology to convert the answer into speech and plays it back. At the same time, it also displays the answer as text. The input is the generated answer, and the output is audio data and text data. In concrete terms, the terminal inputs the answer into the speech synthesis engine, and the audio is played back from the speaker while the text is displayed on the screen.

[1422] (Application example 1)

[1423] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1424] In traditional shopping experiences, consumers often find it difficult to obtain detailed product information instantly in physical stores. This is especially true when selecting the perfect product from the numerous products lined up on the shelves, requiring significant effort from consumers, and it is difficult to obtain detailed information without directly asking a store clerk. This often results in an unsatisfactory shopping experience for consumers.

[1425] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1426] In this invention, the server includes: means for reading a code containing information related to a seat; means for providing an application launched by reading the code; means for a user to select a mode in the application; a generative artificial intelligence model that generates detailed information about an object based on the mode selected by the user; a speech synthesizer that provides the information generated by the generative artificial intelligence model in audio; means for answering a question entered by a user in the application; means for generating an answer to the question and providing it in audio and text; means for the application to read a code placed on a product shelf or the like and provide detailed information about the object; means for generating text based on the generative artificial intelligence model for the detailed information, including the product name, price, ingredient information, user reviews, etc., based on the generative artificial intelligence model; and means for providing an audio commentary of the object using a speech synthesizer based on the generated text. This enables consumers to instantly obtain detailed information about products in a physical store, providing an efficient and satisfying shopping experience.

[1427] A "code" is a symbol, either visual or digital, that contains specific information and that can be read and interpreted by electronic equipment.

[1428] An "application" is a software program designed to provide a specific function or service that a user uses on a smartphone or other device.

[1429] A "user" is a person who operates an application or system and receives specific information or services through operations or inputs.

[1430] A "mode" refers to a specific operating state or function set of an application, which can be selected by the user to change the system's behavior or information presentation.

[1431] A "generative artificial intelligence model" is a type of artificial intelligence that has the ability to automatically generate natural language sentences based on given input data.

[1432] A "voice synthesizer" is a device that converts text data into voice and provides information to a user in voice form.

[1433] "Detailed information" refers to specific and in-depth information about the subject matter, including product name, price, ingredient information, user reviews, etc.

[1434] A "product shelf" refers to the area or facility in a physical store where products are displayed and are set up so that consumers can pick up the products.

[1435] "Subject" refers to the specific goods or services that the application provides information about.

[1436] A "question" is a text input that requests additional information or clarification that the user desires from the application related to the current mode or information.

[1437] An "answer" is text or audio containing information generated by a generative artificial intelligence model based on a user's question.

[1438] This invention relates to a voice interactive guide application that provides consumers with real-time product explanations and promotional information in a physical store. The system is activated when a consumer scans a code placed on a product shelf in the store, and provides detailed product information.

[1439] System Configuration

[1440] Hardware

[1441] Smartphone: Camera, speaker, QR code scanner.

[1442] Product shelf: The shelf where the code is installed.

[1443] software

[1444] QR Code Scanner Library: A library for reading QR codes and obtaining product information.

[1445] HTTP request library: A library for retrieving product information from the server (e.g., Requests for Python).

[1446] Generative AI model: An AI model that generates sentences based on input data (e.g., OpenAI's GPT).

[1447] Speech synthesis API: An API for converting generated text into speech (e.g., Google Text-to-Speech).

[1448] Processing Description

[1449] 1. Scan the QR code

[1450] A user uses their smartphone camera to scan a code on a shelf.

[1451] The terminal analyzes the code it reads and obtains the product ID.

[1452] 2. Obtaining product information

[1453] Based on the product ID, the terminal sends an HTTP request to the server to obtain detailed product information.

[1454] The server collects detailed information such as product name, price, ingredient information, and user reviews and sends it back to the device.

[1455] 3. Utilizing generative artificial intelligence models

[1456] The terminal inputs the acquired product information into a generative artificial intelligence model and generates a detailed product description using prompt sentences.

[1457] For example, the prompt text might look like this:

[1458] "Product name: Organic apples. Characteristics: Grown without pesticides, very sweet. Price: 500 yen. Reviews: Popular for their sweetness and deliciousness."

[1459] The generated product description text is returned to the terminal.

[1460] 4. Speech synthesis and information provision

[1461] The device sends the generated product description text to a speech synthesis API to generate voice data.

[1462] The generated voice is used to provide a voice explanation of the product to the user.

[1463] Specific examples

[1464] For example, if a user scans a QR code for organic apples, the device inputs the following prompt text into the generative AI model: "Product name: Organic apples\nFeatures: Grown without pesticides and very sweet\nPrice: 500 yen\nReviews: Rated for their sweetness and deliciousness." Based on this, the AI ​​generates a detailed product description and then converts it into audio commentary using a speech synthesis API. The user can receive a real-time description on their smartphone, such as, "These organic apples are grown without pesticides and are characterized by their sweetness. The price is 500 yen."

[1465] The system allows consumers to instantly obtain detailed product information in-store, providing an efficient and satisfying shopping experience.

[1466] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1467] Step 1:

[1468] A user uses a smartphone camera to scan a code on a product shelf. The input is the code, and the output is the parsed product ID. Specifically, the user activates the camera and performs an operation to read the code.

[1469] Step 2:

[1470] The terminal analyzes the scanned code and obtains the product ID. The input is the scanned code and the output is the product ID. This includes analyzing the code and extracting string information.

[1471] Step 3:

[1472] The terminal sends an HTTP request to the server based on the product ID to obtain detailed product information. The input is the product ID, and the output is detailed product information such as the product name, price, ingredient information, and user reviews. Specific operations include accessing an API containing detailed product information and obtaining data.

[1473] Step 4:

[1474] The server collects detailed information such as product name, price, ingredient information, and user reviews, and sends it back to the terminal. The input is a request based on the product ID, and the output is detailed product information. The server also organizes the collected data and returns it as an HTTP response.

[1475] Step 5:

[1476] The terminal inputs the acquired product information into a generative AI model and generates a detailed product description using a prompt. The input is the product information, and the output is the generated product description. Specific operations include constructing a prompt and sending it to the AI ​​model to generate a sentence.

[1477] Step 6:

[1478] The device sends the generated product description to a speech synthesis API to generate voice data. The input is the generated product description, and the output is voice data. Specific operations include converting the generated text into voice data.

[1479] Step 7:

[1480] The device uses the generated audio data to provide the user with an audio description of the product. The input is the audio data and the output is the audio description. An operation of playing the audio using the smartphone speaker is included.

[1481] By performing these steps in succession, users can instantly obtain detailed information about products by voice in a physical store.

[1482] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1483] MODE FOR CARRYING OUT THE INVENTION

[1484] This invention relates to a system that combines an emotion engine that recognizes the user's emotions with a voice-activated interactive guide application that provides real-time game commentary and player information during baseball games. Hereinafter, this system will be referred to as the "emotion-recognition AI baseball commentator app."

[1485] 1. Scan the QR code and launch the app

[1486] First, when a user sits down at a baseball stadium, they use their smartphone camera to scan the QR code attached to the seat. The device then interprets the QR code and launches a URL or dedicated application. The device then transmits the seat information obtained from the QR code to the server, which then uses this information to retrieve relevant game data.

[1487] 2. Mode Selection

[1488] When the application starts, the device displays a UI that prompts the user to select either "batter mode" or "pitcher mode." The user selects the mode they are interested in.

[1489] 3. Batter mode processing

[1490] 3.1 Introduction of the next batter

[1491] When the user selects batter mode, the device sends a request for information about the next batter to the server. The server collects basic information about the next batter (such as name, age, alma mater, school achievements, and current season results) and inputs this information into a generative AI model. The generative AI model then generates a detailed introduction based on this information.

[1492] The server then sends the generated introduction back to the device, which uses a speech synthesizer to play the information aloud, and the device also uses an emotion engine to monitor the user's reactions and adjust the tone and content of the introduction according to the user's emotions.

[1493] For example, if the user selects batter mode and the next batter is named "Ichiro Sato," the device will provide audio commentary such as, "The next batter is Ichiro Sato. He is 23 years old and graduated from the University of Tokyo. He played in the university championships as a student. His batting average this season is .300." If the emotion engine recognizes that the user is excited, it can make the tone of the commentary more energetic.

[1494] 4. Pitcher mode processing

[1495] 4.1 Pitching commentary

[1496] When a user selects pitcher mode, the device sends a request for information about the current pitcher to the server. The server collects basic information about the current pitcher (such as name, pitch type, and speed) and inputs this information into a generative AI model. The generative AI model then generates a detailed commentary based on that information.

[1497] The server then sends the generated commentary back to the device, which uses a speech synthesizer to play the information aloud, and the device also uses an emotion engine to monitor the user's reactions and adjust the tone and content of the commentary according to the user's emotions.

[1498] For example, if the user selects pitcher mode and the current pitcher is named "Takahashi Jiro," the device will provide a voice commentary such as, "The current pitcher is Takahashi Jiro. He throws a fastball with a speed of 150 km / h." If the emotion engine determines that the user is calm, the commentary can be more detailed and educational.

[1499] 5. User Questions and Answers

[1500] Users can enter questions into a text input field within the app. For example, if they enter a question like, "How many home runs has Ichiro Sato hit?", the device sends the question to the server. The server provides the question to a generative artificial intelligence model, which generates an appropriate answer.

[1501] The server sends the generated answer back to the device, which uses a speech synthesizer to provide the answer aloud and simultaneously displays the answer as text.The device also uses an emotion engine to monitor the user's reaction and provide the answer in a tone that corresponds to the user's emotion.

[1502] For example, in response to the question, "How many home runs has Ichiro Sato hit?", the device can provide a voice commentary such as, "Ichiro Sato has hit 10 home runs this season." If the emotion engine determines that the user is happy, it can respond by brightening the tone.

[1503] This method allows spectators to obtain detailed information about the game and players in real time, while receiving commentary that is tailored to their own emotions. This is expected to deepen their understanding of the game and increase their enjoyment of watching it.

[1504] The processing flow will be explained below.

[1505] Scan the QR code and launch the app

[1506] Step 1:

[1507] The user scans the QR code in front of the seat with their smartphone.

[1508] Step 2:

[1509] The device analyzes the QR code and launches a dedicated application based on the information obtained.

[1510] Step 3:

[1511] The terminal sends the seat information read from the QR code to the server.

[1512] Step 4:

[1513] The server receives the seat information and retrieves the game data associated with the seat.

[1514] Mode Selection

[1515] Step 5:

[1516] The device displays a UI that allows the user to select "batter mode" or "pitcher mode."

[1517] Step 6:

[1518] The user selects either "batter mode" or "pitcher mode" on the UI.

[1519] Batter mode processing

[1520] Step 7:

[1521] The terminal sends a request for information on the next batter to the server.

[1522] Step 8:

[1523] The server obtains basic information about the next batter (such as name, age, alma mater, achievements during school, and current season's results) and provides it to the generative artificial intelligence model.

[1524] Step 9:

[1525] A generative AI model generates a detailed introduction based on basic information.

[1526] Step 10:

[1527] The server returns the generated introduction to the terminal.

[1528] Step 11:

[1529] The device uses a speech synthesizer to play the detailed introduction aloud.

[1530] Step 12:

[1531] The device uses an emotion engine to monitor the user's reactions and adjust the tone and content of the introduction depending on the user's emotions.

[1532] Pitcher mode processing

[1533] Step 13:

[1534] The terminal sends a request to the server for information on the current pitcher.

[1535] Step 14:

[1536] The server obtains basic information about the current pitcher (such as name, pitch type, and pitch speed) and provides it to the generative artificial intelligence model.

[1537] Step 15:

[1538] A generative AI model generates detailed explanatory text based on basic information.

[1539] Step 16:

[1540] The server returns the generated commentary to the terminal.

[1541] Step 17:

[1542] The device uses a speech synthesizer to play the detailed description aloud.

[1543] Step 18:

[1544] The device uses an emotion engine to monitor the user's reactions and adjust the tone and content of the commentary according to the user's emotions.

[1545] User Questions and Answers

[1546] Step 19:

[1547] The user types a question into the app's text input field.

[1548] Step 20:

[1549] The terminal sends a question to the server.

[1550] Step 21:

[1551] The server provides the question to a generative artificial intelligence model, which generates an appropriate answer.

[1552] Step 22:

[1553] The server generates a response and sends it back to the terminal.

[1554] Step 23:

[1555] The device uses a speech synthesizer to play the generated answer aloud and simultaneously displays the answer as text.

[1556] Step 24:

[1557] The device uses an emotion engine to monitor the user's reactions and provides answers in a tone that corresponds to the user's emotions.

[1558] This allows spectators to receive detailed information about the game and player commentary in real time via audio, and they can also receive the most appropriate commentary based on their own emotions.

[1559] Example 2

[1560] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1561] In modern sports viewing, especially baseball, there is a demand for detailed game commentary and player information in real time. However, conventional systems often provide only one-way information without considering the user's emotions, and improvements are needed to increase spectator satisfaction. In particular, a challenge is to develop technology that dynamically adjusts the tone and content of information provided according to the user's emotions.

[1562] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for reading a code including seat-related information; means for providing an application launched by reading the code; means for a user to select a mode in the application; a generative AI model that generates detailed information about a batter or pitcher based on the mode selected by the user; a voice synthesizer that provides the information generated by the generative AI model as audio; an emotion engine that analyzes the user's facial expressions and tone of voice to recognize emotions; means for adjusting the tone and content of audio commentary based on the analysis results of the emotion engine; means for answering questions entered by the user in the application; and means for generating answers to the questions and providing them in audio and text. This makes it possible to provide personalized real-time game commentary and player information according to the user's emotions.

[1563] A "code containing information related to the seat" is a code containing information such as the seat position and number where the user is seated, and includes QR codes and barcodes.

[1564] "Means for reading the code" refers to a device, such as a QR code reader or barcode scanner, or an application that is used to interpret the information in the code.

[1565] "Means for providing applications" refers to applications that run on smartphones or tablets and have the ability to start automatically by loading code.

[1566] "Means for the user to select the mode" refers to a user interface (UI) that provides the user with the ability to select "batter mode" or "pitcher mode" using on-screen buttons or menus.

[1567] A "generative artificial intelligence model" refers to a generative AI based on natural language processing, for example, which generates appropriate sentences and explanations based on input information.

[1568] A "voice synthesizer" refers to a device or software that converts text data into voice data and plays it back.

[1569] An "emotion engine" refers to a device or software that analyzes a user's facial expressions and tone of voice to recognize their emotional state at that time.

[1570] A "means for answering questions" is a function that generates appropriate answers to questions entered by users and provides them in voice or text, and often uses a generative AI model.

[1571] The "means for adjusting tone and content" provides a function for dynamically changing the tone and amount of detailed information in the generated commentary or introduction based on the analysis results of the emotion engine.

[1572] This invention is implemented as an "emotion-recognition AI baseball commentator app." The following is a detailed explanation of the system, including the hardware and software used to perform each process.

[1573] Hardware and Software

[1574] This system uses the following hardware and software:

[1575] Smartphone: A mobile device held by the user that has a camera to read QR codes, runs applications, inputs voice using a microphone, and outputs voice using a speaker.

[1576] QR Code Reader: QR code analysis software that uses a smartphone camera.

[1577] Server: Retrieves match data, processes specified API endpoints, runs generative AI models, and processes user requests.

[1578] Generative AI model: An artificial intelligence used to generate detailed information and explanatory text (e.g., GPT-3).

[1579] Speech synthesizer: A TTS (Text-to-Speech) engine that converts text data into voice data.

[1580] Emotion engine: Software that analyzes the user's facial expressions and tone of voice in real time.

[1581] System Operation

[1582] Scan the QR code and launch the app

[1583] When a user sits down at a baseball stadium, they use their smartphone camera to scan the QR code installed in front of their seat. The QR code reader analyzes the code and automatically launches a URL or dedicated application based on the results. The device then sends the seat information obtained from the QR code to the server, which then retrieves the relevant game data.

[1584] Mode Selection

[1585] When the application starts, the device displays a UI that prompts the user to select either "batter mode" or "pitcher mode." The user selects the mode they are interested in by tapping a button on the screen.

[1586] Batter mode processing

[1587] When the user selects batter mode, the device sends a request for information about the next batter to the server. The server collects basic information about the next batter from the current batting order and creates a prompt to input to the generative AI model. The generative AI model generates a detailed introduction based on the sent prompt. The generated introduction is sent back to the device by the server. The device uses a speech synthesizer to play the introduction as audio.

[1588] Next, the device uses an emotion engine to analyze the user's facial expressions and tone of voice, monitoring emotions in real time. If the user is excited, the tone of the commentary can be adjusted to be more energetic. For example, if the next batter's name is "Yamada Taro," the prompt would be "Tell me about Yamada Taro. He's 25 years old and graduated from University A. He played in the university championships as a student. His batting average this season is .320." This text would be input into the generative AI model.

[1589] Pitcher mode processing

[1590] When the user selects pitcher mode, the device sends a request for information about the current pitcher to the server. The server collects the current pitcher information and creates a prompt to input to the generative AI model. The generative AI model generates a detailed commentary based on the sent prompt. The generated commentary is sent back to the device by the server. The device uses a speech synthesizer to play the commentary as audio.

[1591] The device then uses an emotion engine to analyze the user's facial expressions and tone of voice to monitor their emotions in real time. If the user is calm, the commentary can be more detailed and educational. For example, if the current pitcher's name is "Jiro Nakamura," the prompt would be "Please describe pitcher Jiro Nakamura. He throws a straight ball with a speed of 145 km / h," and the text would be input into the generative AI model.

[1592] User Questions and Answers

[1593] The user enters a question into a text input field within the app and taps the submit button. The device sends the question to the server, which provides it as a prompt to the generative AI model to generate an appropriate answer. The generated answer is then sent back to the device by the server, and the device uses a speech synthesizer to provide the answer as voice and simultaneously displays the answer as text.

[1594] The device uses an emotion engine to analyze the user's facial expressions and tone of voice, monitoring emotions in real time. Based on the analysis results of the emotion engine, it can provide answers in a tone that corresponds to the user's emotions. For example, if a question such as "Please tell me the schedule for the next game" is entered, the prompt text "Please tell me the schedule for the next game" will be input into the generative AI model.

[1595] In this way, we can provide real-time game commentary and player information based on the user's emotions. This system is expected to make baseball watching more engaging and enjoyable.

[1596] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1597] Step 1:

[1598] A user sits in a seat at a baseball stadium and scans a QR code with a smartphone camera. The input is an image of the QR code, and the output is a URL containing seat information or the launch of a dedicated application. The QR code reader analyzes the image and extracts the seat information.

[1599] Step 2:

[1600] The device sends the seating information obtained from the QR code to the server. The input here is the seating information analyzed from the QR code, and the output is the seating information sent to the server. The device sends this information to the server as an HTTP request and obtains the access key for the match data.

[1601] Step 3:

[1602] The server retrieves data on the currently playing game from the database based on the seating information. The input is the seating information received from the terminal, and the output is the game data. The server searches for related game data based on the seating information and retrieves the necessary information.

[1603] Step 4:

[1604] The device displays a UI on the application's home screen that allows the user to select "batter mode" or "pitcher mode." The input is the game data received from the server, and the output is the mode selection screen displayed on the user interface. The device dynamically renders the UI components.

[1605] Step 5:

[1606] The user taps a button on the screen to select a mode. The input is the user's tap, and the output is the selected mode ("batter mode" or "pitcher mode").

[1607] Step 6:

[1608] When the user selects "batter mode," the terminal sends a request for information about the next batter to the server. The input is the selected "batter mode," and the output is an information request to the server. The terminal sends an HTTP request to the server, requesting information about the next batter.

[1609] Step 7:

[1610] The server collects basic information about the next batter and creates a prompt to input into the generative AI model. The input is the next batter's basic information (e.g., name, age, alma mater, grades), and the output is the prompt to input into the generative AI model. The server generates a detailed introduction based on this.

[1611] Step 8:

[1612] The generative AI model generates a detailed introduction. The input is the prompt sent from the server, and the output is a detailed introduction. The generative AI model uses an NLP algorithm to generate a sentence based on the prompt.

[1613] Step 9:

[1614] The server returns the generated introductory text to the device. The input is the introductory text generated by the generative AI model, and the output is the introductory text returned to the device. The server returns the introductory text as an HTTP response.

[1615] Step 10:

[1616] The device uses a speech synthesizer to play the introduction aloud. The input is the text data of the introduction received from the server, and the output is audio data. The speech synthesizer converts the text data into audio and plays it back.

[1617] Step 11:

[1618] The device uses an emotion engine to monitor the user's reactions. The input is the user's facial expressions and tone of voice, and the output is the analysis results of the emotion engine. The device uses a camera and microphone to analyze the user's emotions in real time.

[1619] Step 12:

[1620] The device adjusts the tone and content of the introduction based on the user's emotions. The input is the analysis result of the emotion engine, and the output is the adjusted voice. Based on the analysis result, the device appropriately changes the tone and specificity of the introduction from the next time onwards.

[1621] Step 13:

[1622] When the user selects "pitcher mode," the terminal sends a request for information about the current pitcher to the server. The input is the selected "pitcher mode," and the output is the information request to the server. The terminal sends an HTTP request to the server, requesting information about the current pitcher.

[1623] Step 14:

[1624] The server collects basic information about the current pitcher and creates a prompt to input to the generative AI model. The input is the basic information about the current pitcher (e.g., name, pitch type, speed), and the output is the prompt to input to the generative AI model. The server generates a detailed explanation based on this.

[1625] Step 15:

[1626] The generative AI model generates detailed explanatory text. The input is the prompt sent from the server, and the output is the detailed explanatory text. The generative AI model uses an NLP algorithm to generate text based on the prompt.

[1627] Step 16:

[1628] The server returns the generated commentary to the device. The input is the commentary generated by the generative AI model, and the output is the commentary returned to the device. The server returns the commentary as an HTTP response.

[1629] Step 17:

[1630] The device uses a speech synthesizer to play the explanatory text aloud. The input is the text data of the explanatory text received from the server, and the output is audio data. The speech synthesizer converts the text data into audio and plays it back.

[1631] Step 18:

[1632] The device uses an emotion engine to monitor the user's reactions. The input is the user's facial expressions and tone of voice, and the output is the analysis results of the emotion engine. The device uses a camera and microphone to analyze the user's emotions in real time.

[1633] Step 19:

[1634] The device adjusts the tone and content of the commentary based on the user's emotions. The input is the analysis result of the emotion engine, and the output is the adjusted voice. Based on the analysis result, the device appropriately changes the tone and specificity of the commentary from the next time onwards.

[1635] Step 20:

[1636] The user enters a question into a text input field in the app and taps the submit button. The input is the question entered by the user, and the output is what the device sends to the server.

[1637] Step 21:

[1638] The terminal sends a question to the server. The input is the question entered by the user, and the output is the question sent to the server.

[1639] Step 22:

[1640] The server provides the generative AI model with a question as a prompt, and it generates an appropriate answer. The input is the prompt sentence of the user question, and the output is the generated answer.

[1641] Step 23:

[1642] The server sends the generated answer back to the device. The input is the answer generated by the generative AI model, and the output is the answer sent back to the device.

[1643] Step 24:

[1644] The terminal uses a speech synthesizer to provide a voice response. The input is the text data of the response received from the server, and the output is voice data. The speech synthesizer converts the text data into voice and plays it back.

[1645] Step 25:

[1646] The device simultaneously displays the answer as text on the UI. The input is the text data of the answer received from the server, and the output is the text displayed on the screen.

[1647] Step 26:

[1648] The device uses an emotion engine to monitor the user's reactions. The input is the user's facial expressions and tone of voice, and the output is the analysis results of the emotion engine. The device uses a camera and microphone to analyze the user's emotions in real time.

[1649] Step 27:

[1650] The device responds with a tone that reflects the user's emotions. The input is the analysis result of the emotion engine, and the output is an adjusted voice. Based on the analysis result, the device appropriately changes the tone and content of future responses.

[1651] (Application example 2)

[1652] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1653] The virtual shopping experience has the problem that product information is provided uniformly and does not respond to the user's individual emotions and interests, which may reduce the user's motivation to purchase and their satisfaction.

[1654] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1655] In this invention, the server includes: means for reading a code containing seat-related information; means for providing an application launched by reading the code; means for a user to select a mode in the application; a generative AI model for generating detailed information about general information based on the mode selected by the user; a speech synthesizer for providing the information generated by the generative AI model in voice; means for recognizing the user's emotion using an emotion engine and adjusting the tone and content of the information based on the recognized emotion; means for answering questions entered by the user in the application; and means for generating answers to the questions and providing them in voice and text. This makes it possible to provide information according to the user's emotional state, thereby improving the satisfaction of the shopping experience in the virtual store.

[1656] A "code containing information related to seats" is a QR code or barcode that stores information related to the location and characteristics of products in a virtual store and user behavior.

[1657] The "means for providing applications" is a mechanism for automatically launching specific programs or applications on a user's device and enabling specific functions or services.

[1658] "Means for user mode selection" means an interface or button within an application that allows a user to select an operating mode according to their interests and needs.

[1659] A "generative artificial intelligence model" is an artificial intelligence model that generates new information and explanatory text based on input data, and utilizes machine learning and natural language processing technologies.

[1660] A "speech synthesizer" refers to a device or software that converts information generated from text data or generative artificial intelligence models into speech format and provides it to users.

[1661] An "emotion engine" is artificial intelligence or software that analyzes a user's facial expressions, tone of voice, etc. to recognize their emotional state.

[1662] "Means for answering questions" refers to a mechanism or interface for generating appropriate answers to questions entered by a user and providing those answers in voice and text format.

[1663] The present invention is a system for providing a voice interactive assistant with emotion recognition capabilities to enhance the shopping experience in a virtual store. The system operates as follows.

[1664] First, while walking through the virtual store, the user scans the QR code of a product with the camera on their smartphone or smart glasses. The server identifies the product information by reading this QR code.

[1665] Next, the server acquires the product information and inputs it into a generative AI model. The generative AI model generates a detailed explanatory text based on the product information. The generated explanatory text is then converted into voice data using a voice synthesizer.

[1666] Additionally, the device uses a camera to capture the user's facial expressions in real time and recognizes their emotions using an emotion engine. The device adjusts the tone and content of the generated commentary depending on the user's emotional state. This process is powered by the dlib and Emotion Recognition libraries.

[1667] When a user asks a question about a product, the device sends the question as text to the server, which uses a generative artificial intelligence model to generate an answer corresponding to the question and provides it to the user in voice and text.

[1668] For example, if a user scans a QR code for a "smartphone case" in a virtual store, the server retrieves the corresponding product information. Based on that information, a generative AI model generates a description such as, "This smartphone case is shock-resistant and available in a variety of colors," and a speech synthesizer converts it into voice data. If the user is excited, the device can provide a description in an energetic tone, such as, "It's on sale now, so now's your chance!"

[1669] Examples of prompts include:

[1670] What product are you looking at right now?

[1671] Could you tell me more about this product?

[1672] Are there any other products you would recommend?

[1673] This system makes it possible to provide information according to the user's emotional state, thereby improving the satisfaction of the shopping experience in the virtual store.

[1674] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1675] Step 1:

[1676] The terminal reads the QR code scanned by the user using the camera on a smartphone or smart glasses. The input is the QR code, and the output is the product identification information obtained from the QR code. Specifically, the terminal analyzes the QR code from the image data captured by the camera and extracts the product identification information.

[1677] Step 2:

[1678] The server retrieves the corresponding product information from the database based on the product identification information obtained from the QR code. The input is the product identification information, and the output is detailed information such as the product name, description, and price. Specifically, it sends a request to the REST API and retrieves the product information as response data.

[1679] Step 3:

[1680] The server inputs product information into a generative artificial intelligence model to generate detailed descriptions. The input is product information, and the output is detailed descriptions. Specifically, the server feeds product information into a machine learning model to generate natural-looking text.

[1681] Step 4:

[1682] The device uses a speech synthesizer to convert the explanatory text into audio data. The input is a detailed explanatory text, and the output is audio data. Specifically, the device converts the text into audio using a text-to-speech engine (e.g., Pyttsx3).

[1683] Step 5:

[1684] The device uses a camera to capture the user's facial expressions in real time and uses an emotion engine to recognize the user's emotions. The input is real-time video data, and the output is the user's emotional state. Specifically, it uses dlib and the Emotion Recognition library to extract facial features and determine the user's emotional state.

[1685] Step 6:

[1686] The device adjusts the tone and content of the generated commentary based on the user's emotional state. The input is the user's emotional state and commentary, and the output is the adjusted commentary. Specifically, the device applies an algorithm that changes the tone and wording of the commentary depending on the results of the emotion engine.

[1687] Step 7:

[1688] The terminal provides the adjusted commentary to the user as audio data. The input is the adjusted commentary, and the output is audio output to the user. Specifically, the terminal uses the speech synthesizer again to convert the adjusted text into audio and play it through the speaker.

[1689] Step 8:

[1690] When a user asks a question about a product, the device sends the question in text format to the server. The input is the user's question, and the output is the question data in text format. Specifically, the device captures the question entered in the text input field within the app and sends it to the server.

[1691] Step 9:

[1692] The server uses a generative artificial intelligence model to generate an answer corresponding to the question and provides the answer to the user in voice and text. The input is a text question, and the output is voice and text data of the answer. Specifically, the server provides the question to a machine learning model, generates an appropriate answer, and converts the answer into voice data using a speech synthesizer.

[1693] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1694] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1695] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1696] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1697] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1698] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1699] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1700] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1701] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1702] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1703] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1704] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1705] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the spe...

Claims

1. means for reading a code containing information relating to the seat; means for providing an application that is launched by reading the code; means for a user to select a mode in said application; a generative artificial intelligence model that generates detailed information about a batter or pitcher based on the mode selected by the user; a speech synthesis device that provides information generated from the generative artificial intelligence model in the form of speech; means for answering questions input by a user in the application; The system includes means for generating and providing answers to said questions in audio and text.

2. 2. The system according to claim 1, wherein the detailed information of the batter includes the name, age, alma mater, achievements during school days, and performances this season.

3. 2. The system of claim 1, wherein the pitcher details include name, pitch type, and pitch velocity.

Images