system

Abstract

We provide the system. [Solution] In a system that provides information via a user terminal, A means for mobile devices to obtain information about players and the pitch in real time, A method by which a server retrieves player and match information from a database and sends it back to a mobile device, A means by which a user terminal captures images of players and the pitch using its camera function and sends the identification results to a server, A means for the server to acquire information on identified players and areas and send it back to the mobile device, A means by which a user terminal receives questions in a chat format and sends those questions to a server, A means by which a server generates an answer to a question using a generative AI and sends it back to the mobile device, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In conventional sports watching, the means for viewers to grasp the detailed information of players and the game situation in real time are limited, and the viewing experience depends on limited information. As a result, especially in stadium viewing and TV viewing, it is difficult for viewers to obtain detailed information about players and games instantaneously, and a visual and information - rich viewing experience has not been provided. In addition, since there is also a lack of means for viewers to solve questions regarding game rules and player achievements in real time, it has been difficult to resolve doubts during viewing on the spot.

Means for Solving the Problems

[0005] To solve the above problems, the present invention proposes a system that provides information via a user terminal. This system includes means for a mobile terminal to acquire information on players and the pitch in real time, means for a server to acquire information on players and matches from a database and send it back to the mobile terminal, means for the user terminal to capture images of players and the pitch using its camera function and send the identification results to the server, means for the server to acquire information on the identified players and areas and send it back to the mobile terminal, means for the user terminal to accept questions in chat format and send the questions to the server, and means for the server to generate answers to the questions using generative AI and send them back to the mobile terminal. As a result, spectators can instantly acquire detailed information on players and matches in real time and enjoy a visually rich and informative viewing experience. In addition, the chat-style question function makes it possible to resolve questions during the viewing on the spot.

[0006] A "user terminal" refers to an electronic device, including a mobile terminal or smartphone, that is operated by a user.

[0007] A "mobile device" refers to a portable electronic device such as a smartphone or tablet.

[0008] A "server" is a computer system that stores data, processes various requests, and provides information to other devices.

[0009] A "database" is a data structure for systematically storing, managing, and retrieving specific information.

[0010] "Match information" refers to detailed information about a specific sporting event, including player lists, scores, opponents, etc.

[0011] "Player information" refers to detailed data about a specific player, including their profile, past performance, and anecdotes.

[0012] The "camera function" is a feature that allows a mobile device to capture video and analyze that video.

[0013] "Identification" refers to recognizing specific players or parts of the field from camera footage and identifying the corresponding information.

[0014] "Chat format" refers to a communication style in which users interact through text-based messages.

[0015] "Generative AI" is an artificial intelligence technology that generates appropriate answers to questions from users.

[0016] "Real-time" means processing and providing information about events and happenings in real time immediately. [Brief explanation of the drawing]

[0017] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10]Shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Mode for Carrying Out the Invention

[0018] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0019] First, the language used in the following description will be explained. [[ID=2,6]]

[0020] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of a plurality of arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of a plurality of types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0021] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0022] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0023] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0025] [First Embodiment]

[0026] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0027] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0030] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0033] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0037] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0038] This invention relates to a system that provides real-time information about watching sports using a user terminal, a server, and a database.

[0039] System Overview

[0040] This system aims to enhance the viewing experience by allowing users to obtain match and player information in real time using their mobile devices. The overall system configuration and operation are described below.

[0041] User terminal operation

[0042] A user terminal is a mobile device (smartphone, tablet, etc.) operated by the user, and it runs when an application is launched. An application includes the following functions:

[0043] 1. Match Selection Function

[0044] The user launches the app and selects the match they want to watch from the list of currently ongoing matches displayed on the home screen. The selected match information is then requested from the server.

[0045] 2. Camera function

[0046] Users activate the camera function and capture video by pointing their mobile device at the field or players. The captured video is analyzed, and information about identified players and areas is sent to the server.

[0047] 3. Information display function

[0048] Player and match information sent from the server is overlaid on the terminal screen in real time. This allows users to instantly check detailed information about players and matches while watching.

[0049] 4. Chat function

[0050] Users enter their questions in text format using the in-app chat function. The entered questions are sent to the server, where a generative AI generates appropriate answers.

[0051] Server operation

[0052] The server works in conjunction with the database to manage large amounts of match and player information, and is responsible for providing this information in response to requests from user terminals. Its specific operations are as follows:

[0053] 1. Providing match information

[0054] When the server receives a request to select a match from the user's terminal, it retrieves the relevant match information from the database and sends it back to the user's terminal.

[0055] 2. Analysis of camera footage

[0056] The system analyzes camera footage transmitted from the user's terminal to identify players and specific areas of the pitch. Based on this identification, it retrieves relevant player and area information from a database and sends it back to the terminal.

[0057] 3. Responses by generative AI

[0058] The system uses a generative AI to analyze questions sent by users through the chat function and generate answers. The generated answers are supplemented with necessary information retrieved from a database and then sent back to the user's device.

[0059] Specific operation examples

[0060] For example, when a user launches the smartphone app and selects "Match A" from the "Match List" screen, the following sequence of events occurs:

[0061] 1. The user requests information about the selected match from the server. The server retrieves information about match A from the database and sends it to the user's terminal.

[0062] 2. When a user points their camera at "Player X" while watching a match, the captured video is sent to the server. The server analyzes the video and identifies Player X. Player X's profile, statistics, etc., are retrieved from the database and sent back to the user's terminal.

[0063] 3. When a user asks a question via the chat function, such as "Please tell me the statistics for player X," the question is sent to the server. The generative AI analyzes the question and generates a relevant answer (for example, "Player X's batting average this season is .300"), which is then sent back to the user's terminal.

[0064] In this way, the system provides users with the information they need while watching a game in real time, resulting in a richer viewing experience.

[0065] The following describes the processing flow.

[0066] Step 1:

[0067] The user launches the app on their mobile device. The app's home screen is displayed, showing a list of currently ongoing matches.

[0068] Step 2:

[0069] The user selects the match they want to watch. A request containing information about the selected match is sent to the server.

[0070] Step 3:

[0071] The server receives the request and retrieves the relevant match information from the database. The retrieved match information is then sent back from the server to the user's terminal.

[0072] Step 4:

[0073] The user's terminal displays match information received on the screen. The displayed information includes the player list, match score, and opponent.

[0074] Step 5:

[0075] The user activates the camera function and points their mobile device at the field or players. The video captured by the camera function is analyzed on the device.

[0076] Step 6:

[0077] The device identifies players and areas from camera footage and sends the identification results to the server. The identification results include the player's location information and identification ID.

[0078] Step 7:

[0079] The server receives the identification result and retrieves the corresponding player and area information from the database. The retrieved information is then sent back from the server to the user's terminal.

[0080] Step 8:

[0081] The system overlays information received by the user's device onto the camera feed. This allows users to view detailed information about players and areas in real time.

[0082] Step 9:

[0083] Users enter questions using the chat function. For example, a question like "What is player X's batting average this season?" might be entered in chat format.

[0084] Step 10:

[0085] The terminal sends the entered question to the server. Upon receiving the question, the server uses a generative AI to generate an answer.

[0086] Step 11:

[0087] The server sends the generated response to the user's terminal. The generated response contains the necessary information, including details such as, for example, "Player X's batting average this season is .300."

[0088] Step 12:

[0089] The system displays the answers received by the user's device on the chat screen. This allows the user to see the answers to their questions in real time.

[0090] (Example 1)

[0091] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0092] Traditional sports viewing systems made it difficult for users to easily obtain detailed match and player information in real time. Furthermore, there was a lack of intuitive means for users to seek additional information to gain a deeper understanding of the game. Additionally, systems for providing appropriate answers in real time via chat were not adequately developed.

[0093] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0094] In this invention, the server includes means for a mobile terminal to acquire information about matches and players in real time; means for the server to acquire information about matches and players from a database and send it back to the mobile terminal; means for the user terminal to capture video of matches and players using its camera function and send the identification results to the server; means for the server to acquire information about identified players and areas and send it back to the mobile terminal; means for the user terminal to accept questions in chat format and send the questions to the server; means for the server to generate answers to the questions using a generative AI model and send them back to the mobile terminal; and means for generating prompt sentences to be input to the generative AI model. As a result, users can intuitively acquire detailed information about matches and players in real time and have a richer viewing experience.

[0095] A "user terminal" is a mobile device operated by a user, which runs applications and displays information about matches and players.

[0096] A "mobile device" refers to a portable electronic device such as a smartphone or tablet.

[0097] A "server" is a computer system that manages information in conjunction with a database and processes requests from user terminals.

[0098] A "database" is a data management system that stores match information and player information, and allows you to retrieve it as needed.

[0099] "Capturing" refers to the act of capturing video using the camera's functions.

[0100] "Identification results" refer to information about players and the field identified through the analysis of captured video footage.

[0101] A "generative AI model" is an artificial intelligence model that analyzes an input question and generates an appropriate answer.

[0102] A "prompt message" is a string of characters input into a generative AI model, containing instructions for the model to generate an answer.

[0103] "Overlay display" is a format that displays additional information on top of an existing screen.

[0104] A "chat format" is an interface format where questions and answers are conducted using text.

[0105] "Live data" refers to data generated in real time during the progress of a match.

[0106] This invention relates to a system that allows users to obtain real-time information about matches and players using their mobile devices, thereby providing a richer viewing experience. This system operates by combining user terminals, a server, and a database.

[0107] Specifically, the user's device is a mobile device such as a smartphone or tablet, with the application installed. This application includes a match selection function, camera function, information display function, and chat function. When the user launches the app, a list of currently ongoing matches is displayed on the home screen. The user selects the match they want to watch, and the match information is requested from the server.

[0108] The server maintains a large amount of match and player information in its database. When a match selection request is sent from a user terminal, the server retrieves the relevant match information from the database and sends it to the user terminal. Also, when a user uses the camera function to capture video of a match or players, the video is sent to the server. The server analyzes the video, retrieves information on identified players and fields from the database, and sends it back to the user terminal.

[0109] Furthermore, the user terminal can overlay the received information on the screen. When a user enters a question in text format using the chat function, the question is sent to the server. The server analyzes the question using a generative AI model and generates an appropriate answer. The generated answer, along with supplementary information retrieved from the database, is sent back to the user terminal.

[0110] For example, a user launches the app on their smartphone and selects "Match A" on the home screen. At this point, the server retrieves detailed information about the selected Match A and sends it to the user's device. If the user points their camera at "Player X" while watching the match, the captured video is sent to the server, which analyzes the video to identify Player X. Player X's profile and performance information are retrieved from the database and sent back to the user's device. Furthermore, if the user asks a question using the chat function, such as "Please tell me Player X's performance," a generative AI model analyzes the question, generates an answer such as "Player X's batting average this season is .300," and sends it to the user's device.

[0111] Examples of prompt statements include the following:

[0112] "Please tell me about match A."

[0113] "I'd like to know player X's performance this season."

[0114] "What is the current score for Match A?"

[0115] This allows users to intuitively operate the system while obtaining a wealth of information in real time, greatly improving the viewing experience. Furthermore, the introduction of generative AI models enables instant responses to chat-style questions, increasing user satisfaction.

[0116] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0117] Step 1:

[0118] The user launches the application on their mobile device. The input is the user's action (tap), and the output is the display of the application's home screen. Specifically, the application refers to an internal database, retrieves a list of currently ongoing matches, and displays it on the home screen.

[0119] Step 2:

[0120] The user selects the match they want to watch. The input is the user's selection of a match, and the output is a request for information about the selected match sent to the server. Specifically, the application sends a request to the server that includes the ID of the selected match.

[0121] Step 3:

[0122] The server processes the received request and retrieves the relevant match information from the database. The input is the match ID sent from the user's terminal, and the output is detailed information about the relevant match sent back to the user's terminal. Specifically, the server searches the database using the match ID as the key, retrieves the relevant match information (score, player list, etc.), and sends it back to the user's terminal.

[0123] Step 4:

[0124] The user activates the camera function on their device to capture video of the match or players. The input is the user's camera activation operation, and the output is the captured video sent to the server. Specifically, the user's device sends the video data acquired from the camera to the server in real time.

[0125] Step 5:

[0126] The server analyzes the received video to identify players and specific areas of the field. The input is captured video data, and the output generates information about the identified players and areas. Specifically, the server uses image recognition technology to analyze the video, retrieves the identified players and areas from a database, and sends it back to the user terminal.

[0127] Step 6:

[0128] The user terminal overlays the received information onto the camera feed. The input is player and area information sent from the server, and the output is the information displayed overlaid on the camera feed. Specifically, the user terminal analyzes the received data and displays player names and profiles on top of the camera feed in real time.

[0129] Step 7:

[0130] The user enters a question using the chat function. The input is the user's text input, and the output is the entered question sent to the server. Specifically, the user's terminal receives the user's input on the chat screen and sends that text to the server.

[0131] Step 8:

[0132] The server uses a generative AI model to analyze the question and generate an appropriate answer. The input is the question sent by the user, and the generated answer is sent back to the user's terminal as output. Specifically, the server inputs the question into the AI ​​model, retrieves supplementary information from the database based on the analysis results to generate an answer, and sends it to the user's terminal.

[0133] Step 9:

[0134] The user's terminal displays the response from the server on the chat screen. The input is the response data sent from the server, and the output is the response displayed on the user's chat screen. Specifically, the user's terminal analyzes the received response and displays it on the chat screen so that the user can confirm the response.

[0135] (Application Example 1)

[0136] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0137] In modern sports viewing, users desire more detailed and real-time information about matches and players. However, current systems lack the means to achieve this quickly and effectively. In particular, there is a need for systems that offer advanced interactive features such as eye-tracking, real-time information overlays, and question answering using generative models.

[0138] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0139] In this invention, the server includes means for the server to retrieve player and match information from a database and send it back to the mobile terminal; means for the user terminal to capture images of players and the field using its camera function and send the identification results to the server; and means for the server to generate answers to questions using a generative model and send them back to the mobile terminal. This allows the user to identify the player or area they are focusing on using an eye-tracking camera and obtain detailed information in real time. Furthermore, by utilizing a generative model, accurate answers to the user's questions can be provided quickly. In addition, because player information is overlaid on the video, the user can obtain visually intuitive information, improving the viewing experience.

[0140] A "user terminal" refers to a device operated by a user in a sports viewing system, such as a smartphone or tablet.

[0141] A "mobile device" is an electronic device that a user can carry with them, and includes smartphones, tablets, and other similar devices.

[0142] "Players and field" refers to the players and the areas within the stadium where their activities take place during a sports match.

[0143] A "server" is a central system for managing and processing information, and its role is to retrieve information from a database and provide it to user terminals.

[0144] A "database" refers to a collection of data used to store and manage information such as match details and player information in a searchable format.

[0145] The "camera function" refers to a function for capturing video, specifically the ability to photograph players and the field using the camera built into the mobile device.

[0146] "Identification results" refer to data that includes information about identified players and areas, obtained by analyzing video footage captured by cameras.

[0147] A "generative model" refers to an artificial intelligence model that has been trained using pre-trained data, and is designed to generate appropriate answers to user questions.

[0148] A "gaze recognition camera" refers to a camera function that tracks the user's gaze and identifies the object the user is focusing on.

[0149] "Overlay display" refers to a technique of displaying additional information superimposed on top of video footage, specifically displaying player information and match information in real time on video during a game.

[0150] This invention relates to a system that provides real-time information about sports viewing using a user terminal, a server, and a database.

[0151] User terminal operation

[0152] The user terminal is a mobile device operated by the user (e.g., a smartphone or tablet), and it runs when an application is launched. The application includes the following functions:

[0153] 1. Match Selection Function

[0154] The user launches the app and selects a match they want to watch from a list of matches displayed on a virtual screen. The selected match information is then requested from the server.

[0155] 2. Camera function

[0156] The system uses a camera mounted on the user's terminal to capture video of the players and the field. The captured video is analyzed, and the identification results are sent to the server.

[0157] 3. Eye-tracking camera function

[0158] The user's device has a built-in eye-tracking camera that tracks the user's gaze and identifies the player or area they are focusing on.

[0159] 4. Information Overlay Display Function

[0160] Player and match information sent from the server is overlaid on the user's terminal screen in real time. This function allows users to intuitively check detailed information about players and matches while watching.

[0161] 5. Chat function

[0162] Users input questions by voice using the in-app chat function. The entered questions are sent to the server, and a generative model generates appropriate answers.

[0163] Server operation

[0164] The server works in conjunction with the database to manage large amounts of match and player information, and is responsible for providing this information in response to requests from user terminals. Its specific operations are as follows:

[0165] 1. Providing match information

[0166] The server receives a match selection request from the user's terminal, retrieves the relevant match information from the database, and sends it to the user's terminal.

[0167] 2. Analysis of camera footage

[0168] The server analyzes camera footage transmitted from the user's terminal to identify players and specific areas of the field. Based on this identification, it retrieves relevant player and area information from the database and sends it back to the user's terminal.

[0169] 3. Analysis of gaze recognition

[0170] The server uses an eye-tracking camera to identify the player or area the user is focusing on and sends that information to the user's terminal.

[0171] 4. Response using a generative model

[0172] The system analyzes questions submitted by users through the chat function using a generative model and generates answers. The generated answers are supplemented with necessary information retrieved from the database and sent back to the user's device.

[0173] Hardware and software to be used

[0174] User devices: Smartphones, tablets, and eye-tracking cameras

[0175] Server: Database management system, generative model

[0176] Software: OpenCV (camera image analysis), requests (communication with server)

[0177] Specific example

[0178] For example, a user launches a smartphone app and selects "Match A" from the "Match List" screen. Then, when they fix their gaze on "Player X" using the eye-tracking camera, Player X's profile and performance information are displayed as an overlay. Additionally, if the user asks a voice question such as "Please tell me Player X's performance," that performance information will be displayed on the screen.

[0179] Example of a prompt

[0180] "Please tell me the statistics of player X."

[0181] "Please tell me the progress of the match."

[0182] "What is the status of player Y's injury?"

[0183] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0184] Step 1:

[0185] (Input) The user launches a sports viewing application on their smartphone and displays the match list screen.

[0186] (Specific action) The user accesses the application's home screen, and a list of currently ongoing matches is displayed.

[0187] (Data processing) Match information is requested from the server, the server retrieves that information from the database, and sends it to the user's terminal as a list of matches.

[0188] (Output) The match list is displayed on the smartphone screen.

[0189] Step 2:

[0190] (Input) The user selects the match they want to watch (for example, "Match A").

[0191] (Specific action) The user selects "Match A" from the match list by tapping or clicking.

[0192] (Data Processing) The selected match information is requested from the server, and the server retrieves that match information from the database and sends it to the user's terminal as detailed information.

[0193] (Output) Detailed information about "Match A" is displayed on the smartphone screen.

[0194] Step 3:

[0195] (Input) The user activates the camera function and captures the players or field with the camera.

[0196] (Specific operation) The smartphone camera captures video of the players and the field in real time.

[0197] (Data processing) The captured video is sent from the user's terminal to the server, which analyzes the video to identify specific parts of the players or field.

[0198] (Output) The identification result is sent from the server to the user terminal.

[0199] Step 4:

[0200] (Input) The user uses eye-tracking technology to focus on a specific player or area.

[0201] (Specific operation) Eye-tracking cameras installed in smartphones or HMDs track the user's gaze and identify the player or area they are focusing on.

[0202] (Data Processing) Eye-tracking data is sent from the user terminal to the server, and the server uses that data to retrieve information about identified players and areas from its database and sends it back to the user terminal.

[0203] (Output) Detailed information about the player or area being watched will be overlaid on the smartphone screen.

[0204] Step 5:

[0205] (Input) The user asks a question using the voice input function (for example, "Please tell me the statistics of player X").

[0206] (Specific action) The user activates the app's voice input function and inputs the question by voice.

[0207] (Data processing) The audio data is converted into text by the speech recognition function, and the question is sent from the user's terminal to the server.

[0208] The output generation model analyzes the question, generates an appropriate answer, and sends it to the user's terminal.

[0209] Step 6:

[0210] (Input) The user receives the response sent from the server and confirms it on the screen.

[0211] (Specific operation) The response sent from the server is displayed on the user's terminal.

[0212] (Data processing) The responses generated by the generative model are displayed on the user's terminal, and supplementary information is provided from the database as needed.

[0213] (Output) The information requested by the user and the answers to the questions are displayed on the screen.

[0214] Step 7:

[0215] (Input) Live status data is sent to the server.

[0216] (Specific operation) The server retrieves real-time live status from the database.

[0217] (Data processing) Live status data is sent to the user's terminal.

[0218] (Output) The live status is displayed in real time on the user's smartphone.

[0219] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0220] This invention relates to a system that provides real-time information about watching sports using a user terminal, a server, a database, and an emotion engine.

[0221] System Overview

[0222] This system aims to acquire match and player information in real time using the user's mobile device, and to provide customized information based on the user's emotions. The overall system configuration and operation are described below.

[0223] User terminal operation

[0224] A user terminal is a mobile device (smartphone, tablet, etc.) operated by the user, and it runs when an application is launched. An application includes the following functions:

[0225] 1. Match Selection Function

[0226] The user launches the app and selects a match they want to watch from the list of currently running matches displayed on the home screen. The selected match information is then requested from the server.

[0227] 2. Camera function

[0228] The user activates the camera function and points their mobile device at the field or players to capture video. The captured video is analyzed, and information about identified players and areas is sent to the server.

[0229] 3. Information display function

[0230] Player and match information sent from the server is overlaid on the terminal screen in real time. This allows users to instantly check detailed information about players and matches while watching.

[0231] 4. Chat function

[0232] Users enter questions in text format using the in-app chat function. The entered questions are sent to the server, where a generative AI generates appropriate answers.

[0233] 5. Emotion recognition function

[0234] The emotion engine recognizes the user's emotions in real time from their facial expressions and voice. The recognized emotion information is sent to the server.

[0235] Server operation

[0236] The server works in conjunction with the database to manage large amounts of match and player information, and is responsible for providing this information in response to requests from user terminals. Its specific operations are as follows:

[0237] 1. Providing match information

[0238] When the server receives a request to select a match from the user's terminal, it retrieves the relevant match information from the database and sends it back to the user's terminal.

[0239] 2. Analysis of camera footage

[0240] The system analyzes camera footage transmitted from the user's terminal to identify players and specific areas of the pitch. Based on this identification, it retrieves relevant player and area information from a database and sends it back to the terminal.

[0241] 3. Responses by generative AI

[0242] The system uses a generative AI to analyze questions sent by users through the chat function and generate answers. The generated answers are supplemented with necessary information retrieved from a database and then sent back to the user's device.

[0243] 4. Processing emotional information

[0244] The system acquires user emotion information recognized by the emotion engine and customizes the information provided to the user's device based on that information. For example, if the user is excited, it provides detailed player statistics, and if the user is relaxed, it provides stories such as past episodes.

[0245] Specific operation examples

[0246] For example, when a user launches the smartphone app and selects "Match A" from the "Match List" screen, the following sequence of events occurs:

[0247] 1. The user requests information about the selected match from the server. The server retrieves information about match A from the database and sends it to the user's terminal.

[0248] 2. When a user points their camera at "Player X" while watching a match, the captured video is sent to the server. The server analyzes the video and identifies Player X. Player X's profile, statistics, etc., are retrieved from the database and sent back to the user's terminal.

[0249] 3. When a user asks a question via the chat function, such as "Please tell me the statistics for player X," the question is sent to the server. The generative AI analyzes the question and generates a relevant answer (for example, "Player X's batting average this season is .300"), which is then sent back to the user's terminal.

[0250] 4. Simultaneously, the emotion engine recognizes the user's emotions from their facial expressions and voice, and displays detailed performance information if the user is excited, or related episodes if they are relaxed.

[0251] In this way, the system meets users' real-time information needs and further enriches the viewing experience by providing customized information based on emotions.

[0252] The following describes the processing flow.

[0253] Step 1:

[0254] The user launches the app on their mobile device. The app's home screen is displayed, showing a list of currently ongoing matches.

[0255] Step 2:

[0256] The user selects the match they want to watch. A request containing information about the selected match is sent to the server.

[0257] Step 3:

[0258] The server receives the request and retrieves the relevant match information from the database. The retrieved match information is then sent back from the server to the user's terminal.

[0259] Step 4:

[0260] The user's terminal displays match information received on the screen. The displayed information includes the player list, match score, and opponent.

[0261] Step 5:

[0262] The user activates the camera function and points their mobile device at the field or players. The video captured by the camera function is analyzed on the device.

[0263] Step 6:

[0264] The device identifies players and areas from camera footage and sends the identification results to the server. The identification results include the player's location information and identification ID.

[0265] Step 7:

[0266] The server receives the identification result and retrieves the corresponding player and area information from the database. The retrieved information is then sent back from the server to the user's terminal.

[0267] Step 8:

[0268] The system overlays information received by the user's device onto the camera feed. This allows users to view detailed information about players and areas in real time.

[0269] Step 9:

[0270] Users enter questions using the chat function. For example, a question like "What is player X's batting average this season?" might be entered in chat format.

[0271] Step 10:

[0272] The terminal sends the entered question to the server. Upon receiving the question, the server uses a generative AI to generate an answer.

[0273] Step 11:

[0274] The server sends the generated answer to the user terminal. The generated answer contains the necessary information, and details such as "Player X's batting average this season is.300" are included as an example.

[0275] Step 12:

[0276] The user terminal displays the received answer on the chat screen. This allows the user to view the answer to the question in real time.

[0277] Step 13:

[0278] The emotion engine recognizes the emotion from the user's expression and voice. For example, it detects emotions such as excitement, joy, and relaxation.

[0279] Step 14:

[0280] The emotion engine sends the recognized emotion information to the server. The server receives the emotion information.

[0281] Step 15:

[0282] The server customizes the information content provided based on the emotion information. For example, when the user is excited, it provides detailed player performance information, and when the user is relaxed, it provides episodes related to the players and games.

[0283] Step 16:

[0284] The server sends the customized information to the user terminal. The user terminal displays the information, and the user can view the customized information corresponding to the emotion.

[0285] Through this series of processes, the user can obtain customized information based on emotions in real time and enjoy a rich viewing experience.

[0286] (Example 2)

[0287] Next, Example 2 will be described. In the following description, the data processing device 12 is referred to as a "server", and the smart device 14 is referred to as a "terminal".

[0288] In a conventional sports viewing system, there is a lack of means for users to obtain game information and player information in real time. Also, since customized information based on the user's emotions is not provided, it is difficult to provide a viewing experience optimized for individual users. Furthermore, when responding to questions from users through the chat function, there is a lack of means to generate quick and accurate answers. To solve these problems, a more advanced and personalized information providing system is required.

[0289] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0290] In this invention, the server includes means for a mobile terminal to obtain information on players and the field in real time, means for the server to obtain information on players and games from a database and send it back to the mobile terminal, means for the user terminal to capture images of players and the field using a camera function and send the identification result to the server, means for the server to generate an answer to a question using a generation AI model and send it back to the mobile terminal, means for the user terminal to analyze the user's expression and voice to obtain emotion information and send the emotion information to the server, and means for the server to generate customized information based on the emotion data and send it back to the mobile terminal. Thereby, the user can obtain game information and player information in real time, and personalized information can be provided according to the user's emotions. Also, since quick and accurate information is provided through the chat function, the viewing experience is improved.

[0291] A "user terminal" is a portable information terminal operated by a user, which acquires and displays information using a specific application.

[0292] A "mobile device" refers to a portable information and communication device such as a smartphone or tablet, which is used to provide users with real-time information on matches and players.

[0293] A "server" is a computer system that communicates with user terminals via a network to process data and provide information.

[0294] A "database" is an information management system that stores match information and player information, and allows a server to retrieve this information as needed.

[0295] The "camera function" is a feature that allows the user's device to take pictures of players and the field, capture video footage, and send that video data to a server for analysis.

[0296] A "generative AI model" is an algorithm or program that uses artificial intelligence to analyze a user's question and generate an appropriate answer.

[0297] An "emotion engine" is a combination of software or hardware that analyzes a user's emotions from their facial expressions and voice and acquires that data.

[0298] "Identification results" refer to information about specific players or parts of the field obtained by analyzing video captured by the camera function.

[0299] "Overlay display" is a technology that displays additional information by superimposing it on the video or image currently displayed on the user's device's screen.

[0300] A "chat-style question" is an inquiry that a user enters in text format and sends to the server.

[0301] "Customized information" refers to information optimized for each user's state based on the user's emotional data.

[0302] This invention relates to a system that provides real-time information on sports viewing using a user terminal, a server, a database, and an emotion engine. This system aims to obtain game information and player information in real time using the user's mobile terminal and further provide customized information based on the user's emotions.

[0303] System Configuration

[0304] Hardware and Software of the User Terminal

[0305] The user terminal uses a mobile terminal such as a smartphone or a tablet. A dedicated application is installed on this terminal, which includes functions such as game selection, camera function, chat function, and emotion recognition function.

[0306] Hardware and Software of the Server

[0307] The server uses a high-performance network server to manage a large amount of data. It cooperates with a database that stores player information and game information and responds to requests from users. A generation AI model and an emotion engine are implemented on the server.

[0308] Database

[0309] The database is responsible for storing and managing game information and player information. This includes player profiles, performance information, game progress, etc.

[0310] Emotion Engine

[0311] An emotion engine is software or hardware that analyzes a user's facial expressions and voice to recognize their emotional state in real time. Its role is to acquire the user's emotional data and transmit it to a server.

[0312] Program processing

[0313] Match selection function

[0314] The user launches the application and selects a match they want to watch from the match list. The selected match information is requested from the server. The server retrieves the corresponding match information from the database and sends it back to the user's terminal.

[0315] Camera function

[0316] The user activates the camera function and points their mobile device at the field or players to capture video. The captured video is sent to a server, where it is analyzed. Based on the analysis results, the server retrieves information on identified players and areas from its database and sends it back to the user's device.

[0317] Information display function

[0318] Player and match information sent from the server is overlaid on the user's terminal screen in real time. This allows users to instantly check detailed information about players and matches while watching.

[0319] Chat function

[0320] Users enter questions in text format using the in-app chat function. The entered questions are sent to the server and analyzed by a generative AI model. The generative AI model generates appropriate answers to the questions, retrieves supplementary information from the database, and sends it back to the user's device.

[0321] Emotion recognition function

[0322] The emotion engine recognizes the user's emotions in real time from their facial expressions and voice. The recognized emotion information is sent to a server, which generates customized information based on that information and sends it back to the user's terminal. For example, if the user is excited, detailed player performance information is provided; if they are relaxed, relevant anecdotal information is provided.

[0323] Examples of specific actions

[0324] For example, when a user launches the smartphone app and selects "Match A" from the "Match List" screen, the following sequence of events occurs:

[0325] 1. The user requests information about "Match A" from the server. The server retrieves the information about Match A from the database and sends it to the user's terminal.

[0326] 2. When a user points their camera at "Player X" while watching a match, the captured video is sent to the server. The server analyzes the video and identifies Player X. Player X's profile, statistics, etc., are retrieved from the database and sent back to the user's terminal.

[0327] 3. When a user asks a question using the chat function, such as "Please tell me the statistics for player X," the question is sent to the server. The generating AI model analyzes the question and generates a relevant answer (for example, "Player X's batting average this season is .300"), which is then sent back to the user's terminal.

[0328] 4. Simultaneously, the emotion engine recognizes the user's emotions from their facial expressions and voice, and displays detailed performance information if the user is excited, or related episodes if they are relaxed.

[0329] Example of a prompt

[0330] Examples of prompt statements to input into a generative AI model include the following:

[0331] 1. Example of a prompt when selecting a match

[0332] The user selected "Match A" from the match list. Retrieve the information for Match A and send it back to the user.

[0333] 2. Example prompts during camera image analysis

[0334] A user has taken a picture of player X with their camera. Please analyze the video footage and provide player X's profile and latest statistics.

[0335] 3. Examples of prompts when asking questions in a chat.

[0336] A user asked, "What are player Y's statistics for this season?" Please generate an answer based on player Y's latest statistics.

[0337] 4. Examples of prompts during emotion recognition

[0338] The user is excited. In this case, prioritize providing detailed performance information. If the user is relaxed, provide relevant episode information.

[0339] In this way, this system can further enhance the individual viewing experience by meeting users' real-time information needs and providing emotion-based, customized information.

[0340] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0341] Step 1:

[0342] Match Selection

[0343] Input: The user selects the match they want to watch from the match list on the application's home screen.

[0344] Data processing: Generate user selection information (match ID).

[0345] Output: The match ID is requested from the server.

[0346] Specific action: The user taps "Match A". The selected match ID is sent to the server.

[0347] Step 2:

[0348] Server-based acquisition of match information

[0349] Input: Match ID

[0350] Data processing: Search and retrieve relevant match information from the database.

[0351] Output: Match information is sent to the user's terminal.

[0352] Specific operation: The server accesses the database, retrieves information about match A (date, time, location, participating teams, etc.), and sends it back to the user's terminal.

[0353] Step 3:

[0354] Camera video capture and transmission

[0355] Input: Camera footage

[0356] Data processing: Capture video data and send it to the server.

[0357] Output: Video data is sent to the server.

[0358] Specific operation: The user takes a picture of player X using a mobile device, and the video is sent to the server.

[0359] Step 4:

[0360] Video analysis and information return by the server

[0361] Input: Video data

[0362] Data processing: Analyze the video to identify player X and retrieve related information from the database.

[0363] Output: Player X's information is sent back to the user's terminal.

[0364] Specific operation: The server analyzes the video data and identifies player X. It retrieves player X's profile and latest statistics from the database and sends them back to the user's terminal.

[0365] Step 5:

[0366] Information overlay display

[0367] Input: Player information, match information

[0368] Data processing: The received information is displayed as an overlay on the screen.

[0369] Output: Displayed in real time on the user's terminal.

[0370] Specific operation: When the user is filming player X, player X's profile and performance information will be overlaid on the video.

[0371] Step 6:

[0372] Send a chat question

[0373] Input: Chat text

[0374] Data processing: Send chat input text to the server.

[0375] Output: The question content is sent to the server.

[0376] Specific action: The user types "Please tell me the stats of player X" in the chat and sends it to the server.

[0377] Step 7:

[0378] Answer generation using a generative AI model

[0379] Input: Question content

[0380] Data processing: Analyze the question, retrieve the necessary information from the database, and generate the answer.

[0381] Output: The generated response is sent to the user's terminal.

[0382] Specific operation: The generative AI model analyzes the question and generates an answer, such as "Player X's batting average this season is .300," which is then sent to the user's terminal.

[0383] Step 8:

[0384] Sending emotion recognition data

[0385] Input: User facial expression data, voice data

[0386] Data processing: Analyze facial expressions and voice to generate emotion data.

[0387] Output: Emotional data is sent to the server.

[0388] Specific operation: The system analyzes the user's face and voice, and the emotion engine generates emotional data such as excitement or relaxation, which is then sent to the server.

[0389] Step 9:

[0390] Server-based customization of information

[0391] Input: Sentiment data

[0392] Data processing: Customize the information provided based on emotional data.

[0393] Output: Customization information is sent to the user's terminal.

[0394] Specific operation: The server selects detailed performance and episode information based on the user's emotional information and sends it to the user's terminal.

[0395] By having each processing step work in conjunction in this way, users can obtain match and player information in real time and enjoy customized information tailored to their individual emotions.

[0396] (Application Example 2)

[0397] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0398] Modern users demand more sophisticated information in experiences such as shopping and watching sports. Furthermore, providing timely and relevant information based on user emotions can dramatically improve the user experience. However, existing systems struggle to adapt to users' real-time emotional states and the information retrieval process is often cumbersome.

[0399] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0400] In this invention, the server includes means for acquiring information on objects and events from a database and sending it back to a mobile terminal; means for acquiring information on identified objects and areas and sending it back to the mobile terminal; means for generating answers to questions using a generative AI model and sending them back to the mobile terminal; and means for acquiring emotional information and providing the user terminal with customized information based on the user's emotions. As a result, the user can easily acquire detailed information on products and objects on the spot, and furthermore, receive optimal information in real time based on the user's emotions.

[0401] A "user terminal" is a device equipped with communication functions that a user uses to acquire and manipulate information.

[0402] A "mobile device" is a device that a user can carry with them, equipped with a camera and communication functions, and capable of displaying and transmitting information.

[0403] A "server" is a computer system that manages and processes information in conjunction with a database and provides that information in response to requests from user terminals.

[0404] A "database" is a system for organizing, storing, and searching information, and is a storage medium where match information, player information, product information, and so on are stored.

[0405] A "subject" is an item or person of interest to the user, which is captured and identified in the camera footage.

[0406] A "domain" refers to a specific place or area, specifically the field or area where an object exists.

[0407] A "generative AI model" is an artificial intelligence model that generates appropriate answers or information in response to given input.

[0408] An "emotion engine" is an algorithm that analyzes a user's facial expressions and voice data to recognize and determine their emotional state.

[0409] "Identification results" refer to information about objects and areas obtained when analyzing camera footage.

[0410] "Customized information" refers to information that is optimized and provided according to the user's emotions and circumstances.

[0411] This invention is a system that allows users in physical stores to obtain product and promotional information in real time using smartphones or smart glasses, and further provides a customized shopping experience based on the user's emotions.

[0412] System Overview

[0413] This system includes user terminals (smartphones and smart glasses), a server, a database, an emotion engine, and a generative AI model. The user terminal provides functions such as retrieving product information, answering questions, and emotion recognition through applications. The server works in conjunction with the database to manage vast amounts of product and promotional information, providing it in response to requests from user terminals.

[0414] User terminal operation

[0415] The user terminal is a smartphone or smart glasses operated by the user, and various functions are performed by launching an application. The application includes the following functions:

[0416] 1. Product Selection Function

[0417] The user launches the app and selects the product they want to find from the product list or category within the store. The selected product information is requested from the server.

[0418] 2. Camera function

[0419] Users obtain product information by scanning items using their smartphones or smart glasses. The camera footage is analyzed, and the information of the identified products is sent to a server.

[0420] 3. Information display function

[0421] Product information and promotional information sent from the server are overlaid in real time on the terminal's screen or visual display.

[0422] 4. Chat function

[0423] Users enter questions within the app, and an AI model generates appropriate answers. For example, the AI ​​generates an answer to a question such as, "Are there any similar products to this one?"

[0424] 5. Emotion recognition function

[0425] The emotion engine recognizes the user's emotions in real time from their facial expressions and voice. The recognized emotion information is sent to the server, and the information provided to the user is customized based on that information.

[0426] Server operation

[0427] The server interacts with the database and is responsible for managing and providing information about objects and events. Its specific operations are as follows:

[0428] 1. Providing product information

[0429] When the server receives a product selection request from the user's terminal, it retrieves the corresponding product information from the database and sends it back to the user's terminal.

[0430] 2. Analysis of camera footage

[0431] The system analyzes camera footage transmitted from the user's terminal to identify products. Based on the identification results, it retrieves relevant product and promotional information from a database and sends it back to the terminal.

[0432] 3. Responses by generative AI models

[0433] The system analyzes questions submitted by users via the chat function using an AI model and generates answers. The generated answers are supplemented with necessary information retrieved from a database and then sent back to the user's device.

[0434] 4. Processing of emotional information

[0435] The system acquires user emotion information recognized by the emotion engine and customizes the information provided to the user's device based on that information. For example, if the user appears happy, it provides information about special promotions; if the user appears surprised, it provides detailed product information and reviews.

[0436] Specific operation examples

[0437] For example, if a user uses smart glasses to find a specific product, the following sequence of events will occur:

[0438] 1. When a user scans a product with the smart glasses' camera, the captured video is sent to the server. The server analyzes the video and identifies the product. Detailed information and promotional information for the identified product are retrieved from the database and sent back to the user's device.

[0439] 2. When a user asks "Are there any similar products to this one?" using the chat function, the question is sent to the server. The generation AI model analyzes the question, generates a relevant answer (for example, "There is a similar product called XX"), and sends it back to the user's device.

[0440] 3. Simultaneously, the emotion engine recognizes the user's emotions from their facial expressions and voice. If the user appears happy, it displays additional promotional information; if they are interested, it displays detailed product information and reviews from other users.

[0441] In this way, the system meets users' real-time information needs and further enriches the shopping experience by providing customized information based on their emotions.

[0442] Prompt example

[0443] "Please add a feature to this shopping assistant app that provides information based on real-time emotions. When a user views a particular product, it should display customized information based on the user's emotions and past purchase history."

[0444] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0445] Step 1:

[0446] Users scan products using their smartphones or smart glasses.

[0447] Input: The user takes a picture of the product with their camera.

[0448] Output: Captured product video is obtained.

[0449] Specific operation: The user points the camera at the product and presses the capture button. The smart device's camera captures the image and saves the image data locally.

[0450] Step 2:

[0451] The device sends the captured video to the server.

[0452] Input: Data from captured product video.

[0453] Output: Product video data is sent to the server.

[0454] Specific operation: The application retrieves the captured video data and sends it to the server via the network.

[0455] Step 3:

[0456] The server analyzes the received video data to identify the product.

[0457] Input: Product video data sent to the server.

[0458] Output: Identified product ID and related information.

[0459] Specific operation: The server applies a video analysis algorithm to identify the product. As a result of the identification, it extracts a product ID and retrieves product information from the database based on that ID.

[0460] Step 4:

[0461] The server retrieves product information from the database and sends it back to the user's terminal.

[0462] Input: Identified product ID.

[0463] Output: Corresponding product information data.

[0464] Specific operation: The server executes a database query based on the product ID and retrieves the corresponding product information (price, stock, description, etc.). The retrieved data is then sent back to the user's terminal.

[0465] Step 5:

[0466] The device displays the product information it has received to the user.

[0467] Input: Received product information data.

[0468] Output: Product information is displayed on the user's terminal screen.

[0469] Specific operation: The application analyzes the received product information data and displays detailed information, including product name, price, and description, as an overlay on the screen.

[0470] Step 6:

[0471] Users enter their questions using the in-app chat function.

[0472] Input: User's question text.

[0473] Output: The question text is sent to the server.

[0474] Specific operation: The user types a question in the chat window and presses the send button. The application sends the question text to the server.

[0475] Step 7:

[0476] The server uses a generated AI model to produce an answer to the question and sends it back to the user's terminal.

[0477] Input: User's question text.

[0478] Output: Generated answer text.

[0479] Specific operation: The server inputs the question text into an AI model, which generates an appropriate answer. The generated answer is then supplemented with information retrieved from a database and sent back to the user's terminal.

[0480] Step 8:

[0481] The user's terminal displays the generated response in the chat window.

[0482] Input: Generated response text.

[0483] Output: Chat response displayed on the user's device.

[0484] Specific operation: The application displays the received response text in the chat window for the user to review.

[0485] Step 9:

[0486] The device captures the user's facial expressions and voice, analyzes them using an emotion engine, and sends the emotional information to the server.

[0487] Input: User's facial expressions and voice data.

[0488] Output: Analyzed emotion information.

[0489] Specific operation: The smart device's camera and microphone are used to capture the user's facial expressions and voice, and the emotion engine analyzes this data. The resulting emotion information is then sent to the server.

[0490] Step 10:

[0491] The server generates customized information based on emotional data and sends it back to the user's terminal.

[0492] Input: Analyzed emotion information.

[0493] Output: Customized informational data.

[0494] Specific operation: The server analyzes the received emotional information and generates optimal product and promotional information based on the user's emotional state. The generated information is then sent back to the user's terminal.

[0495] Step 11:

[0496] The device provides users with customized information.

[0497] Input: Customized information data.

[0498] Output: Customization information displayed to the user.

[0499] Specific operation: The application displays the customization information it has received and provides the user with appropriate product suggestions and promotions.

[0500] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0501] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0502] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0503] [Second Embodiment]

[0504] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0505] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0506] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0507] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0508] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0509] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0510] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0511] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0512] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0513] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0514] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0515] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0516] This invention relates to a system that provides real-time information about watching sports using a user terminal, a server, and a database.

[0517] System Overview

[0518] This system aims to enhance the viewing experience by allowing users to obtain match and player information in real time using their mobile devices. The overall system configuration and operation are described below.

[0519] User terminal operation

[0520] A user terminal is a mobile device (smartphone, tablet, etc.) operated by the user, and it runs when an application is launched. An application includes the following functions:

[0521] 1. Match Selection Function

[0522] The user launches the app and selects the match they want to watch from the list of currently ongoing matches displayed on the home screen. The selected match information is then requested from the server.

[0523] 2. Camera function

[0524] Users activate the camera function and capture video by pointing their mobile device at the field or players. The captured video is analyzed, and information about identified players and areas is sent to the server.

[0525] 3. Information display function

[0526] Player and match information sent from the server is overlaid on the terminal screen in real time. This allows users to instantly check detailed information about players and matches while watching.

[0527] 4. Chat function

[0528] Users enter their questions in text format using the in-app chat function. The entered questions are sent to the server, where a generative AI generates appropriate answers.

[0529] Server operation

[0530] The server works in conjunction with the database to manage large amounts of match and player information, and is responsible for providing this information in response to requests from user terminals. Its specific operations are as follows:

[0531] 1. Providing match information

[0532] When the server receives a request to select a match from the user's terminal, it retrieves the relevant match information from the database and sends it back to the user's terminal.

[0533] 2. Analysis of camera footage

[0534] The system analyzes camera footage transmitted from the user's terminal to identify players and specific areas of the pitch. Based on this identification, it retrieves relevant player and area information from a database and sends it back to the terminal.

[0535] 3. Responses by generative AI

[0536] The system uses a generative AI to analyze questions sent by users through the chat function and generate answers. The generated answers are supplemented with necessary information retrieved from a database and then sent back to the user's device.

[0537] Specific operation examples

[0538] For example, when a user launches the smartphone app and selects "Match A" from the "Match List" screen, the following sequence of events occurs:

[0539] 1. The user requests information about the selected match from the server. The server retrieves information about match A from the database and sends it to the user's terminal.

[0540] 2. When a user points their camera at "Player X" while watching a match, the captured video is sent to the server. The server analyzes the video and identifies Player X. Player X's profile, statistics, etc., are retrieved from the database and sent back to the user's terminal.

[0541] 3. When a user asks a question via the chat function, such as "Please tell me the statistics for player X," the question is sent to the server. The generative AI analyzes the question and generates a relevant answer (for example, "Player X's batting average this season is .300"), which is then sent back to the user's terminal.

[0542] In this way, the system provides users with the information they need while watching a game in real time, resulting in a richer viewing experience.

[0543] The following describes the processing flow.

[0544] Step 1:

[0545] The user launches the app on their mobile device. The app's home screen is displayed, showing a list of currently ongoing matches.

[0546] Step 2:

[0547] The user selects the match they want to watch. A request containing information about the selected match is sent to the server.

[0548] Step 3:

[0549] The server receives the request and retrieves the relevant match information from the database. The retrieved match information is then sent back from the server to the user's terminal.

[0550] Step 4:

[0551] The user's terminal displays match information received on the screen. The displayed information includes the player list, match score, and opponent.

[0552] Step 5:

[0553] The user activates the camera function and points their mobile device at the field or players. The video captured by the camera function is analyzed on the device.

[0554] Step 6:

[0555] The device identifies players and areas from camera footage and sends the identification results to the server. The identification results include the player's location information and identification ID.

[0556] Step 7:

[0557] The server receives the identification result and retrieves the corresponding player and area information from the database. The retrieved information is then sent back from the server to the user's terminal.

[0558] Step 8:

[0559] The system overlays information received by the user's device onto the camera feed. This allows users to view detailed information about players and areas in real time.

[0560] Step 9:

[0561] Users enter questions using the chat function. For example, a question like "What is player X's batting average this season?" might be entered in chat format.

[0562] Step 10:

[0563] The terminal sends the entered question to the server. Upon receiving the question, the server uses a generative AI to generate an answer.

[0564] Step 11:

[0565] The server sends the generated response to the user's terminal. The generated response contains the necessary information, including details such as, for example, "Player X's batting average this season is .300."

[0566] Step 12:

[0567] The system displays the answers received by the user's device on the chat screen. This allows the user to see the answers to their questions in real time.

[0568] (Example 1)

[0569] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0570] Traditional sports viewing systems made it difficult for users to easily obtain detailed match and player information in real time. Furthermore, there was a lack of intuitive means for users to seek additional information to gain a deeper understanding of the game. Additionally, systems for providing appropriate answers in real time via chat were not adequately developed.

[0571] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0572] In this invention, the server includes means for a mobile terminal to acquire information about matches and players in real time; means for the server to acquire information about matches and players from a database and send it back to the mobile terminal; means for the user terminal to capture video of matches and players using its camera function and send the identification results to the server; means for the server to acquire information about identified players and areas and send it back to the mobile terminal; means for the user terminal to accept questions in chat format and send the questions to the server; means for the server to generate answers to the questions using a generative AI model and send them back to the mobile terminal; and means for generating prompt sentences to be input to the generative AI model. As a result, users can intuitively acquire detailed information about matches and players in real time and have a richer viewing experience.

[0573] A "user terminal" is a mobile device operated by a user, which runs applications and displays information about matches and players.

[0574] A "mobile device" refers to a portable electronic device such as a smartphone or tablet.

[0575] A "server" is a computer system that manages information in conjunction with a database and processes requests from user terminals.

[0576] A "database" is a data management system that stores match information and player information, and allows you to retrieve it as needed.

[0577] "Capturing" refers to the act of capturing video using the camera's functions.

[0578] "Identification results" refer to information about players and the field identified through the analysis of captured video footage.

[0579] A "generative AI model" is an artificial intelligence model that analyzes an input question and generates an appropriate answer.

[0580] A "prompt message" is a string of characters input into a generative AI model, containing instructions for the model to generate an answer.

[0581] "Overlay display" is a format that displays additional information on top of an existing screen.

[0582] A "chat format" is an interface format where questions and answers are conducted using text.

[0583] "Live data" refers to data generated in real time during the progress of a match.

[0584] This invention relates to a system that allows users to obtain real-time information about matches and players using their mobile devices, thereby providing a richer viewing experience. This system operates by combining user terminals, a server, and a database.

[0585] Specifically, the user's device is a mobile device such as a smartphone or tablet, with the application installed. This application includes a match selection function, camera function, information display function, and chat function. When the user launches the app, a list of currently ongoing matches is displayed on the home screen. The user selects the match they want to watch, and the match information is requested from the server.

[0586] The server maintains a large amount of match and player information in its database. When a match selection request is sent from a user terminal, the server retrieves the relevant match information from the database and sends it to the user terminal. Also, when a user uses the camera function to capture video of a match or players, the video is sent to the server. The server analyzes the video, retrieves information on identified players and fields from the database, and sends it back to the user terminal.

[0587] Furthermore, the user terminal can overlay the received information on the screen. When a user enters a question in text format using the chat function, the question is sent to the server. The server analyzes the question using a generative AI model and generates an appropriate answer. The generated answer, along with supplementary information retrieved from the database, is sent back to the user terminal.

[0588] For example, a user launches the app on their smartphone and selects "Match A" on the home screen. At this point, the server retrieves detailed information about the selected Match A and sends it to the user's device. If the user points their camera at "Player X" while watching the match, the captured video is sent to the server, which analyzes the video to identify Player X. Player X's profile and performance information are retrieved from the database and sent back to the user's device. Furthermore, if the user asks a question using the chat function, such as "Please tell me Player X's performance," a generative AI model analyzes the question, generates an answer such as "Player X's batting average this season is .300," and sends it to the user's device.

[0589] Examples of prompt statements include the following:

[0590] "Please tell me about match A."

[0591] "I'd like to know player X's performance this season."

[0592] "What is the current score for Match A?"

[0593] This allows users to intuitively operate the system while obtaining a wealth of information in real time, greatly improving the viewing experience. Furthermore, the introduction of generative AI models enables instant responses to chat-style questions, increasing user satisfaction.

[0594] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0595] Step 1:

[0596] The user launches the application on their mobile device. The input is the user's action (tap), and the output is the display of the application's home screen. Specifically, the application refers to an internal database, retrieves a list of currently ongoing matches, and displays it on the home screen.

[0597] Step 2:

[0598] The user selects the match they want to watch. The input is the user's selection of a match, and the output is a request for information about the selected match sent to the server. Specifically, the application sends a request to the server that includes the ID of the selected match.

[0599] Step 3:

[0600] The server processes the received request and retrieves the relevant match information from the database. The input is the match ID sent from the user's terminal, and the output is detailed information about the relevant match sent back to the user's terminal. Specifically, the server searches the database using the match ID as the key, retrieves the relevant match information (score, player list, etc.), and sends it back to the user's terminal.

[0601] Step 4:

[0602] The user activates the camera function on their device to capture video of the match or players. The input is the user's camera activation operation, and the output is the captured video sent to the server. Specifically, the user's device sends the video data acquired from the camera to the server in real time.

[0603] Step 5:

[0604] The server analyzes the received video to identify players and specific areas of the field. The input is captured video data, and the output generates information about the identified players and areas. Specifically, the server uses image recognition technology to analyze the video, retrieves the identified players and areas from a database, and sends it back to the user terminal.

[0605] Step 6:

[0606] The user terminal overlays the received information onto the camera feed. The input is player and area information sent from the server, and the output is the information displayed overlaid on the camera feed. Specifically, the user terminal analyzes the received data and displays player names and profiles on top of the camera feed in real time.

[0607] Step 7:

[0608] The user enters a question using the chat function. The input is the user's text input, and the output is the entered question sent to the server. Specifically, the user's terminal receives the user's input on the chat screen and sends that text to the server.

[0609] Step 8:

[0610] The server uses a generative AI model to analyze the question and generate an appropriate answer. The input is the question sent by the user, and the generated answer is sent back to the user's terminal as output. Specifically, the server inputs the question into the AI ​​model, retrieves supplementary information from the database based on the analysis results to generate an answer, and sends it to the user's terminal.

[0611] Step 9:

[0612] The user's terminal displays the response from the server on the chat screen. The input is the response data sent from the server, and the output is the response displayed on the user's chat screen. Specifically, the user's terminal analyzes the received response and displays it on the chat screen so that the user can confirm the response.

[0613] (Application Example 1)

[0614] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0615] In modern sports viewing, users desire more detailed and real-time information about matches and players. However, current systems lack the means to achieve this quickly and effectively. In particular, there is a need for systems that offer advanced interactive features such as eye-tracking, real-time information overlays, and question answering using generative models.

[0616] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0617] In this invention, the server includes means for the server to retrieve player and match information from a database and send it back to the mobile terminal; means for the user terminal to capture images of players and the field using its camera function and send the identification results to the server; and means for the server to generate answers to questions using a generative model and send them back to the mobile terminal. This allows the user to identify the player or area they are focusing on using an eye-tracking camera and obtain detailed information in real time. Furthermore, by utilizing a generative model, accurate answers to the user's questions can be provided quickly. In addition, because player information is overlaid on the video, the user can obtain visually intuitive information, improving the viewing experience.

[0618] A "user terminal" refers to a device operated by a user in a sports viewing system, such as a smartphone or tablet.

[0619] A "mobile device" is an electronic device that a user can carry with them, and includes smartphones, tablets, and other similar devices.

[0620] "Players and field" refers to the players and the areas within the stadium where their activities take place during a sports match.

[0621] A "server" is a central system for managing and processing information, and its role is to retrieve information from a database and provide it to user terminals.

[0622] A "database" refers to a collection of data used to store and manage information such as match details and player information in a searchable format.

[0623] The "camera function" refers to a function for capturing video, specifically the ability to photograph players and the field using the camera built into the mobile device.

[0624] "Identification results" refer to data that includes information about identified players and areas, obtained by analyzing video footage captured by cameras.

[0625] A "generative model" refers to an artificial intelligence model that has been trained using pre-trained data, and is designed to generate appropriate answers to user questions.

[0626] A "gaze recognition camera" refers to a camera function that tracks the user's gaze and identifies the object the user is focusing on.

[0627] "Overlay display" refers to a technique of displaying additional information superimposed on top of video footage, specifically displaying player information and match information in real time on video during a game.

[0628] This invention relates to a system that provides real-time information about sports viewing using a user terminal, a server, and a database.

[0629] User terminal operation

[0630] The user terminal is a mobile device operated by the user (e.g., a smartphone or tablet), and it runs when an application is launched. The application includes the following functions:

[0631] 1. Match Selection Function

[0632] The user launches the app and selects a match they want to watch from a list of matches displayed on a virtual screen. The selected match information is then requested from the server.

[0633] 2. Camera function

[0634] The system uses a camera mounted on the user's terminal to capture video of the players and the field. The captured video is analyzed, and the identification results are sent to the server.

[0635] 3. Eye-tracking camera function

[0636] The user's device has a built-in eye-tracking camera that tracks the user's gaze and identifies the player or area they are focusing on.

[0637] 4. Information Overlay Display Function

[0638] Player and match information sent from the server is overlaid on the user's terminal screen in real time. This function allows users to intuitively check detailed information about players and matches while watching.

[0639] 5. Chat function

[0640] Users input questions by voice using the in-app chat function. The entered questions are sent to the server, and a generative model generates appropriate answers.

[0641] Server operation

[0642] The server works in conjunction with the database to manage large amounts of match and player information, and is responsible for providing this information in response to requests from user terminals. Its specific operations are as follows:

[0643] 1. Providing match information

[0644] The server receives a match selection request from the user's terminal, retrieves the relevant match information from the database, and sends it to the user's terminal.

[0645] 2. Analysis of camera footage

[0646] The server analyzes camera footage transmitted from the user's terminal to identify players and specific areas of the field. Based on this identification, it retrieves relevant player and area information from the database and sends it back to the user's terminal.

[0647] 3. Analysis of gaze recognition

[0648] The server uses an eye-tracking camera to identify the player or area the user is focusing on and sends that information to the user's terminal.

[0649] 4. Response using a generative model

[0650] The system analyzes questions submitted by users through the chat function using a generative model and generates answers. The generated answers are supplemented with necessary information retrieved from the database and sent back to the user's device.

[0651] Hardware and software to be used

[0652] User devices: Smartphones, tablets, and eye-tracking cameras

[0653] Server: Database management system, generative model

[0654] Software: OpenCV (camera image analysis), requests (communication with server)

[0655] Specific example

[0656] For example, a user launches a smartphone app and selects "Match A" from the "Match List" screen. Then, when they fix their gaze on "Player X" using the eye-tracking camera, Player X's profile and performance information are displayed as an overlay. Additionally, if the user asks a voice question such as "Please tell me Player X's performance," that performance information will be displayed on the screen.

[0657] Example of a prompt

[0658] "Please tell me the statistics of player X."

[0659] "Please tell me the progress of the match."

[0660] "What is the status of player Y's injury?"

[0661] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0662] Step 1:

[0663] (Input) The user launches a sports viewing application on their smartphone and displays the match list screen.

[0664] (Specific action) The user accesses the application's home screen, and a list of currently ongoing matches is displayed.

[0665] (Data processing) Match information is requested from the server, the server retrieves that information from the database, and sends it to the user's terminal as a list of matches.

[0666] (Output) The match list is displayed on the smartphone screen.

[0667] Step 2:

[0668] (Input) The user selects the match they want to watch (for example, "Match A").

[0669] (Specific action) The user selects "Match A" from the match list by tapping or clicking.

[0670] (Data Processing) The selected match information is requested from the server, and the server retrieves that match information from the database and sends it to the user's terminal as detailed information.

[0671] (Output) Detailed information about "Match A" is displayed on the smartphone screen.

[0672] Step 3:

[0673] (Input) The user activates the camera function and captures the players or field with the camera.

[0674] (Specific operation) The smartphone camera captures video of the players and the field in real time.

[0675] (Data processing) The captured video is sent from the user's terminal to the server, which analyzes the video to identify specific parts of the players or field.

[0676] (Output) The identification result is sent from the server to the user terminal.

[0677] Step 4:

[0678] (Input) The user uses eye-tracking technology to focus on a specific player or area.

[0679] (Specific operation) Eye-tracking cameras installed in smartphones or HMDs track the user's gaze and identify the player or area they are focusing on.

[0680] (Data Processing) Eye-tracking data is sent from the user terminal to the server, and the server uses that data to retrieve information about identified players and areas from its database and sends it back to the user terminal.

[0681] (Output) Detailed information about the player or area being watched will be overlaid on the smartphone screen.

[0682] Step 5:

[0683] (Input) The user asks a question using the voice input function (for example, "Please tell me the statistics of player X").

[0684] (Specific action) The user activates the app's voice input function and inputs the question by voice.

[0685] (Data processing) The audio data is converted into text by the speech recognition function, and the question is sent from the user's terminal to the server.

[0686] The output generation model analyzes the question, generates an appropriate answer, and sends it to the user's terminal.

[0687] Step 6:

[0688] (Input) The user receives the response sent from the server and confirms it on the screen.

[0689] (Specific operation) The response sent from the server is displayed on the user's terminal.

[0690] (Data processing) The responses generated by the generative model are displayed on the user's terminal, and supplementary information is provided from the database as needed.

[0691] (Output) The information requested by the user and the answers to the questions are displayed on the screen.

[0692] Step 7:

[0693] (Input) Live status data is sent to the server.

[0694] (Specific operation) The server retrieves real-time live status from the database.

[0695] (Data processing) Live status data is sent to the user's terminal.

[0696] (Output) The live status is displayed in real time on the user's smartphone.

[0697] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0698] This invention relates to a system that provides real-time information about watching sports using a user terminal, a server, a database, and an emotion engine.

[0699] System Overview

[0700] This system aims to acquire match and player information in real time using the user's mobile device, and to provide customized information based on the user's emotions. The overall system configuration and operation are described below.

[0701] User terminal operation

[0702] A user terminal is a mobile device (smartphone, tablet, etc.) operated by the user, and it runs when an application is launched. An application includes the following functions:

[0703] 1. Match Selection Function

[0704] The user launches the app and selects a match they want to watch from the list of currently running matches displayed on the home screen. The selected match information is then requested from the server.

[0705] 2. Camera function

[0706] The user activates the camera function and points their mobile device at the field or players to capture video. The captured video is analyzed, and information about identified players and areas is sent to the server.

[0707] 3. Information display function

[0708] Player and match information sent from the server is overlaid on the terminal screen in real time. This allows users to instantly check detailed information about players and matches while watching.

[0709] 4. Chat function

[0710] Users enter questions in text format using the in-app chat function. The entered questions are sent to the server, where a generative AI generates appropriate answers.

[0711] 5. Emotion recognition function

[0712] The emotion engine recognizes the user's emotions in real time from their facial expressions and voice. The recognized emotion information is sent to the server.

[0713] Server operation

[0714] The server works in conjunction with the database to manage large amounts of match and player information, and is responsible for providing this information in response to requests from user terminals. Its specific operations are as follows:

[0715] 1. Providing match information

[0716] When the server receives a request to select a match from the user's terminal, it retrieves the relevant match information from the database and sends it back to the user's terminal.

[0717] 2. Analysis of camera footage

[0718] The system analyzes camera footage transmitted from the user's terminal to identify players and specific areas of the pitch. Based on this identification, it retrieves relevant player and area information from a database and sends it back to the terminal.

[0719] 3. Responses by generative AI

[0720] The system uses a generative AI to analyze questions sent by users through the chat function and generate answers. The generated answers are supplemented with necessary information retrieved from a database and then sent back to the user's device.

[0721] 4. Processing emotional information

[0722] The system acquires user emotion information recognized by the emotion engine and customizes the information provided to the user's device based on that information. For example, if the user is excited, it provides detailed player statistics, and if the user is relaxed, it provides stories such as past episodes.

[0723] Specific operation examples

[0724] For example, when a user launches the smartphone app and selects "Match A" from the "Match List" screen, the following sequence of events occurs:

[0725] 1. The user requests information about the selected match from the server. The server retrieves information about match A from the database and sends it to the user's terminal.

[0726] 2. When a user points their camera at "Player X" while watching a match, the captured video is sent to the server. The server analyzes the video and identifies Player X. Player X's profile, statistics, etc., are retrieved from the database and sent back to the user's terminal.

[0727] 3. When a user asks a question via the chat function, such as "Please tell me the statistics for player X," the question is sent to the server. The generative AI analyzes the question and generates a relevant answer (for example, "Player X's batting average this season is .300"), which is then sent back to the user's terminal.

[0728] 4. Simultaneously, the emotion engine recognizes the user's emotions from their facial expressions and voice, and displays detailed performance information if the user is excited, or related episodes if they are relaxed.

[0729] In this way, the system meets users' real-time information needs and further enriches the viewing experience by providing customized information based on emotions.

[0730] The following describes the processing flow.

[0731] Step 1:

[0732] The user launches the app on their mobile device. The app's home screen is displayed, showing a list of currently ongoing matches.

[0733] Step 2:

[0734] The user selects the match they want to watch. A request containing information about the selected match is sent to the server.

[0735] Step 3:

[0736] The server receives the request and retrieves the relevant match information from the database. The retrieved match information is then sent back from the server to the user's terminal.

[0737] Step 4:

[0738] The user's terminal displays match information received on the screen. The displayed information includes the player list, match score, and opponent.

[0739] Step 5:

[0740] The user activates the camera function and points their mobile device at the field or players. The video captured by the camera function is analyzed on the device.

[0741] Step 6:

[0742] The device identifies players and areas from camera footage and sends the identification results to the server. The identification results include the player's location information and identification ID.

[0743] Step 7:

[0744] The server receives the identification result and retrieves the corresponding player and area information from the database. The retrieved information is then sent back from the server to the user's terminal.

[0745] Step 8:

[0746] The system overlays information received by the user's device onto the camera feed. This allows users to view detailed information about players and areas in real time.

[0747] Step 9:

[0748] Users enter questions using the chat function. For example, a question like "What is player X's batting average this season?" might be entered in chat format.

[0749] Step 10:

[0750] The terminal sends the entered question to the server. Upon receiving the question, the server uses a generative AI to generate an answer.

[0751] Step 11:

[0752] The server sends the generated response to the user's terminal. The generated response contains the necessary information, including details such as, for example, "Player X's batting average this season is .300."

[0753] Step 12:

[0754] The system displays the answers received by the user's device on the chat screen. This allows the user to see the answers to their questions in real time.

[0755] Step 13:

[0756] The emotion engine recognizes emotions from the user's facial expressions and voice. For example, it can detect emotions such as excitement, joy, and relaxation.

[0757] Step 14:

[0758] The emotion engine sends the recognized emotion information to the server. The server receives the emotion information.

[0759] Step 15:

[0760] The server customizes the information it provides based on the user's emotional state. For example, if the user is excited, it provides detailed player statistics; if they are relaxed, it provides anecdotes about the players and the game.

[0761] Step 16:

[0762] The server sends customized information to the user's terminal. The user's terminal displays this information, allowing the user to see customized information tailored to their emotions.

[0763] This series of processes allows users to receive personalized, emotion-based information in real time, enabling them to enjoy a richer viewing experience.

[0764] (Example 2)

[0765] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0766] Traditional sports viewing systems lack the means for users to obtain real-time information on matches and players. Furthermore, they fail to provide customized information based on user sentiment, making it difficult to offer an optimized viewing experience for individual users. Additionally, they lack the means to generate quick and accurate answers when responding to user questions via chat functions. To address these issues, a more advanced and personalized information delivery system is needed.

[0767] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0768] In this invention, the server includes means for a mobile terminal to acquire player and field information in real time, means for the server to acquire player and match information from a database and send it back to the mobile terminal, means for the user terminal to capture images of players and the field using its camera function and send the identification results to the server, means for the server to generate answers to questions using a generated AI model and send them back to the mobile terminal, means for the user terminal to analyze the user's facial expressions and voice to acquire emotional information and send the emotional information to the server, and means for the server to generate customized information based on the emotional data and send it back to the mobile terminal. As a result, users can acquire match and player information in real time, and personalized information tailored to the user's emotions can be provided. In addition, the viewing experience is improved because information is provided quickly and accurately through the chat function.

[0769] A "user terminal" is a portable information terminal operated by a user, which acquires and displays information using a specific application.

[0770] A "mobile device" refers to a portable information and communication device such as a smartphone or tablet, which is used to provide users with real-time information on matches and players.

[0771] A "server" is a computer system that communicates with user terminals via a network to process data and provide information.

[0772] A "database" is an information management system that stores match information and player information, and allows a server to retrieve this information as needed.

[0773] The "camera function" is a feature that allows the user's device to take pictures of players and the field, capture video footage, and send that video data to a server for analysis.

[0774] A "generative AI model" is an algorithm or program that uses artificial intelligence to analyze a user's question and generate an appropriate answer.

[0775] An "emotion engine" is a combination of software or hardware that analyzes a user's emotions from their facial expressions and voice and acquires that data.

[0776] "Identification results" refer to information about specific players or parts of the field obtained by analyzing video captured by the camera function.

[0777] "Overlay display" is a technology that displays additional information by superimposing it on the video or image currently displayed on the user's device's screen.

[0778] A "chat-style question" is an inquiry that a user enters in text format and sends to the server.

[0779] "Customized information" refers to information optimized for each individual user's situation, based on their emotional data.

[0780] This invention relates to a system that provides real-time information for sports viewing using a user terminal, server, database, and emotion engine. The system aims to acquire match information and player information in real time using the user's mobile terminal, and further to provide customized information based on the user's emotions.

[0781] System Configuration

[0782] User terminal hardware and software

[0783] The user terminal will be a mobile device such as a smartphone or tablet. This terminal will have a dedicated application installed, which will include features such as match selection, camera function, chat function, and emotion recognition function.

[0784] Server hardware and software

[0785] The server uses a high-performance network server to manage large amounts of data. It interacts with databases that store player and match information and responds to user requests. The server has a generative AI model and an emotion engine implemented in it.

[0786] database

[0787] The database is responsible for storing and managing match and player information. This includes player profiles, statistics, and match progress.

[0788] Emotional Engine

[0789] An emotion engine is software or hardware that analyzes a user's facial expressions and voice to recognize their emotional state in real time. Its role is to acquire the user's emotional data and transmit it to a server.

[0790] Program processing

[0791] Match selection function

[0792] The user launches the application and selects a match they want to watch from the match list. The selected match information is requested from the server. The server retrieves the corresponding match information from the database and sends it back to the user's terminal.

[0793] Camera function

[0794] The user activates the camera function and points their mobile device at the field or players to capture video. The captured video is sent to a server, where it is analyzed. Based on the analysis results, the server retrieves information on identified players and areas from its database and sends it back to the user's device.

[0795] Information display function

[0796] Player and match information sent from the server is overlaid on the user's terminal screen in real time. This allows users to instantly check detailed information about players and matches while watching.

[0797] Chat function

[0798] Users enter questions in text format using the in-app chat function. The entered questions are sent to the server and analyzed by a generative AI model. The generative AI model generates appropriate answers to the questions, retrieves supplementary information from the database, and sends it back to the user's device.

[0799] Emotion recognition function

[0800] The emotion engine recognizes the user's emotions in real time from their facial expressions and voice. The recognized emotion information is sent to a server, which generates customized information based on that information and sends it back to the user's terminal. For example, if the user is excited, detailed player performance information is provided; if they are relaxed, relevant anecdotal information is provided.

[0801] Examples of specific actions

[0802] For example, when a user launches the smartphone app and selects "Match A" from the "Match List" screen, the following sequence of events occurs:

[0803] 1. The user requests information about "Match A" from the server. The server retrieves the information about Match A from the database and sends it to the user's terminal.

[0804] 2. When a user points their camera at "Player X" while watching a match, the captured video is sent to the server. The server analyzes the video and identifies Player X. Player X's profile, statistics, etc., are retrieved from the database and sent back to the user's terminal.

[0805] 3. When a user asks a question using the chat function, such as "Please tell me the statistics for player X," the question is sent to the server. The generating AI model analyzes the question and generates a relevant answer (for example, "Player X's batting average this season is .300"), which is then sent back to the user's terminal.

[0806] 4. Simultaneously, the emotion engine recognizes the user's emotions from their facial expressions and voice, and displays detailed performance information if the user is excited, or related episodes if they are relaxed.

[0807] Example of a prompt

[0808] Examples of prompt statements to input into a generative AI model include the following:

[0809] 1. Example of a prompt when selecting a match

[0810] The user selected "Match A" from the match list. Retrieve the information for Match A and send it back to the user.

[0811] 2. Example prompts during camera image analysis

[0812] A user has taken a picture of player X with their camera. Please analyze the video footage and provide player X's profile and latest statistics.

[0813] 3. Examples of prompts when asking questions in a chat.

[0814] A user asked, "What are player Y's statistics for this season?" Please generate an answer based on player Y's latest statistics.

[0815] 4. Examples of prompts during emotion recognition

[0816] The user is excited. In this case, prioritize providing detailed performance information. If the user is relaxed, provide relevant episode information.

[0817] In this way, this system can further enhance the individual viewing experience by meeting users' real-time information needs and providing emotion-based, customized information.

[0818] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0819] Step 1:

[0820] Match Selection

[0821] Input: The user selects the match they want to watch from the match list on the application's home screen.

[0822] Data processing: Generate user selection information (match ID).

[0823] Output: The match ID is requested from the server.

[0824] Specific action: The user taps "Match A". The selected match ID is sent to the server.

[0825] Step 2:

[0826] Server-based acquisition of match information

[0827] Input: Match ID

[0828] Data processing: Search and retrieve relevant match information from the database.

[0829] Output: Match information is sent to the user's terminal.

[0830] Specific operation: The server accesses the database, retrieves information about match A (date, time, location, participating teams, etc.), and sends it back to the user's terminal.

[0831] Step 3:

[0832] Camera video capture and transmission

[0833] Input: Camera footage

[0834] Data processing: Capture video data and send it to the server.

[0835] Output: Video data is sent to the server.

[0836] Specific operation: The user takes a picture of player X using a mobile device, and the video is sent to the server.

[0837] Step 4:

[0838] Video analysis and information return by the server

[0839] Input: Video data

[0840] Data processing: Analyze the video to identify player X and retrieve related information from the database.

[0841] Output: Player X's information is sent back to the user's terminal.

[0842] Specific operation: The server analyzes the video data and identifies player X. It retrieves player X's profile and latest statistics from the database and sends them back to the user's terminal.

[0843] Step 5:

[0844] Information overlay display

[0845] Input: Player information, match information

[0846] Data processing: The received information is displayed as an overlay on the screen.

[0847] Output: Displayed in real time on the user's terminal.

[0848] Specific operation: When the user is filming player X, player X's profile and performance information will be overlaid on the video.

[0849] Step 6:

[0850] Send a chat question

[0851] Input: Chat text

[0852] Data processing: Send chat input text to the server.

[0853] Output: The question content is sent to the server.

[0854] Specific action: The user types "Please tell me the stats of player X" in the chat and sends it to the server.

[0855] Step 7:

[0856] Answer generation using a generative AI model

[0857] Input: Question content

[0858] Data processing: Analyze the question, retrieve the necessary information from the database, and generate the answer.

[0859] Output: The generated response is sent to the user's terminal.

[0860] Specific operation: The generative AI model analyzes the question and generates an answer, such as "Player X's batting average this season is .300," which is then sent to the user's terminal.

[0861] Step 8:

[0862] Sending emotion recognition data

[0863] Input: User facial expression data, voice data

[0864] Data processing: Analyze facial expressions and voice to generate emotion data.

[0865] Output: Emotional data is sent to the server.

[0866] Specific operation: The system analyzes the user's face and voice, and the emotion engine generates emotional data such as excitement or relaxation, which is then sent to the server.

[0867] Step 9:

[0868] Server-based customization of information

[0869] Input: Sentiment data

[0870] Data processing: Customize the information provided based on emotional data.

[0871] Output: Customization information is sent to the user's terminal.

[0872] Specific operation: The server selects detailed performance and episode information based on the user's emotional information and sends it to the user's terminal.

[0873] By having each processing step work in conjunction in this way, users can obtain match and player information in real time and enjoy customized information tailored to their individual emotions.

[0874] (Application Example 2)

[0875] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0876] Modern users demand more sophisticated information in experiences such as shopping and watching sports. Furthermore, providing timely and relevant information based on user emotions can dramatically improve the user experience. However, existing systems struggle to adapt to users' real-time emotional states and the information retrieval process is often cumbersome.

[0877] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0878] In this invention, the server includes means for acquiring information on objects and events from a database and sending it back to a mobile terminal; means for acquiring information on identified objects and areas and sending it back to the mobile terminal; means for generating answers to questions using a generative AI model and sending them back to the mobile terminal; and means for acquiring emotional information and providing the user terminal with customized information based on the user's emotions. As a result, the user can easily acquire detailed information on products and objects on the spot, and furthermore, receive optimal information in real time based on the user's emotions.

[0879] A "user terminal" is a device equipped with communication functions that a user uses to acquire and manipulate information.

[0880] A "mobile device" is a device that a user can carry with them, equipped with a camera and communication functions, and capable of displaying and transmitting information.

[0881] A "server" is a computer system that manages and processes information in conjunction with a database and provides that information in response to requests from user terminals.

[0882] A "database" is a system for organizing, storing, and searching information, and is a storage medium where match information, player information, product information, and so on are stored.

[0883] A "subject" is an item or person of interest to the user, which is captured and identified in the camera footage.

[0884] A "domain" refers to a specific place or area, specifically the field or area where an object exists.

[0885] A "generative AI model" is an artificial intelligence model that generates appropriate answers or information in response to given input.

[0886] An "emotion engine" is an algorithm that analyzes a user's facial expressions and voice data to recognize and determine their emotional state.

[0887] "Identification results" refer to information about objects and areas obtained when analyzing camera footage.

[0888] "Customized information" refers to information that is optimized and provided according to the user's emotions and circumstances.

[0889] This invention is a system that allows users in physical stores to obtain product and promotional information in real time using smartphones or smart glasses, and further provides a customized shopping experience based on the user's emotions.

[0890] System Overview

[0891] This system includes user terminals (smartphones and smart glasses), a server, a database, an emotion engine, and a generative AI model. The user terminal provides functions such as retrieving product information, answering questions, and emotion recognition through applications. The server works in conjunction with the database to manage vast amounts of product and promotional information, providing it in response to requests from user terminals.

[0892] User terminal operation

[0893] The user terminal is a smartphone or smart glasses operated by the user, and various functions are performed by launching an application. The application includes the following functions:

[0894] 1. Product Selection Function

[0895] The user launches the app and selects the product they want to find from the product list or category within the store. The selected product information is requested from the server.

[0896] 2. Camera function

[0897] Users obtain product information by scanning items using their smartphones or smart glasses. The camera footage is analyzed, and the information of the identified products is sent to a server.

[0898] 3. Information display function

[0899] Product information and promotional information sent from the server are overlaid in real time on the terminal's screen or visual display.

[0900] 4. Chat function

[0901] Users enter questions within the app, and an AI model generates appropriate answers. For example, the AI ​​generates an answer to a question such as, "Are there any similar products to this one?"

[0902] 5. Emotion recognition function

[0903] The emotion engine recognizes the user's emotions in real time from their facial expressions and voice. The recognized emotion information is sent to the server, and the information provided to the user is customized based on that information.

[0904] Server operation

[0905] The server interacts with the database and is responsible for managing and providing information about objects and events. Its specific operations are as follows:

[0906] 1. Providing product information

[0907] When the server receives a product selection request from the user's terminal, it retrieves the relevant product information from the database and sends it back to the user's terminal.

[0908] 2. Analysis of camera footage

[0909] The system analyzes camera footage transmitted from the user's terminal to identify products. Based on the identification results, it retrieves relevant product and promotional information from a database and sends it back to the terminal.

[0910] 3. Responses by generative AI models

[0911] The system analyzes questions submitted by users via the chat function using an AI model and generates answers. The generated answers are then supplemented with necessary information retrieved from a database and sent back to the user's device.

[0912] 4. Processing emotional information

[0913] The system acquires user emotion information recognized by the emotion engine and customizes the information provided to the user's device based on that information. For example, if the user appears happy, it provides information about special promotions; if the user appears surprised, it provides detailed product information and reviews.

[0914] Specific operation examples

[0915] For example, if a user uses smart glasses to find a specific product, the following sequence of events will occur:

[0916] 1. When a user scans a product with the smart glasses' camera, the captured image is sent to the server. The server analyzes the image and identifies the product. Detailed information and promotional information for the identified product are retrieved from the database and sent back to the user's device.

[0917] 2. When a user asks "Are there any similar products to this one?" using the chat function, the question is sent to the server. The generation AI model analyzes the question, generates a relevant answer (for example, "There is a similar product called XX"), and sends it back to the user's device.

[0918] 3. Simultaneously, the emotion engine recognizes the user's emotions from their facial expressions and voice. If the user appears happy, it displays additional promotional information; if they are interested, it displays detailed product information and reviews from other users.

[0919] In this way, the system meets users' real-time information needs and further enriches the shopping experience by providing customized information based on their emotions.

[0920] Prompt example

[0921] "Please add a feature to this shopping assistant app that provides information based on real-time emotions. When a user views a particular product, it should display customized information based on the user's emotions and past purchase history."

[0922] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0923] Step 1:

[0924] Users scan products using their smartphones or smart glasses.

[0925] Input: The user takes a picture of the product with their camera.

[0926] Output: Captured product video is obtained.

[0927] Specific operation: The user points the camera at the product and presses the capture button. The smart device's camera captures the image and saves the image data locally.

[0928] Step 2:

[0929] The device sends the captured video to the server.

[0930] Input: Data from captured product video.

[0931] Output: Product video data is sent to the server.

[0932] Specific operation: The application retrieves the captured video data and sends it to the server via the network.

[0933] Step 3:

[0934] The server analyzes the received video data to identify the product.

[0935] Input: Product video data sent to the server.

[0936] Output: Identified product ID and related information.

[0937] Specific operation: The server applies a video analysis algorithm to identify the product. As a result of the identification, it extracts a product ID and retrieves product information from the database based on that ID.

[0938] Step 4:

[0939] The server retrieves product information from the database and sends it back to the user's terminal.

[0940] Input: Identified product ID.

[0941] Output: Corresponding product information data.

[0942] Specific operation: The server executes a database query based on the product ID and retrieves the corresponding product information (price, stock, description, etc.). The retrieved data is then sent back to the user's terminal.

[0943] Step 5:

[0944] The device displays the product information it has received to the user.

[0945] Input: Received product information data.

[0946] Output: Product information is displayed on the user's terminal screen.

[0947] Specific operation: The application analyzes the received product information data and displays detailed information, including product name, price, and description, as an overlay on the screen.

[0948] Step 6:

[0949] Users enter their questions using the in-app chat function.

[0950] Input: User's question text.

[0951] Output: The question text is sent to the server.

[0952] Specific operation: The user types a question in the chat window and presses the send button. The application sends the question text to the server.

[0953] Step 7:

[0954] The server uses a generated AI model to produce an answer to the question and sends it back to the user's terminal.

[0955] Input: User's question text.

[0956] Output: Generated answer text.

[0957] Specific operation: The server inputs the question text into an AI model, which generates an appropriate answer. The generated answer is then supplemented with information retrieved from a database and sent back to the user's terminal.

[0958] Step 8:

[0959] The user's terminal displays the generated response in the chat window.

[0960] Input: Generated response text.

[0961] Output: Chat response displayed on the user's device.

[0962] Specific operation: The application displays the received response text in the chat window for the user to review.

[0963] Step 9:

[0964] The device captures the user's facial expressions and voice, analyzes them using an emotion engine, and sends the emotional information to the server.

[0965] Input: User's facial expressions and voice data.

[0966] Output: Analyzed emotion information.

[0967] Specific operation: The smart device's camera and microphone are used to capture the user's facial expressions and voice, and the emotion engine analyzes this data. The resulting emotion information is then sent to the server.

[0968] Step 10:

[0969] The server generates customized information based on emotional data and sends it back to the user's terminal.

[0970] Input: Analyzed emotion information.

[0971] Output: Customized informational data.

[0972] Specific operation: The server analyzes the received emotional information and generates optimal product and promotional information based on the user's emotional state. The generated information is then sent back to the user's terminal.

[0973] Step 11:

[0974] The device provides users with customized information.

[0975] Input: Customized information data.

[0976] Output: Customization information displayed to the user.

[0977] Specific operation: The application displays the customization information it has received and provides the user with appropriate product suggestions and promotions.

[0978] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0979] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0980] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0981] [Third Embodiment]

[0982] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0983] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0984] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0985] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0986] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0987] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0988] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0989] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0990] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0991] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0992] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0993] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0994] This invention relates to a system that provides real-time information about watching sports using a user terminal, a server, and a database.

[0995] System Overview

[0996] This system aims to enhance the viewing experience by allowing users to obtain match and player information in real time using their mobile devices. The overall system configuration and operation are described below.

[0997] User terminal operation

[0998] A user terminal is a mobile device (smartphone, tablet, etc.) operated by the user, and it runs when an application is launched. An application includes the following functions:

[0999] 1. Match Selection Function

[1000] The user launches the app and selects the match they want to watch from the list of currently ongoing matches displayed on the home screen. The selected match information is then requested from the server.

[1001] 2. Camera function

[1002] Users activate the camera function and capture video by pointing their mobile device at the field or players. The captured video is analyzed, and information about identified players and areas is sent to the server.

[1003] 3. Information display function

[1004] Player and match information sent from the server is overlaid on the terminal screen in real time. This allows users to instantly check detailed information about players and matches while watching.

[1005] 4. Chat function

[1006] Users enter their questions in text format using the in-app chat function. The entered questions are sent to the server, where a generative AI generates appropriate answers.

[1007] Server operation

[1008] The server works in conjunction with the database to manage large amounts of match and player information, and is responsible for providing this information in response to requests from user terminals. Its specific operations are as follows:

[1009] 1. Providing match information

[1010] When the server receives a request to select a match from the user's terminal, it retrieves the relevant match information from the database and sends it back to the user's terminal.

[1011] 2. Analysis of camera footage

[1012] The system analyzes camera footage transmitted from the user's terminal to identify players and specific areas of the pitch. Based on this identification, it retrieves relevant player and area information from a database and sends it back to the terminal.

[1013] 3. Responses by generative AI

[1014] The system uses a generative AI to analyze questions sent by users through the chat function and generate answers. The generated answers are supplemented with necessary information retrieved from a database and then sent back to the user's device.

[1015] Specific operation examples

[1016] For example, when a user launches the smartphone app and selects "Match A" from the "Match List" screen, the following sequence of events occurs:

[1017] 1. The user requests information about the selected match from the server. The server retrieves information about match A from the database and sends it to the user's terminal.

[1018] 2. When a user points their camera at "Player X" while watching a match, the captured video is sent to the server. The server analyzes the video and identifies Player X. Player X's profile, statistics, etc., are retrieved from the database and sent back to the user's terminal.

[1019] 3. When a user asks a question via the chat function, such as "Please tell me the statistics for player X," the question is sent to the server. The generative AI analyzes the question and generates a relevant answer (for example, "Player X's batting average this season is .300"), which is then sent back to the user's terminal.

[1020] In this way, the system provides users with the information they need while watching a game in real time, resulting in a richer viewing experience.

[1021] The following describes the processing flow.

[1022] Step 1:

[1023] The user launches the app on their mobile device. The app's home screen is displayed, showing a list of currently ongoing matches.

[1024] Step 2:

[1025] The user selects the match they want to watch. A request containing information about the selected match is sent to the server.

[1026] Step 3:

[1027] The server receives the request and retrieves the relevant match information from the database. The retrieved match information is then sent back from the server to the user's terminal.

[1028] Step 4:

[1029] The user's terminal displays match information received on the screen. The displayed information includes the player list, match score, and opponent.

[1030] Step 5:

[1031] The user activates the camera function and points their mobile device at the field or players. The video captured by the camera function is analyzed on the device.

[1032] Step 6:

[1033] The device identifies players and areas from camera footage and sends the identification results to the server. The identification results include the player's location information and identification ID.

[1034] Step 7:

[1035] The server receives the identification result and retrieves the corresponding player and area information from the database. The retrieved information is then sent back from the server to the user's terminal.

[1036] Step 8:

[1037] The system overlays information received by the user's device onto the camera feed. This allows users to view detailed information about players and areas in real time.

[1038] Step 9:

[1039] Users enter questions using the chat function. For example, a question like "What is player X's batting average this season?" might be entered in chat format.

[1040] Step 10:

[1041] The terminal sends the entered question to the server. Upon receiving the question, the server uses a generative AI to generate an answer.

[1042] Step 11:

[1043] The server sends the generated response to the user's terminal. The generated response contains the necessary information, including details such as, for example, "Player X's batting average this season is .300."

[1044] Step 12:

[1045] The system displays the answers received by the user's device on the chat screen. This allows the user to see the answers to their questions in real time.

[1046] (Example 1)

[1047] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1048] Traditional sports viewing systems made it difficult for users to easily obtain detailed match and player information in real time. Furthermore, there was a lack of intuitive means for users to seek additional information to gain a deeper understanding of the game. Additionally, systems for providing appropriate answers in real time via chat were not adequately developed.

[1049] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1050] In this invention, the server includes means for a mobile terminal to acquire information about matches and players in real time; means for the server to acquire information about matches and players from a database and send it back to the mobile terminal; means for the user terminal to capture video of matches and players using its camera function and send the identification results to the server; means for the server to acquire information about identified players and areas and send it back to the mobile terminal; means for the user terminal to accept questions in chat format and send the questions to the server; means for the server to generate answers to the questions using a generative AI model and send them back to the mobile terminal; and means for generating prompt sentences to be input to the generative AI model. As a result, users can intuitively acquire detailed information about matches and players in real time and have a richer viewing experience.

[1051] A "user terminal" is a mobile device operated by a user, which runs applications and displays information about matches and players.

[1052] A "mobile device" refers to a portable electronic device such as a smartphone or tablet.

[1053] A "server" is a computer system that manages information in conjunction with a database and processes requests from user terminals.

[1054] A "database" is a data management system that stores match information and player information, and allows you to retrieve it as needed.

[1055] "Capturing" refers to the act of capturing video using the camera's functions.

[1056] "Identification results" refer to information about players and the field identified through the analysis of captured video footage.

[1057] A "generative AI model" is an artificial intelligence model that analyzes an input question and generates an appropriate answer.

[1058] A "prompt message" is a string of characters input into a generative AI model, containing instructions for the model to generate an answer.

[1059] "Overlay display" is a format that displays additional information on top of an existing screen.

[1060] A "chat format" is an interface format where questions and answers are conducted using text.

[1061] "Live data" refers to data generated in real time during the progress of a match.

[1062] This invention relates to a system that allows users to obtain real-time information about matches and players using their mobile devices, thereby providing a richer viewing experience. This system operates by combining user terminals, a server, and a database.

[1063] Specifically, the user's device is a mobile device such as a smartphone or tablet, with the application installed. This application includes a match selection function, camera function, information display function, and chat function. When the user launches the app, a list of currently ongoing matches is displayed on the home screen. The user selects the match they want to watch, and the match information is requested from the server.

[1064] The server maintains a large amount of match and player information in its database. When a match selection request is sent from a user terminal, the server retrieves the relevant match information from the database and sends it to the user terminal. Also, when a user uses the camera function to capture video of a match or players, the video is sent to the server. The server analyzes the video, retrieves information on identified players and fields from the database, and sends it back to the user terminal.

[1065] Furthermore, the user terminal can overlay the received information on the screen. When a user enters a question in text format using the chat function, the question is sent to the server. The server analyzes the question using a generative AI model and generates an appropriate answer. The generated answer, along with supplementary information retrieved from the database, is sent back to the user terminal.

[1066] For example, a user launches the app on their smartphone and selects "Match A" on the home screen. At this point, the server retrieves detailed information about the selected Match A and sends it to the user's device. If the user points their camera at "Player X" while watching the match, the captured video is sent to the server, which analyzes the video to identify Player X. Player X's profile and performance information are retrieved from the database and sent back to the user's device. Furthermore, if the user asks a question using the chat function, such as "Please tell me Player X's performance," a generative AI model analyzes the question, generates an answer such as "Player X's batting average this season is .300," and sends it to the user's device.

[1067] Examples of prompt statements include the following:

[1068] "Please tell me about match A."

[1069] "I'd like to know player X's performance this season."

[1070] "What is the current score for Match A?"

[1071] This allows users to intuitively operate the system while obtaining a wealth of information in real time, greatly improving the viewing experience. Furthermore, the introduction of generative AI models enables instant responses to chat-style questions, increasing user satisfaction.

[1072] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1073] Step 1:

[1074] The user launches the application on their mobile device. The input is the user's action (tap), and the output is the display of the application's home screen. Specifically, the application refers to an internal database, retrieves a list of currently ongoing matches, and displays it on the home screen.

[1075] Step 2:

[1076] The user selects the match they want to watch. The input is the user's selection of a match, and the output is a request for information about the selected match sent to the server. Specifically, the application sends a request to the server that includes the ID of the selected match.

[1077] Step 3:

[1078] The server processes the received request and retrieves the relevant match information from the database. The input is the match ID sent from the user's terminal, and the output is detailed information about the relevant match sent back to the user's terminal. Specifically, the server searches the database using the match ID as the key, retrieves the relevant match information (score, player list, etc.), and sends it back to the user's terminal.

[1079] Step 4:

[1080] The user activates the camera function on their device to capture video of the match or players. The input is the user's camera activation operation, and the output is the captured video sent to the server. Specifically, the user's device sends the video data acquired from the camera to the server in real time.

[1081] Step 5:

[1082] The server analyzes the received video to identify players and specific areas of the field. The input is captured video data, and the output generates information about the identified players and areas. Specifically, the server uses image recognition technology to analyze the video, retrieves the identified players and areas from a database, and sends it back to the user terminal.

[1083] Step 6:

[1084] The user terminal overlays the received information onto the camera feed. The input is player and area information sent from the server, and the output is the information displayed overlaid on the camera feed. Specifically, the user terminal analyzes the received data and displays player names and profiles on top of the camera feed in real time.

[1085] Step 7:

[1086] The user enters a question using the chat function. The input is the user's text input, and the output is the entered question sent to the server. Specifically, the user's terminal receives the user's input on the chat screen and sends that text to the server.

[1087] Step 8:

[1088] The server uses a generative AI model to analyze the question and generate an appropriate answer. The input is the question sent by the user, and the generated answer is sent back to the user's terminal as output. Specifically, the server inputs the question into the AI ​​model, retrieves supplementary information from the database based on the analysis results to generate an answer, and sends it to the user's terminal.

[1089] Step 9:

[1090] The user's terminal displays the response from the server on the chat screen. The input is the response data sent from the server, and the output is the response displayed on the user's chat screen. Specifically, the user's terminal analyzes the received response and displays it on the chat screen so that the user can confirm the response.

[1091] (Application Example 1)

[1092] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1093] In modern sports viewing, users desire more detailed and real-time information about matches and players. However, current systems lack the means to achieve this quickly and effectively. In particular, there is a need for systems that offer advanced interactive features such as eye-tracking, real-time information overlays, and question answering using generative models.

[1094] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1095] In this invention, the server includes means for the server to retrieve player and match information from a database and send it back to the mobile terminal; means for the user terminal to capture images of players and the field using its camera function and send the identification results to the server; and means for the server to generate answers to questions using a generative model and send them back to the mobile terminal. This allows the user to identify the player or area they are focusing on using an eye-tracking camera and obtain detailed information in real time. Furthermore, by utilizing a generative model, accurate answers to the user's questions can be provided quickly. In addition, because player information is overlaid on the video, the user can obtain visually intuitive information, improving the viewing experience.

[1096] A "user terminal" refers to a device operated by a user in a sports viewing system, such as a smartphone or tablet.

[1097] A "mobile device" is an electronic device that a user can carry with them, and includes smartphones, tablets, and other similar devices.

[1098] "Players and field" refers to the players and the areas within the stadium where their activities take place during a sports match.

[1099] A "server" is a central system for managing and processing information, and its role is to retrieve information from a database and provide it to user terminals.

[1100] A "database" refers to a collection of data used to store and manage information such as match details and player information in a searchable format.

[1101] The "camera function" refers to a function for capturing video, specifically the ability to photograph players and the field using the camera built into the mobile device.

[1102] "Identification results" refer to data that includes information about identified players and areas, obtained by analyzing video footage captured by cameras.

[1103] A "generative model" refers to an artificial intelligence model that has been trained using pre-trained data, and is designed to generate appropriate answers to user questions.

[1104] A "gaze recognition camera" refers to a camera function that tracks the user's gaze and identifies the object the user is focusing on.

[1105] "Overlay display" refers to a technique of displaying additional information superimposed on top of video footage, specifically displaying player information and match information in real time on video during a game.

[1106] This invention relates to a system that provides real-time information about sports viewing using a user terminal, a server, and a database.

[1107] User terminal operation

[1108] The user terminal is a mobile device operated by the user (e.g., a smartphone or tablet), and it runs when an application is launched. The application includes the following functions:

[1109] 1. Match Selection Function

[1110] The user launches the app and selects a match they want to watch from a list of matches displayed on a virtual screen. The selected match information is then requested from the server.

[1111] 2. Camera function

[1112] The system uses a camera mounted on the user's terminal to capture video of the players and the field. The captured video is analyzed, and the identification results are sent to the server.

[1113] 3. Eye-tracking camera function

[1114] The user's device has a built-in eye-tracking camera that tracks the user's gaze and identifies the player or area they are focusing on.

[1115] 4. Information Overlay Display Function

[1116] Player and match information sent from the server is overlaid on the user's terminal screen in real time. This function allows users to intuitively check detailed information about players and matches while watching.

[1117] 5. Chat function

[1118] Users input questions by voice using the in-app chat function. The entered questions are sent to the server, and a generative model generates appropriate answers.

[1119] Server operation

[1120] The server works in conjunction with the database to manage large amounts of match and player information, and is responsible for providing this information in response to requests from user terminals. Its specific operations are as follows:

[1121] 1. Providing match information

[1122] The server receives a match selection request from the user's terminal, retrieves the relevant match information from the database, and sends it to the user's terminal.

[1123] 2. Analysis of camera footage

[1124] The server analyzes camera footage transmitted from the user's terminal to identify players and specific areas of the field. Based on this identification, it retrieves relevant player and area information from the database and sends it back to the user's terminal.

[1125] 3. Analysis of gaze recognition

[1126] The server uses an eye-tracking camera to identify the player or area the user is focusing on and sends that information to the user's terminal.

[1127] 4. Response using a generative model

[1128] The system analyzes questions submitted by users through the chat function using a generative model and generates answers. The generated answers are supplemented with necessary information retrieved from the database and sent back to the user's device.

[1129] Hardware and software to be used

[1130] User devices: Smartphones, tablets, and eye-tracking cameras

[1131] Server: Database management system, generative model

[1132] Software: OpenCV (camera image analysis), requests (communication with server)

[1133] Specific example

[1134] For example, a user launches a smartphone app and selects "Match A" from the "Match List" screen. Then, when they fix their gaze on "Player X" using the eye-tracking camera, Player X's profile and performance information are displayed as an overlay. Additionally, if the user asks a voice question such as "Please tell me Player X's performance," that performance information will be displayed on the screen.

[1135] Example of a prompt

[1136] "Please tell me the statistics of player X."

[1137] "Please tell me the progress of the match."

[1138] "What is the status of player Y's injury?"

[1139] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1140] Step 1:

[1141] (Input) The user launches a sports viewing application on their smartphone and displays the match list screen.

[1142] (Specific action) The user accesses the application's home screen, and a list of currently ongoing matches is displayed.

[1143] (Data processing) Match information is requested from the server, the server retrieves that information from the database, and sends it to the user's terminal as a list of matches.

[1144] (Output) The match list is displayed on the smartphone screen.

[1145] Step 2:

[1146] (Input) The user selects the match they want to watch (for example, "Match A").

[1147] (Specific action) The user selects "Match A" from the match list by tapping or clicking.

[1148] (Data Processing) The selected match information is requested from the server, and the server retrieves that match information from the database and sends it to the user's terminal as detailed information.

[1149] (Output) Detailed information about "Match A" is displayed on the smartphone screen.

[1150] Step 3:

[1151] (Input) The user activates the camera function and captures the players or field with the camera.

[1152] (Specific operation) The smartphone camera captures video of the players and the field in real time.

[1153] (Data processing) The captured video is sent from the user's terminal to the server, which analyzes the video to identify specific parts of the players or field.

[1154] (Output) The identification result is sent from the server to the user terminal.

[1155] Step 4:

[1156] (Input) The user uses eye-tracking technology to focus on a specific player or area.

[1157] (Specific operation) Eye-tracking cameras installed in smartphones or HMDs track the user's gaze and identify the player or area they are focusing on.

[1158] (Data Processing) Eye-tracking data is sent from the user terminal to the server, and the server uses that data to retrieve information about identified players and areas from its database and sends it back to the user terminal.

[1159] (Output) Detailed information about the player or area being watched will be overlaid on the smartphone screen.

[1160] Step 5:

[1161] (Input) The user asks a question using the voice input function (for example, "Please tell me the statistics of player X").

[1162] (Specific action) The user activates the app's voice input function and inputs the question by voice.

[1163] (Data processing) The audio data is converted into text by the speech recognition function, and the question is sent from the user's terminal to the server.

[1164] The output generation model analyzes the question, generates an appropriate answer, and sends it to the user's terminal.

[1165] Step 6:

[1166] (Input) The user receives the response sent from the server and confirms it on the screen.

[1167] (Specific operation) The response sent from the server is displayed on the user's terminal.

[1168] (Data processing) The responses generated by the generative model are displayed on the user's terminal, and supplementary information is provided from the database as needed.

[1169] (Output) The information requested by the user and the answers to the questions are displayed on the screen.

[1170] Step 7:

[1171] (Input) Live status data is sent to the server.

[1172] (Specific operation) The server retrieves real-time live status from the database.

[1173] (Data processing) Live status data is sent to the user's terminal.

[1174] (Output) The live status is displayed in real time on the user's smartphone.

[1175] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1176] This invention relates to a system that provides real-time information about watching sports using a user terminal, a server, a database, and an emotion engine.

[1177] System Overview

[1178] This system aims to acquire match and player information in real time using the user's mobile device, and to provide customized information based on the user's emotions. The overall system configuration and operation are described below.

[1179] User terminal operation

[1180] A user terminal is a mobile device (smartphone, tablet, etc.) operated by the user, and it runs when an application is launched. An application includes the following functions:

[1181] 1. Match Selection Function

[1182] The user launches the app and selects a match they want to watch from the list of currently running matches displayed on the home screen. The selected match information is then requested from the server.

[1183] 2. Camera function

[1184] The user activates the camera function and points their mobile device at the field or players to capture video. The captured video is analyzed, and information about identified players and areas is sent to the server.

[1185] 3. Information display function

[1186] Player and match information sent from the server is overlaid on the terminal screen in real time. This allows users to instantly check detailed information about players and matches while watching.

[1187] 4. Chat function

[1188] Users enter questions in text format using the in-app chat function. The entered questions are sent to the server, where a generative AI generates appropriate answers.

[1189] 5. Emotion recognition function

[1190] The emotion engine recognizes the user's emotions in real time from their facial expressions and voice. The recognized emotion information is sent to the server.

[1191] Server operation

[1192] The server works in conjunction with the database to manage large amounts of match and player information, and is responsible for providing this information in response to requests from user terminals. Its specific operations are as follows:

[1193] 1. Providing match information

[1194] When the server receives a request to select a match from the user's terminal, it retrieves the relevant match information from the database and sends it back to the user's terminal.

[1195] 2. Analysis of camera footage

[1196] The system analyzes camera footage transmitted from the user's terminal to identify players and specific areas of the pitch. Based on this identification, it retrieves relevant player and area information from a database and sends it back to the terminal.

[1197] 3. Responses by generative AI

[1198] The system uses a generative AI to analyze questions sent by users through the chat function and generate answers. The generated answers are supplemented with necessary information retrieved from a database and then sent back to the user's device.

[1199] 4. Processing emotional information

[1200] The system acquires user emotion information recognized by the emotion engine and customizes the information provided to the user's device based on that information. For example, if the user is excited, it provides detailed player statistics, and if the user is relaxed, it provides stories such as past episodes.

[1201] Specific operation examples

[1202] For example, when a user launches the smartphone app and selects "Match A" from the "Match List" screen, the following sequence of events occurs:

[1203] 1. The user requests information about the selected match from the server. The server retrieves information about match A from the database and sends it to the user's terminal.

[1204] 2. When a user points their camera at "Player X" while watching a match, the captured video is sent to the server. The server analyzes the video and identifies Player X. Player X's profile, statistics, etc., are retrieved from the database and sent back to the user's terminal.

[1205] 3. When a user asks a question via the chat function, such as "Please tell me the statistics for player X," the question is sent to the server. The generative AI analyzes the question and generates a relevant answer (for example, "Player X's batting average this season is .300"), which is then sent back to the user's terminal.

[1206] 4. Simultaneously, the emotion engine recognizes the user's emotions from their facial expressions and voice, and displays detailed performance information if the user is excited, or related episodes if they are relaxed.

[1207] In this way, the system meets users' real-time information needs and further enriches the viewing experience by providing customized information based on emotions.

[1208] The following describes the processing flow.

[1209] Step 1:

[1210] The user launches the app on their mobile device. The app's home screen is displayed, showing a list of currently ongoing matches.

[1211] Step 2:

[1212] The user selects the match they want to watch. A request containing information about the selected match is sent to the server.

[1213] Step 3:

[1214] The server receives the request and retrieves the relevant match information from the database. The retrieved match information is then sent back from the server to the user's terminal.

[1215] Step 4:

[1216] The user's terminal displays match information received on the screen. The displayed information includes the player list, match score, and opponent.

[1217] Step 5:

[1218] The user activates the camera function and points their mobile device at the field or players. The video captured by the camera function is analyzed on the device.

[1219] Step 6:

[1220] The device identifies players and areas from camera footage and sends the identification results to the server. The identification results include the player's location information and identification ID.

[1221] Step 7:

[1222] The server receives the identification result and retrieves the corresponding player and area information from the database. The retrieved information is then sent back from the server to the user's terminal.

[1223] Step 8:

[1224] The system overlays information received by the user's device onto the camera feed. This allows users to view detailed information about players and areas in real time.

[1225] Step 9:

[1226] Users enter questions using the chat function. For example, a question like "What is player X's batting average this season?" might be entered in chat format.

[1227] Step 10:

[1228] The terminal sends the entered question to the server. Upon receiving the question, the server uses a generative AI to generate an answer.

[1229] Step 11:

[1230] The server sends the generated response to the user's terminal. The generated response contains the necessary information, including details such as, for example, "Player X's batting average this season is .300."

[1231] Step 12:

[1232] The system displays the answers received by the user's device on the chat screen. This allows the user to see the answers to their questions in real time.

[1233] Step 13:

[1234] The emotion engine recognizes emotions from the user's facial expressions and voice. For example, it can detect emotions such as excitement, joy, and relaxation.

[1235] Step 14:

[1236] The emotion engine sends the recognized emotion information to the server. The server receives the emotion information.

[1237] Step 15:

[1238] The server customizes the information it provides based on the user's emotional state. For example, if the user is excited, it provides detailed player statistics; if they are relaxed, it provides anecdotes about the players and the game.

[1239] Step 16:

[1240] The server sends customized information to the user's terminal. The user's terminal displays this information, allowing the user to see customized information tailored to their emotions.

[1241] This series of processes allows users to receive personalized, emotion-based information in real time, enabling them to enjoy a richer viewing experience.

[1242] (Example 2)

[1243] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1244] Traditional sports viewing systems lack the means for users to obtain real-time information on matches and players. Furthermore, they fail to provide customized information based on user sentiment, making it difficult to offer an optimized viewing experience for individual users. Additionally, they lack the means to generate quick and accurate answers when responding to user questions via chat functions. To address these issues, a more advanced and personalized information delivery system is needed.

[1245] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1246] In this invention, the server includes means for a mobile terminal to acquire player and field information in real time, means for the server to acquire player and match information from a database and send it back to the mobile terminal, means for the user terminal to capture images of players and the field using its camera function and send the identification results to the server, means for the server to generate answers to questions using a generated AI model and send them back to the mobile terminal, means for the user terminal to analyze the user's facial expressions and voice to acquire emotional information and send the emotional information to the server, and means for the server to generate customized information based on the emotional data and send it back to the mobile terminal. As a result, users can acquire match and player information in real time, and personalized information tailored to the user's emotions can be provided. In addition, the viewing experience is improved because information is provided quickly and accurately through the chat function.

[1247] A "user terminal" is a portable information terminal operated by a user, which acquires and displays information using a specific application.

[1248] A "mobile device" refers to a portable information and communication device such as a smartphone or tablet, which is used to provide users with real-time information on matches and players.

[1249] A "server" is a computer system that communicates with user terminals via a network to process data and provide information.

[1250] A "database" is an information management system that stores match information and player information, and allows a server to retrieve this information as needed.

[1251] The "camera function" is a feature that allows the user's device to take pictures of players and the field, capture video footage, and send that video data to a server for analysis.

[1252] A "generative AI model" is an algorithm or program that uses artificial intelligence to analyze a user's question and generate an appropriate answer.

[1253] An "emotion engine" is a combination of software or hardware that analyzes a user's emotions from their facial expressions and voice and acquires that data.

[1254] "Identification results" refer to information about specific players or parts of the field obtained by analyzing video captured by the camera function.

[1255] "Overlay display" is a technology that displays additional information by superimposing it on the video or image currently displayed on the user's device's screen.

[1256] A "chat-style question" is an inquiry that a user enters in text format and sends to the server.

[1257] "Customized information" refers to information optimized for each individual user's situation, based on their emotional data.

[1258] This invention relates to a system that provides real-time information for sports viewing using a user terminal, server, database, and emotion engine. The system aims to acquire match information and player information in real time using the user's mobile terminal, and further to provide customized information based on the user's emotions.

[1259] System Configuration

[1260] User terminal hardware and software

[1261] The user terminal will be a mobile device such as a smartphone or tablet. This terminal will have a dedicated application installed, which will include features such as match selection, camera function, chat function, and emotion recognition function.

[1262] Server hardware and software

[1263] The server uses a high-performance network server to manage large amounts of data. It interacts with databases that store player and match information and responds to user requests. The server has a generative AI model and an emotion engine implemented in it.

[1264] database

[1265] The database is responsible for storing and managing match and player information. This includes player profiles, statistics, and match progress.

[1266] Emotional Engine

[1267] An emotion engine is software or hardware that analyzes a user's facial expressions and voice to recognize their emotional state in real time. Its role is to acquire the user's emotional data and transmit it to a server.

[1268] Program processing

[1269] Match selection function

[1270] The user launches the application and selects a match they want to watch from the match list. The selected match information is requested from the server. The server retrieves the corresponding match information from the database and sends it back to the user's terminal.

[1271] Camera function

[1272] The user activates the camera function and points their mobile device at the field or players to capture video. The captured video is sent to a server, where it is analyzed. Based on the analysis results, the server retrieves information on identified players and areas from its database and sends it back to the user's device.

[1273] Information display function

[1274] Player and match information sent from the server is overlaid on the user's terminal screen in real time. This allows users to instantly check detailed information about players and matches while watching.

[1275] Chat function

[1276] Users enter questions in text format using the in-app chat function. The entered questions are sent to the server and analyzed by a generative AI model. The generative AI model generates appropriate answers to the questions, retrieves supplementary information from the database, and sends it back to the user's device.

[1277] Emotion recognition function

[1278] The emotion engine recognizes the user's emotions in real time from their facial expressions and voice. The recognized emotion information is sent to a server, which generates customized information based on that information and sends it back to the user's terminal. For example, if the user is excited, detailed player performance information is provided; if they are relaxed, relevant anecdotal information is provided.

[1279] Examples of specific actions

[1280] For example, when a user launches the smartphone app and selects "Match A" from the "Match List" screen, the following sequence of events occurs:

[1281] 1. The user requests information about "Match A" from the server. The server retrieves the information about Match A from the database and sends it to the user's terminal.

[1282] 2. When a user points their camera at "Player X" while watching a match, the captured video is sent to the server. The server analyzes the video and identifies Player X. Player X's profile, statistics, etc., are retrieved from the database and sent back to the user's terminal.

[1283] 3. When a user asks a question using the chat function, such as "Please tell me the statistics for player X," the question is sent to the server. The generating AI model analyzes the question and generates a relevant answer (for example, "Player X's batting average this season is .300"), which is then sent back to the user's terminal.

[1284] 4. Simultaneously, the emotion engine recognizes the user's emotions from their facial expressions and voice, and displays detailed performance information if the user is excited, or related episodes if they are relaxed.

[1285] Example of a prompt

[1286] Examples of prompt statements to input into a generative AI model include the following:

[1287] 1. Example of a prompt when selecting a match

[1288] The user selected "Match A" from the match list. Retrieve the information for Match A and send it back to the user.

[1289] 2. Example prompts during camera image analysis

[1290] A user has taken a picture of player X with their camera. Please analyze the video footage and provide player X's profile and latest statistics.

[1291] 3. Examples of prompts when asking questions in a chat.

[1292] A user asked, "What are player Y's statistics for this season?" Please generate an answer based on player Y's latest statistics.

[1293] 4. Examples of prompts during emotion recognition

[1294] The user is excited. In this case, prioritize providing detailed performance information. If the user is relaxed, provide relevant episode information.

[1295] In this way, this system can further enhance the individual viewing experience by meeting users' real-time information needs and providing emotion-based, customized information.

[1296] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1297] Step 1:

[1298] Match Selection

[1299] Input: The user selects the match they want to watch from the match list on the application's home screen.

[1300] Data processing: Generate user selection information (match ID).

[1301] Output: The match ID is requested from the server.

[1302] Specific action: The user taps "Match A". The selected match ID is sent to the server.

[1303] Step 2:

[1304] Server-based acquisition of match information

[1305] Input: Match ID

[1306] Data processing: Search and retrieve relevant match information from the database.

[1307] Output: Match information is sent to the user's terminal.

[1308] Specific operation: The server accesses the database, retrieves information about match A (date, time, location, participating teams, etc.), and sends it back to the user's terminal.

[1309] Step 3:

[1310] Camera video capture and transmission

[1311] Input: Camera footage

[1312] Data processing: Capture video data and send it to the server.

[1313] Output: Video data is sent to the server.

[1314] Specific operation: The user takes a picture of player X using a mobile device, and the video is sent to the server.

[1315] Step 4:

[1316] Video analysis and information return by the server

[1317] Input: Video data

[1318] Data processing: Analyze the video to identify player X and retrieve related information from the database.

[1319] Output: Player X's information is sent back to the user's terminal.

[1320] Specific operation: The server analyzes the video data and identifies player X. It retrieves player X's profile and latest statistics from the database and sends them back to the user's terminal.

[1321] Step 5:

[1322] Information overlay display

[1323] Input: Player information, match information

[1324] Data processing: The received information is displayed as an overlay on the screen.

[1325] Output: Displayed in real time on the user's terminal.

[1326] Specific operation: When the user is filming player X, player X's profile and performance information will be overlaid on the video.

[1327] Step 6:

[1328] Send a chat question

[1329] Input: Chat text

[1330] Data processing: Send chat input text to the server.

[1331] Output: The question content is sent to the server.

[1332] Specific action: The user types "Please tell me the stats of player X" in the chat and sends it to the server.

[1333] Step 7:

[1334] Answer generation using a generative AI model

[1335] Input: Question content

[1336] Data processing: Analyze the question, retrieve the necessary information from the database, and generate the answer.

[1337] Output: The generated response is sent to the user's terminal.

[1338] Specific operation: The generative AI model analyzes the question and generates an answer, such as "Player X's batting average this season is .300," which is then sent to the user's terminal.

[1339] Step 8:

[1340] Sending emotion recognition data

[1341] Input: User facial expression data, voice data

[1342] Data processing: Analyze facial expressions and voice to generate emotion data.

[1343] Output: Emotional data is sent to the server.

[1344] Specific operation: The system analyzes the user's face and voice, and the emotion engine generates emotional data such as excitement or relaxation, which is then sent to the server.

[1345] Step 9:

[1346] Server-based customization of information

[1347] Input: Sentiment data

[1348] Data processing: Customize the information provided based on emotional data.

[1349] Output: Customization information is sent to the user's terminal.

[1350] Specific operation: The server selects detailed performance and episode information based on the user's emotional information and sends it to the user's terminal.

[1351] By having each processing step work in conjunction in this way, users can obtain match and player information in real time and enjoy customized information tailored to their individual emotions.

[1352] (Application Example 2)

[1353] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1354] Modern users demand more sophisticated information in experiences such as shopping and watching sports. Furthermore, providing timely and relevant information based on user emotions can dramatically improve the user experience. However, existing systems struggle to adapt to users' real-time emotional states and the information retrieval process is often cumbersome.

[1355] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1356] In this invention, the server includes means for acquiring information on objects and events from a database and sending it back to a mobile terminal; means for acquiring information on identified objects and areas and sending it back to the mobile terminal; means for generating answers to questions using a generative AI model and sending them back to the mobile terminal; and means for acquiring emotional information and providing the user terminal with customized information based on the user's emotions. As a result, the user can easily acquire detailed information on products and objects on the spot, and furthermore, receive optimal information in real time based on the user's emotions.

[1357] A "user terminal" is a device equipped with communication functions that a user uses to acquire and manipulate information.

[1358] A "mobile device" is a device that a user can carry with them, equipped with a camera and communication functions, and capable of displaying and transmitting information.

[1359] A "server" is a computer system that manages and processes information in conjunction with a database and provides that information in response to requests from user terminals.

[1360] A "database" is a system for organizing, storing, and searching information, and is a storage medium where match information, player information, product information, and so on are stored.

[1361] A "subject" is an item or person of interest to the user, which is captured and identified in the camera footage.

[1362] A "domain" refers to a specific place or area, specifically the field or area where an object exists.

[1363] A "generative AI model" is an artificial intelligence model that generates appropriate answers or information in response to given input.

[1364] An "emotion engine" is an algorithm that analyzes a user's facial expressions and voice data to recognize and determine their emotional state.

[1365] "Identification results" refer to information about objects and areas obtained when analyzing camera footage.

[1366] "Customized information" refers to information that is optimized and provided according to the user's emotions and circumstances.

[1367] This invention is a system that allows users in physical stores to obtain product and promotional information in real time using smartphones or smart glasses, and further provides a customized shopping experience based on the user's emotions.

[1368] System Overview

[1369] This system includes user terminals (smartphones and smart glasses), a server, a database, an emotion engine, and a generative AI model. The user terminal provides functions such as retrieving product information, answering questions, and emotion recognition through applications. The server works in conjunction with the database to manage vast amounts of product and promotional information, providing it in response to requests from user terminals.

[1370] User terminal operation

[1371] The user terminal is a smartphone or smart glasses operated by the user, and various functions are performed by launching an application. The application includes the following functions:

[1372] 1. Product Selection Function

[1373] The user launches the app and selects the product they want to find from the product list or category within the store. The selected product information is requested from the server.

[1374] 2. Camera function

[1375] Users obtain product information by scanning items using their smartphones or smart glasses. The camera footage is analyzed, and the information of the identified products is sent to a server.

[1376] 3. Information display function

[1377] Product information and promotional information sent from the server are overlaid in real time on the terminal's screen or visual display.

[1378] 4. Chat function

[1379] Users enter questions within the app, and an AI model generates appropriate answers. For example, the AI ​​generates an answer to a question such as, "Are there any similar products to this one?"

[1380] 5. Emotion recognition function

[1381] The emotion engine recognizes the user's emotions in real time from their facial expressions and voice. The recognized emotion information is sent to the server, and the information provided to the user is customized based on that information.

[1382] Server operation

[1383] The server interacts with the database and is responsible for managing and providing information about objects and events. Its specific operations are as follows:

[1384] 1. Providing product information

[1385] When the server receives a product selection request from the user's terminal, it retrieves the relevant product information from the database and sends it back to the user's terminal.

[1386] 2. Analysis of camera footage

[1387] The system analyzes camera footage transmitted from the user's terminal to identify products. Based on the identification results, it retrieves relevant product and promotional information from a database and sends it back to the terminal.

[1388] 3. Responses by generative AI models

[1389] The system analyzes questions submitted by users via the chat function using an AI model and generates answers. The generated answers are then supplemented with necessary information retrieved from a database and sent back to the user's device.

[1390] 4. Processing emotional information

[1391] The system acquires user emotion information recognized by the emotion engine and customizes the information provided to the user's device based on that information. For example, if the user appears happy, it provides information about special promotions; if the user appears surprised, it provides detailed product information and reviews.

[1392] Specific operation examples

[1393] For example, if a user uses smart glasses to find a specific product, the following sequence of events will occur:

[1394] 1. When a user scans a product with the smart glasses' camera, the captured image is sent to the server. The server analyzes the image and identifies the product. Detailed information and promotional information for the identified product are retrieved from the database and sent back to the user's device.

[1395] 2. When a user asks "Are there any similar products to this one?" using the chat function, the question is sent to the server. The generation AI model analyzes the question, generates a relevant answer (for example, "There is a similar product called XX"), and sends it back to the user's device.

[1396] 3. Simultaneously, the emotion engine recognizes the user's emotions from their facial expressions and voice. If the user appears happy, it displays additional promotional information; if they are interested, it displays detailed product information and reviews from other users.

[1397] In this way, the system meets users' real-time information needs and further enriches the shopping experience by providing customized information based on their emotions.

[1398] Prompt example

[1399] "Please add a feature to this shopping assistant app that provides information based on real-time emotions. When a user views a particular product, it should display customized information based on the user's emotions and past purchase history."

[1400] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1401] Step 1:

[1402] Users scan products using their smartphones or smart glasses.

[1403] Input: The user takes a picture of the product with their camera.

[1404] Output: Captured product video is obtained.

[1405] Specific operation: The user points the camera at the product and presses the capture button. The smart device's camera captures the image and saves the image data locally.

[1406] Step 2:

[1407] The device sends the captured video to the server.

[1408] Input: Data from captured product video.

[1409] Output: Product video data is sent to the server.

[1410] Specific operation: The application retrieves the captured video data and sends it to the server via the network.

[1411] Step 3:

[1412] The server analyzes the received video data to identify the product.

[1413] Input: Product video data sent to the server.

[1414] Output: Identified product ID and related information.

[1415] Specific operation: The server applies a video analysis algorithm to identify the product. As a result of the identification, it extracts a product ID and retrieves product information from the database based on that ID.

[1416] Step 4:

[1417] The server retrieves product information from the database and sends it back to the user's terminal.

[1418] Input: Identified product ID.

[1419] Output: Corresponding product information data.

[1420] Specific operation: The server executes a database query based on the product ID and retrieves the corresponding product information (price, stock, description, etc.). The retrieved data is then sent back to the user's terminal.

[1421] Step 5:

[1422] The device displays the product information it has received to the user.

[1423] Input: Received product information data.

[1424] Output: Product information is displayed on the user's terminal screen.

[1425] Specific operation: The application analyzes the received product information data and displays detailed information, including product name, price, and description, as an overlay on the screen.

[1426] Step 6:

[1427] Users enter their questions using the in-app chat function.

[1428] Input: User's question text.

[1429] Output: The question text is sent to the server.

[1430] Specific operation: The user types a question in the chat window and presses the send button. The application sends the question text to the server.

[1431] Step 7:

[1432] The server uses a generated AI model to produce an answer to the question and sends it back to the user's terminal.

[1433] Input: User's question text.

[1434] Output: Generated answer text.

[1435] Specific operation: The server inputs the question text into an AI model, which generates an appropriate answer. The generated answer is then supplemented with information retrieved from a database and sent back to the user's terminal.

[1436] Step 8:

[1437] The user's terminal displays the generated response in the chat window.

[1438] Input: Generated response text.

[1439] Output: Chat response displayed on the user's device.

[1440] Specific operation: The application displays the received response text in the chat window for the user to review.

[1441] Step 9:

[1442] The device captures the user's facial expressions and voice, analyzes them using an emotion engine, and sends the emotional information to the server.

[1443] Input: User's facial expressions and voice data.

[1444] Output: Analyzed emotion information.

[1445] Specific operation: The smart device's camera and microphone are used to capture the user's facial expressions and voice, and the emotion engine analyzes this data. The resulting emotion information is then sent to the server.

[1446] Step 10:

[1447] The server generates customized information based on emotional data and sends it back to the user's terminal.

[1448] Input: Analyzed emotion information.

[1449] Output: Customized informational data.

[1450] Specific operation: The server analyzes the received emotional information and generates optimal product and promotional information based on the user's emotional state. The generated information is then sent back to the user's terminal.

[1451] Step 11:

[1452] The device provides users with customized information.

[1453] Input: Customized information data.

[1454] Output: Customization information displayed to the user.

[1455] Specific operation: The application displays the customization information it has received and provides the user with appropriate product suggestions and promotions.

[1456] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1457] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1458] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1459] [Fourth Embodiment]

[1460] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1461] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1462] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1463] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1464] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1465] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1466] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1467] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1468] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1469] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1470] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1471] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1472] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1473] This invention relates to a system that provides real-time information about watching sports using a user terminal, a server, and a database.

[1474] System Overview

[1475] This system aims to enhance the viewing experience by allowing users to obtain match and player information in real time using their mobile devices. The overall system configuration and operation are described below.

[1476] User terminal operation

[1477] A user terminal is a mobile device (smartphone, tablet, etc.) operated by the user, and it runs when an application is launched. An application includes the following functions:

[1478] 1. Match Selection Function

[1479] The user launches the app and selects the match they want to watch from the list of currently ongoing matches displayed on the home screen. The selected match information is then requested from the server.

[1480] 2. Camera function

[1481] Users activate the camera function and capture video by pointing their mobile device at the field or players. The captured video is analyzed, and information about identified players and areas is sent to the server.

[1482] 3. Information display function

[1483] Player and match information sent from the server is overlaid on the terminal screen in real time. This allows users to instantly check detailed information about players and matches while watching.

[1484] 4. Chat function

[1485] Users enter their questions in text format using the in-app chat function. The entered questions are sent to the server, where a generative AI generates appropriate answers.

[1486] Server operation

[1487] The server works in conjunction with the database to manage large amounts of match and player information, and is responsible for providing this information in response to requests from user terminals. Its specific operations are as follows:

[1488] 1. Providing match information

[1489] When the server receives a request to select a match from the user's terminal, it retrieves the relevant match information from the database and sends it back to the user's terminal.

[1490] 2. Analysis of camera footage

[1491] The system analyzes camera footage transmitted from the user's terminal to identify players and specific areas of the pitch. Based on this identification, it retrieves relevant player and area information from a database and sends it back to the terminal.

[1492] 3. Responses by generative AI

[1493] The system uses a generative AI to analyze questions sent by users through the chat function and generate answers. The generated answers are supplemented with necessary information retrieved from a database and then sent back to the user's device.

[1494] Specific operation examples

[1495] For example, when a user launches the smartphone app and selects "Match A" from the "Match List" screen, the following sequence of events occurs:

[1496] 1. The user requests information about the selected match from the server. The server retrieves information about match A from the database and sends it to the user's terminal.

[1497] 2. When a user points their camera at "Player X" while watching a match, the captured video is sent to the server. The server analyzes the video and identifies Player X. Player X's profile, statistics, etc., are retrieved from the database and sent back to the user's terminal.

[1498] 3. When a user asks a question via the chat function, such as "Please tell me the statistics for player X," the question is sent to the server. The generative AI analyzes the question and generates a relevant answer (for example, "Player X's batting average this season is .300"), which is then sent back to the user's terminal.

[1499] In this way, the system provides users with the information they need while watching a game in real time, resulting in a richer viewing experience.

[1500] The following describes the processing flow.

[1501] Step 1:

[1502] The user launches the app on their mobile device. The app's home screen is displayed, showing a list of currently ongoing matches.

[1503] Step 2:

[1504] The user selects the match they want to watch. A request containing information about the selected match is sent to the server.

[1505] Step 3:

[1506] The server receives the request and retrieves the relevant match information from the database. The retrieved match information is then sent back from the server to the user's terminal.

[1507] Step 4:

[1508] The user's terminal displays match information received on the screen. The displayed information includes the player list, match score, and opponent.

[1509] Step 5:

[1510] The user activates the camera function and points their mobile device at the field or players. The video captured by the camera function is analyzed on the device.

[1511] Step 6:

[1512] The device identifies players and areas from camera footage and sends the identification results to the server. The identification results include the player's location information and identification ID.

[1513] Step 7:

[1514] The server receives the identification result and retrieves the corresponding player and area information from the database. The retrieved information is then sent back from the server to the user's terminal.

[1515] Step 8:

[1516] The system overlays information received by the user's device onto the camera feed. This allows users to view detailed information about players and areas in real time.

[1517] Step 9:

[1518] Users enter questions using the chat function. For example, a question like "What is player X's batting average this season?" might be entered in chat format.

[1519] Step 10:

[1520] The terminal sends the entered question to the server. Upon receiving the question, the server uses a generative AI to generate an answer.

[1521] Step 11:

[1522] The server sends the generated response to the user's terminal. The generated response contains the necessary information, including details such as, for example, "Player X's batting average this season is .300."

[1523] Step 12:

[1524] The system displays the answers received by the user's device on the chat screen. This allows the user to see the answers to their questions in real time.

[1525] (Example 1)

[1526] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1527] Traditional sports viewing systems made it difficult for users to easily obtain detailed match and player information in real time. Furthermore, there was a lack of intuitive means for users to seek additional information to gain a deeper understanding of the game. Additionally, systems for providing appropriate answers in real time via chat were not adequately developed.

[1528] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1529] In this invention, the server includes means for a mobile terminal to acquire information about matches and players in real time; means for the server to acquire information about matches and players from a database and send it back to the mobile terminal; means for the user terminal to capture video of matches and players using its camera function and send the identification results to the server; means for the server to acquire information about identified players and areas and send it back to the mobile terminal; means for the user terminal to accept questions in chat format and send the questions to the server; means for the server to generate answers to the questions using a generative AI model and send them back to the mobile terminal; and means for generating prompt sentences to be input to the generative AI model. As a result, users can intuitively acquire detailed information about matches and players in real time and have a richer viewing experience.

[1530] A "user terminal" is a mobile device operated by a user, which runs applications and displays information about matches and players.

[1531] A "mobile device" refers to a portable electronic device such as a smartphone or tablet.

[1532] A "server" is a computer system that manages information in conjunction with a database and processes requests from user terminals.

[1533] A "database" is a data management system that stores match information and player information, and allows you to retrieve it as needed.

[1534] "Capturing" refers to the act of capturing video using the camera's functions.

[1535] "Identification results" refer to information about players and the field identified through the analysis of captured video footage.

[1536] A "generative AI model" is an artificial intelligence model that analyzes an input question and generates an appropriate answer.

[1537] A "prompt message" is a string of characters input into a generative AI model, containing instructions for the model to generate an answer.

[1538] "Overlay display" is a format that displays additional information on top of an existing screen.

[1539] A "chat format" is an interface format where questions and answers are conducted using text.

[1540] "Live data" refers to data generated in real time during the progress of a match.

[1541] This invention relates to a system that allows users to obtain real-time information about matches and players using their mobile devices, thereby providing a richer viewing experience. This system operates by combining user terminals, a server, and a database.

[1542] Specifically, the user's device is a mobile device such as a smartphone or tablet, with the application installed. This application includes a match selection function, camera function, information display function, and chat function. When the user launches the app, a list of currently ongoing matches is displayed on the home screen. The user selects the match they want to watch, and the match information is requested from the server.

[1543] The server maintains a large amount of match and player information in its database. When a match selection request is sent from a user terminal, the server retrieves the relevant match information from the database and sends it to the user terminal. Also, when a user uses the camera function to capture video of a match or players, the video is sent to the server. The server analyzes the video, retrieves information on identified players and fields from the database, and sends it back to the user terminal.

[1544] Furthermore, the user terminal can overlay the received information on the screen. When a user enters a question in text format using the chat function, the question is sent to the server. The server analyzes the question using a generative AI model and generates an appropriate answer. The generated answer, along with supplementary information retrieved from the database, is sent back to the user terminal.

[1545] For example, a user launches the app on their smartphone and selects "Match A" on the home screen. At this point, the server retrieves detailed information about the selected Match A and sends it to the user's device. If the user points their camera at "Player X" while watching the match, the captured video is sent to the server, which analyzes the video to identify Player X. Player X's profile and performance information are retrieved from the database and sent back to the user's device. Furthermore, if the user asks a question using the chat function, such as "Please tell me Player X's performance," a generative AI model analyzes the question, generates an answer such as "Player X's batting average this season is .300," and sends it to the user's device.

[1546] Examples of prompt statements include the following:

[1547] "Please tell me about match A."

[1548] "I'd like to know player X's performance this season."

[1549] "What is the current score for Match A?"

[1550] This allows users to intuitively operate the system while obtaining a wealth of information in real time, greatly improving the viewing experience. Furthermore, the introduction of generative AI models enables instant responses to chat-style questions, increasing user satisfaction.

[1551] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1552] Step 1:

[1553] The user launches the application on their mobile device. The input is the user's action (tap), and the output is the display of the application's home screen. Specifically, the application refers to an internal database, retrieves a list of currently ongoing matches, and displays it on the home screen.

[1554] Step 2:

[1555] The user selects the match they want to watch. The input is the user's selection of a match, and the output is a request for information about the selected match sent to the server. Specifically, the application sends a request to the server that includes the ID of the selected match.

[1556] Step 3:

[1557] The server processes the received request and retrieves the relevant match information from the database. The input is the match ID sent from the user's terminal, and the output is detailed information about the relevant match sent back to the user's terminal. Specifically, the server searches the database using the match ID as the key, retrieves the relevant match information (score, player list, etc.), and sends it back to the user's terminal.

[1558] Step 4:

[1559] The user activates the camera function on their device to capture video of the match or players. The input is the user's camera activation operation, and the output is the captured video sent to the server. Specifically, the user's device sends the video data acquired from the camera to the server in real time.

[1560] Step 5:

[1561] The server analyzes the received video to identify players and specific areas of the field. The input is captured video data, and the output generates information about the identified players and areas. Specifically, the server uses image recognition technology to analyze the video, retrieves the identified players and areas from a database, and sends it back to the user terminal.

[1562] Step 6:

[1563] The user terminal overlays the received information onto the camera feed. The input is player and area information sent from the server, and the output is the information displayed overlaid on the camera feed. Specifically, the user terminal analyzes the received data and displays player names and profiles on top of the camera feed in real time.

[1564] Step 7:

[1565] The user enters a question using the chat function. The input is the user's text input, and the output is the entered question sent to the server. Specifically, the user's terminal receives the user's input on the chat screen and sends that text to the server.

[1566] Step 8:

[1567] The server uses a generative AI model to analyze the question and generate an appropriate answer. The input is the question sent by the user, and the generated answer is sent back to the user's terminal as output. Specifically, the server inputs the question into the AI ​​model, retrieves supplementary information from the database based on the analysis results to generate an answer, and sends it to the user's terminal.

[1568] Step 9:

[1569] The user's terminal displays the response from the server on the chat screen. The input is the response data sent from the server, and the output is the response displayed on the user's chat screen. Specifically, the user's terminal analyzes the received response and displays it on the chat screen so that the user can confirm the response.

[1570] (Application Example 1)

[1571] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1572] In modern sports viewing, users desire more detailed and real-time information about matches and players. However, current systems lack the means to achieve this quickly and effectively. In particular, there is a need for systems that offer advanced interactive features such as eye-tracking, real-time information overlays, and question answering using generative models.

[1573] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1574] In this invention, the server includes means for the server to retrieve player and match information from a database and send it back to the mobile terminal; means for the user terminal to capture images of players and the field using its camera function and send the identification results to the server; and means for the server to generate answers to questions using a generative model and send them back to the mobile terminal. This allows the user to identify the player or area they are focusing on using an eye-tracking camera and obtain detailed information in real time. Furthermore, by utilizing a generative model, accurate answers to the user's questions can be provided quickly. In addition, because player information is overlaid on the video, the user can obtain visually intuitive information, improving the viewing experience.

[1575] A "user terminal" refers to a device operated by a user in a sports viewing system, such as a smartphone or tablet.

[1576] A "mobile device" is an electronic device that a user can carry with them, and includes smartphones, tablets, and other similar devices.

[1577] "Players and field" refers to the players and the areas within the stadium where their activities take place during a sports match.

[1578] A "server" is a central system for managing and processing information, and its role is to retrieve information from a database and provide it to user terminals.

[1579] A "database" refers to a collection of data used to store and manage information such as match details and player information in a searchable format.

[1580] The "camera function" refers to a function for capturing video, specifically the ability to photograph players and the field using the camera built into the mobile device.

[1581] "Identification results" refer to data that includes information about identified players and areas, obtained by analyzing video footage captured by cameras.

[1582] A "generative model" refers to an artificial intelligence model that has been trained using pre-trained data, and is designed to generate appropriate answers to user questions.

[1583] A "gaze recognition camera" refers to a camera function that tracks the user's gaze and identifies the object the user is focusing on.

[1584] "Overlay display" refers to a technique of displaying additional information superimposed on top of video footage, specifically displaying player information and match information in real time on video during a game.

[1585] This invention relates to a system that provides real-time information about sports viewing using a user terminal, a server, and a database.

[1586] User terminal operation

[1587] The user terminal is a mobile device operated by the user (e.g., a smartphone or tablet), and it runs when an application is launched. The application includes the following functions:

[1588] 1. Match Selection Function

[1589] The user launches the app and selects a match they want to watch from a list of matches displayed on a virtual screen. The selected match information is then requested from the server.

[1590] 2. Camera function

[1591] The system uses a camera mounted on the user's terminal to capture video of the players and the field. The captured video is analyzed, and the identification results are sent to the server.

[1592] 3. Eye-tracking camera function

[1593] The user's device has a built-in eye-tracking camera that tracks the user's gaze and identifies the player or area they are focusing on.

[1594] 4. Information Overlay Display Function

[1595] Player and match information sent from the server is overlaid on the user's terminal screen in real time. This function allows users to intuitively check detailed information about players and matches while watching.

[1596] 5. Chat function

[1597] Users input questions by voice using the in-app chat function. The entered questions are sent to the server, and a generative model generates appropriate answers.

[1598] Server operation

[1599] The server works in conjunction with the database to manage large amounts of match and player information, and is responsible for providing this information in response to requests from user terminals. Its specific operations are as follows:

[1600] 1. Providing match information

[1601] The server receives a match selection request from the user's terminal, retrieves the relevant match information from the database, and sends it to the user's terminal.

[1602] 2. Analysis of camera footage

[1603] The server analyzes camera footage transmitted from the user's terminal to identify players and specific areas of the field. Based on this identification, it retrieves relevant player and area information from the database and sends it back to the user's terminal.

[1604] 3. Analysis of gaze recognition

[1605] The server uses an eye-tracking camera to identify the player or area the user is focusing on and sends that information to the user's terminal.

[1606] 4. Response using a generative model

[1607] The system analyzes questions submitted by users through the chat function using a generative model and generates answers. The generated answers are supplemented with necessary information retrieved from the database and sent back to the user's device.

[1608] Hardware and software to be used

[1609] User devices: Smartphones, tablets, and eye-tracking cameras

[1610] Server: Database management system, generative model

[1611] Software: OpenCV (camera image analysis), requests (communication with server)

[1612] Specific example

[1613] For example, a user launches a smartphone app and selects "Match A" from the "Match List" screen. Then, when they fix their gaze on "Player X" using the eye-tracking camera, Player X's profile and performance information are displayed as an overlay. Additionally, if the user asks a voice question such as "Please tell me Player X's performance," that performance information will be displayed on the screen.

[1614] Example of a prompt

[1615] "Please tell me the statistics of player X."

[1616] "Please tell me the progress of the match."

[1617] "What is the status of player Y's injury?"

[1618] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1619] Step 1:

[1620] (Input) The user launches a sports viewing application on their smartphone and displays the match list screen.

[1621] (Specific action) The user accesses the application's home screen, and a list of currently ongoing matches is displayed.

[1622] (Data processing) Match information is requested from the server, the server retrieves that information from the database, and sends it to the user's terminal as a list of matches.

[1623] (Output) The match list is displayed on the smartphone screen.

[1624] Step 2:

[1625] (Input) The user selects the match they want to watch (for example, "Match A").

[1626] (Specific action) The user selects "Match A" from the match list by tapping or clicking.

[1627] (Data Processing) The selected match information is requested from the server, and the server retrieves that match information from the database and sends it to the user's terminal as detailed information.

[1628] (Output) Detailed information about "Match A" is displayed on the smartphone screen.

[1629] Step 3:

[1630] (Input) The user activates the camera function and captures the players or field with the camera.

[1631] (Specific operation) The smartphone camera captures video of the players and the field in real time.

[1632] (Data processing) The captured video is sent from the user's terminal to the server, which analyzes the video to identify specific parts of the players or field.

[1633] (Output) The identification result is sent from the server to the user terminal.

[1634] Step 4:

[1635] (Input) The user uses eye-tracking technology to focus on a specific player or area.

[1636] (Specific operation) Eye-tracking cameras installed in smartphones or HMDs track the user's gaze and identify the player or area they are focusing on.

[1637] (Data Processing) Eye-tracking data is sent from the user terminal to the server, and the server uses that data to retrieve information about identified players and areas from its database and sends it back to the user terminal.

[1638] (Output) Detailed information about the player or area being watched will be overlaid on the smartphone screen.

[1639] Step 5:

[1640] (Input) The user asks a question using the voice input function (for example, "Please tell me the statistics of player X").

[1641] (Specific action) The user activates the app's voice input function and inputs the question by voice.

[1642] (Data processing) The audio data is converted into text by the speech recognition function, and the question is sent from the user's terminal to the server.

[1643] The output generation model analyzes the question, generates an appropriate answer, and sends it to the user's terminal.

[1644] Step 6:

[1645] (Input) The user receives the response sent from the server and confirms it on the screen.

[1646] (Specific operation) The response sent from the server is displayed on the user's terminal.

[1647] (Data processing) The responses generated by the generative model are displayed on the user's terminal, and supplementary information is provided from the database as needed.

[1648] (Output) The information requested by the user and the answers to the questions are displayed on the screen.

[1649] Step 7:

[1650] (Input) Live status data is sent to the server.

[1651] (Specific operation) The server retrieves real-time live status from the database.

[1652] (Data processing) Live status data is sent to the user's terminal.

[1653] (Output) The live status is displayed in real time on the user's smartphone.

[1654] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1655] This invention relates to a system that provides real-time information about watching sports using a user terminal, a server, a database, and an emotion engine.

[1656] System Overview

[1657] This system aims to acquire match and player information in real time using the user's mobile device, and to provide customized information based on the user's emotions. The overall system configuration and operation are described below.

[1658] User terminal operation

[1659] A user terminal is a mobile device (smartphone, tablet, etc.) operated by the user, and it runs when an application is launched. An application includes the following functions:

[1660] 1. Match Selection Function

[1661] The user launches the app and selects a match they want to watch from the list of currently running matches displayed on the home screen. The selected match information is then requested from the server.

[1662] 2. Camera function

[1663] The user activates the camera function and points their mobile device at the field or players to capture video. The captured video is analyzed, and information about identified players and areas is sent to the server.

[1664] 3. Information display function

[1665] Player and match information sent from the server is overlaid on the terminal screen in real time. This allows users to instantly check detailed information about players and matches while watching.

[1666] 4. Chat function

[1667] Users enter questions in text format using the in-app chat function. The entered questions are sent to the server, where a generative AI generates appropriate answers.

[1668] 5. Emotion recognition function

[1669] The emotion engine recognizes the user's emotions in real time from their facial expressions and voice. The recognized emotion information is sent to the server.

[1670] Server operation

[1671] The server works in conjunction with the database to manage large amounts of match and player information, and is responsible for providing this information in response to requests from user terminals. Its specific operations are as follows:

[1672] 1. Providing match information

[1673] When the server receives a request to select a match from the user's terminal, it retrieves the relevant match information from the database and sends it back to the user's terminal.

[1674] 2. Analysis of camera footage

[1675] The system analyzes camera footage transmitted from the user's terminal to identify players and specific areas of the pitch. Based on this identification, it retrieves relevant player and area information from a database and sends it back to the terminal.

[1676] 3. Responses by generative AI

[1677] The system uses a generative AI to analyze questions sent by users through the chat function and generate answers. The generated answers are supplemented with necessary information retrieved from a database and then sent back to the user's device.

[1678] 4. Processing emotional information

[1679] The system acquires user emotion information recognized by the emotion engine and customizes the information provided to the user's device based on that information. For example, if the user is excited, it provides detailed player statistics, and if the user is relaxed, it provides stories such as past episodes.

[1680] Specific operation examples

[1681] For example, when a user launches the smartphone app and selects "Match A" from the "Match List" screen, the following sequence of events occurs:

[1682] 1. The user requests information about the selected match from the server. The server retrieves information about match A from the database and sends it to the user's terminal.

[1683] 2. When a user points their camera at "Player X" while watching a match, the captured video is sent to the server. The server analyzes the video and identifies Player X. Player X's profile, statistics, etc., are retrieved from the database and sent back to the user's terminal.

[1684] 3. When a user asks a question via the chat function, such as "Please tell me the statistics for player X," the question is sent to the server. The generative AI analyzes the question and generates a relevant answer (for example, "Player X's batting average this season is .300"), which is then sent back to the user's terminal.

[1685] 4. Simultaneously, the emotion engine recognizes the user's emotions from their facial expressions and voice, and displays detailed performance information if the user is excited, or related episodes if they are relaxed.

[1686] In this way, the system meets users' real-time information needs and further enriches the viewing experience by providing customized information based on emotions.

[1687] The following describes the processing flow.

[1688] Step 1:

[1689] The user launches the app on their mobile device. The app's home screen is displayed, showing a list of currently ongoing matches.

[1690] Step 2:

[1691] The user selects the match they want to watch. A request containing information about the selected match is sent to the server.

[1692] Step 3:

[1693] The server receives the request and retrieves the relevant match information from the database. The retrieved match information is then sent back from the server to the user's terminal.

[1694] Step 4:

[1695] The user's terminal displays match information received on the screen. The displayed information includes the player list, match score, and opponent.

[1696] Step 5:

[1697] The user activates the camera function and points their mobile device at the field or players. The video captured by the camera function is analyzed on the device.

[1698] Step 6:

[1699] The device identifies players and areas from camera footage and sends the identification results to the server. The identification results include the player's location information and identification ID.

[1700] Step 7:

[1701] The server receives the identification result and retrieves the corresponding player and area information from the database. The retrieved information is then sent back from the server to the user's terminal.

[1702] Step 8:

[1703] The system overlays information received by the user's device onto the camera feed. This allows users to view detailed information about players and areas in real time.

[1704] Step 9:

[1705] Users enter questions using the chat function. For example, a question like "What is player X's batting average this season?" might be entered in chat format.

[1706] Step 10:

[1707] The terminal sends the entered question to the server. Upon receiving the question, the server uses a generative AI to generate an answer.

[1708] Step 11:

[1709] The server sends the generated response to the user's terminal. The generated response contains the necessary information, including details such as, for example, "Player X's batting average this season is .300."

[1710] Step 12:

[1711] The system displays the answers received by the user's device on the chat screen. This allows the user to see the answers to their questions in real time.

[1712] Step 13:

[1713] The emotion engine recognizes emotions from the user's facial expressions and voice. For example, it can detect emotions such as excitement, joy, and relaxation.

[1714] Step 14:

[1715] The emotion engine sends the recognized emotion information to the server. The server receives the emotion information.

[1716] Step 15:

[1717] The server customizes the information it provides based on the user's emotional state. For example, if the user is excited, it provides detailed player statistics; if they are relaxed, it provides anecdotes about the players and the game.

[1718] Step 16:

[1719] The server sends customized information to the user's terminal. The user's terminal displays this information, allowing the user to see customized information tailored to their emotions.

[1720] This series of processes allows users to receive personalized, emotion-based information in real time, enabling them to enjoy a richer viewing experience.

[1721] (Example 2)

[1722] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1723] Traditional sports viewing systems lack the means for users to obtain real-time information on matches and players. Furthermore, they fail to provide customized information based on user sentiment, making it difficult to offer an optimized viewing experience for individual users. Additionally, they lack the means to generate quick and accurate answers when responding to user questions via chat functions. To address these issues, a more advanced and personalized information delivery system is needed.

[1724] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1725] In this invention, the server includes means for a mobile terminal to acquire player and field information in real time, means for the server to acquire player and match information from a database and send it back to the mobile terminal, means for the user terminal to capture images of players and the field using its camera function and send the identification results to the server, means for the server to generate answers to questions using a generated AI model and send them back to the mobile terminal, means for the user terminal to analyze the user's facial expressions and voice to acquire emotional information and send the emotional information to the server, and means for the server to generate customized information based on the emotional data and send it back to the mobile terminal. As a result, users can acquire match and player information in real time, and personalized information tailored to the user's emotions can be provided. In addition, the viewing experience is improved because information is provided quickly and accurately through the chat function.

[1726] A "user terminal" is a portable information terminal operated by a user, which acquires and displays information using a specific application.

[1727] A "mobile device" refers to a portable information and communication device such as a smartphone or tablet, which is used to provide users with real-time information on matches and players.

[1728] A "server" is a computer system that communicates with user terminals via a network to process data and provide information.

[1729] A "database" is an information management system that stores match information and player information, and allows a server to retrieve this information as needed.

[1730] The "camera function" is a feature that allows the user's device to take pictures of players and the field, capture video footage, and send that video data to a server for analysis.

[1731] A "generative AI model" is an algorithm or program that uses artificial intelligence to analyze a user's question and generate an appropriate answer.

[1732] An "emotion engine" is a combination of software or hardware that analyzes a user's emotions from their facial expressions and voice and acquires that data.

[1733] "Identification results" refer to information about specific players or parts of the field obtained by analyzing video captured by the camera function.

[1734] "Overlay display" is a technology that displays additional information by superimposing it on the video or image currently displayed on the user's device's screen.

[1735] A "chat-style question" is an inquiry that a user enters in text format and sends to the server.

[1736] "Customized information" refers to information optimized for each individual user's situation, based on their emotional data.

[1737] This invention relates to a system that provides real-time information for sports viewing using a user terminal, server, database, and emotion engine. The system aims to acquire match information and player information in real time using the user's mobile terminal, and further to provide customized information based on the user's emotions.

[1738] System Configuration

[1739] User terminal hardware and software

[1740] The user terminal will be a mobile device such as a smartphone or tablet. This terminal will have a dedicated application installed, which will include features such as match selection, camera function, chat function, and emotion recognition function.

[1741] Server hardware and software

[1742] The server uses a high-performance network server to manage large amounts of data. It interacts with databases that store player and match information and responds to user requests. The server has a generative AI model and an emotion engine implemented in it.

[1743] database

[1744] The database is responsible for storing and managing match and player information. This includes player profiles, statistics, and match progress.

[1745] Emotional Engine

[1746] An emotion engine is software or hardware that analyzes a user's facial expressions and voice to recognize their emotional state in real time. Its role is to acquire the user's emotional data and transmit it to a server.

[1747] Program processing

[1748] Match selection function

[1749] The user launches the application and selects a match they want to watch from the match list. The selected match information is requested from the server. The server retrieves the corresponding match information from the database and sends it back to the user's terminal.

[1750] Camera function

[1751] The user activates the camera function and points their mobile device at the field or players to capture video. The captured video is sent to a server, where it is analyzed. Based on the analysis results, the server retrieves information on identified players and areas from its database and sends it back to the user's device.

[1752] Information display function

[1753] Player and match information sent from the server is overlaid on the user's terminal screen in real time. This allows users to instantly check detailed information about players and matches while watching.

[1754] Chat function

[1755] Users enter questions in text format using the in-app chat function. The entered questions are sent to the server and analyzed by a generative AI model. The generative AI model generates appropriate answers to the questions, retrieves supplementary information from the database, and sends it back to the user's device.

[1756] Emotion recognition function

[1757] The emotion engine recognizes the user's emotions in real time from their facial expressions and voice. The recognized emotion information is sent to a server, which generates customized information based on that information and sends it back to the user's terminal. For example, if the user is excited, detailed player performance information is provided; if they are relaxed, relevant anecdotal information is provided.

[1758] Examples of specific actions

[1759] For example, when a user launches the smartphone app and selects "Match A" from the "Match List" screen, the following sequence of events occurs:

[1760] 1. The user requests information about "Match A" from the server. The server retrieves the information about Match A from the database and sends it to the user's terminal.

[1761] 2. When a user points their camera at "Player X" while watching a match, the captured video is sent to the server. The server analyzes the video and identifies Player X. Player X's profile, statistics, etc., are retrieved from the database and sent back to the user's terminal.

[1762] 3. When a user asks a question using the chat function, such as "Please tell me the statistics for player X," the question is sent to the server. The generating AI model analyzes the question and generates a relevant answer (for example, "Player X's batting average this season is .300"), which is then sent back to the user's terminal.

[1763] 4. Simultaneously, the emotion engine recognizes the user's emotions from their facial expressions and voice, and displays detailed performance information if the user is excited, or related episodes if they are relaxed.

[1764] Example of a prompt

[1765] Examples of prompt statements to input into a generative AI model include the following:

[1766] 1. Example of a prompt when selecting a match

[1767] The user selected "Match A" from the match list. Retrieve the information for Match A and send it back to the user.

[1768] 2. Example prompts during camera image analysis

[1769] A user has taken a picture of player X with their camera. Please analyze the video footage and provide player X's profile and latest statistics.

[1770] 3. Examples of prompts when asking questions in a chat.

[1771] A user asked, "What are player Y's statistics for this season?" Please generate an answer based on player Y's latest statistics.

[1772] 4. Examples of prompts during emotion recognition

[1773] The user is excited. In this case, prioritize providing detailed performance information. If the user is relaxed, provide relevant episode information.

[1774] In this way, this system can further enhance the individual viewing experience by meeting users' real-time information needs and providing emotion-based, customized information.

[1775] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1776] Step 1:

[1777] Match Selection

[1778] Input: The user selects the match they want to watch from the match list on the application's home screen.

[1779] Data processing: Generate user selection information (match ID).

[1780] Output: The match ID is requested from the server.

[1781] Specific action: The user taps "Match A". The selected match ID is sent to the server.

[1782] Step 2:

[1783] Server-based acquisition of match information

[1784] Input: Match ID

[1785] Data processing: Search and retrieve relevant match information from the database.

[1786] Output: Match information is sent to the user's terminal.

[1787] Specific operation: The server accesses the database, retrieves information about match A (date, time, location, participating teams, etc.), and sends it back to the user's terminal.

[1788] Step 3:

[1789] Camera video capture and transmission

[1790] Input: Camera footage

[1791] Data processing: Capture video data and send it to the server.

[1792] Output: Video data is sent to the server.

[1793] Specific operation: The user takes a picture of player X using a mobile device, and the video is sent to the server.

[1794] Step 4:

[1795] Video analysis and information return by the server

[1796] Input: Video data

[1797] Data processing: Analyze the video to identify player X and retrieve related information from the database.

[1798] Output: Player X's information is sent back to the user's terminal.

[1799] Specific operation: The server analyzes the video data and identifies player X. It retrieves player X's profile and latest statistics from the database and sends them back to the user's terminal.

[1800] Step 5:

[1801] Information overlay display

[1802] Input: Player information, match information

[1803] Data processing: The received information is displayed as an overlay on the screen.

[1804] Output: Displayed in real time on the user's terminal.

[1805] Specific operation: When the user is filming player X, player X's profile and performance information will be overlaid on the video.

[1806] Step 6:

[1807] Send a chat question

[1808] Input: Chat text

[1809] Data processing: Send chat input text to the server.

[1810] Output: The question content is sent to the server.

[1811] Specific action: The user types "Please tell me the stats of player X" in the chat and sends it to the server.

[1812] Step 7:

[1813] Answer generation using a generative AI model

[1814] Input: Question content

[1815] Data processing: Analyze the question, retrieve the necessary information from the database, and generate the answer.

[1816] Output: The generated response is sent to the user's terminal.

[1817] Specific operation: The generative AI model analyzes the question and generates an answer, such as "Player X's batting average this season is .300," which is then sent to the user's terminal.

[1818] Step 8:

[1819] Sending emotion recognition data

[1820] Input: User facial expression data, voice data

[1821] Data processing: Analyze facial expressions and voice to generate emotion data.

[1822] Output: Emotional data is sent to the server.

[1823] Specific operation: The system analyzes the user's face and voice, and the emotion engine generates emotional data such as excitement or relaxation, which is then sent to the server.

[1824] Step 9:

[1825] Server-based customization of information

[1826] Input: Sentiment data

[1827] Data processing: Customize the information provided based on emotional data.

[1828] Output: Customization information is sent to the user's terminal.

[1829] Specific operation: The server selects detailed performance and episode information based on the user's emotional information and sends it to the user's terminal.

[1830] By having each processing step work in conjunction in this way, users can obtain match and player information in real time and enjoy customized information tailored to their individual emotions.

[1831] (Application Example 2)

[1832] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1833] Modern users demand more sophisticated information in experiences such as shopping and watching sports. Furthermore, providing timely and relevant information based on user emotions can dramatically improve the user experience. However, existing systems struggle to adapt to users' real-time emotional states and the information retrieval process is often cumbersome.

[1834] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1835] In this invention, the server includes means for acquiring information on objects and events from a database and sending it back to a mobile terminal; means for acquiring information on identified objects and areas and sending it back to the mobile terminal; means for generating answers to questions using a generative AI model and sending them back to the mobile terminal; and means for acquiring emotional information and providing the user terminal with customized information based on the user's emotions. As a result, the user can easily acquire detailed information on products and objects on the spot, and furthermore, receive optimal information in real time based on the user's emotions.

[1836] A "user terminal" is a device equipped with communication functions that a user uses to acquire and manipulate information.

[1837] A "mobile device" is a device that a user can carry with them, equipped with a camera and communication functions, and capable of displaying and transmitting information.

[1838] A "server" is a computer system that manages and processes information in conjunction with a database and provides that information in response to requests from user terminals.

[1839] A "database" is a system for organizing, storing, and searching information, and is a storage medium where match information, player information, product information, and so on are stored.

[1840] A "subject" is an item or person of interest to the user, which is captured and identified in the camera footage.

[1841] A "domain" refers to a specific place or area, specifically the field or area where an object exists.

[1842] A "generative AI model" is an artificial intelligence model that generates appropriate answers or information in response to given input.

[1843] An "emotion engine" is an algorithm that analyzes a user's facial expressions and voice data to recognize and determine their emotional state.

[1844] "Identification results" refer to information about objects and areas obtained when analyzing camera footage.

[1845] "Customized information" refers to information that is optimized and provided according to the user's emotions and circumstances.

[1846] This invention is a system that allows users in physical stores to obtain product and promotional information in real time using smartphones or smart glasses, and further provides a customized shopping experience based on the user's emotions.

[1847] System Overview

[1848] This system includes user terminals (smartphones and smart glasses), a server, a database, an emotion engine, and a generative AI model. The user terminal provides functions such as retrieving product information, answering questions, and emotion recognition through applications. The server works in conjunction with the database to manage vast amounts of product and promotional information, providing it in response to requests from user terminals.

[1849] User terminal operation

[1850] The user terminal is a smartphone or smart glasses operated by the user, and various functions are performed by launching an application. The application includes the following functions:

[1851] 1. Product Selection Function

[1852] The user launches the app and selects the product they want to find from the product list or category within the store. The selected product information is requested from the server.

[1853] 2. Camera function

[1854] Users obtain product information by scanning items using their smartphones or smart glasses. The camera footage is analyzed, and the information of the identified products is sent to a server.

[1855] 3. Information display function

[1856] Product information and promotional information sent from the server are overlaid in real time on the terminal's screen or visual display.

[1857] 4. Chat function

[1858] Users enter questions within the app, and an AI model generates appropriate answers. For example, the AI ​​generates an answer to a question such as, "Are there any similar products to this one?"

[1859] 5. Emotion recognition function

[1860] The emotion engine recognizes the user's emotions in real time from their facial expressions and voice. The recognized emotion information is sent to the server, and the information provided to the user is customized based on that information.

[1861] Server operation

[1862] The server interacts with the database and is responsible for managing and providing information about objects and events. Its specific operations are as follows:

[1863] 1. Providing product information

[1864] When the server receives a product selection request from the user's terminal, it retrieves the relevant product information from the database and sends it back to the user's terminal.

[1865] 2. Analysis of camera footage

[1866] The system analyzes camera footage transmitted from the user's terminal to identify products. Based on the identification results, it retrieves relevant product and promotional information from a database and sends it back to the terminal.

[1867] 3. Responses by generative AI models

[1868] The system analyzes questions submitted by users via the chat function using an AI model and generates answers. The generated answers are then supplemented with necessary information retrieved from a database and sent back to the user's device.

[1869] 4. Processing emotional information

[1870] The system acquires user emotion information recognized by the emotion engine and customizes the information provided to the user's device based on that information. For example, if the user appears happy, it provides information about special promotions; if the user appears surprised, it provides detailed product information and reviews.

[1871] Specific operation examples

[1872] For example, if a user uses smart glasses to find a specific product, the following sequence of events will occur:

[1873] 1. When a user scans a product with the smart glasses' camera, the captured image is sent to the server. The server analyzes the image and identifies the product. Detailed information and promotional information for the identified product are retrieved from the database and sent back to the user's device.

[1874] 2. When a user asks "Are there any similar products to this one?" using the chat function, the question is sent to the server. The generation AI model analyzes the question, generates a relevant answer (for example, "There is a similar product called XX"), and sends it back to the user's device.

[1875] 3. Simultaneously, the emotion engine recognizes the user's emotions from their facial expressions and voice. If the user appears happy, it displays additional promotional information; if they are interested, it displays detailed product information and reviews from other users.

[1876] In this way, the system meets users' real-time information needs and further enriches the shopping experience by providing customized information based on their emotions.

[1877] Prompt example

[1878] "Please add a feature to this shopping assistant app that provides information based on real-time emotions. When a user views a particular product, it should display customized information based on the user's emotions and past purchase history."

[1879] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1880] Step 1:

[1881] Users scan products using their smartphones or smart glasses.

[1882] Input: The user takes a picture of the product with their camera.

[1883] Output: Captured product video is obtained.

[1884] Specific operation: The user points the camera at the product and presses the capture button. The smart device's camera captures the image and saves the image data locally.

[1885] Step 2:

[1886] The device sends the captured video to the server.

[1887] Input: Data from captured product video.

[1888] Output: Product video data is sent to the server.

[1889] Specific operation: The application retrieves the captured video data and sends it to the server via the network.

[1890] Step 3:

[1891] The server analyzes the received video data to identify the product.

[1892] Input: Product video data sent to the server.

[1893] Output: Identified product ID and related information.

[1894] Specific operation: The server applies a video analysis algorithm to identify the product. As a result of the identification, it extracts a product ID and retrieves product information from the database based on that ID.

[1895] Step 4:

[1896] The server retrieves product information from the database and sends it back to the user's terminal.

[1897] Input: Identified product ID.

[1898] Output: Corresponding product information data.

[1899] Specific operation: The server executes a database query based on the product ID and retrieves the corresponding product information (price, stock, description, etc.). The retrieved data is then sent back to the user's terminal.

[1900] Step 5:

[1901] The device displays the product information it has received to the user.

[1902] Input: Received product information data.

[1903] Output: Product information is displayed on the user's terminal screen.

[1904] Specific operation: The application analyzes the received product information data and displays detailed information, including product name, price, and description, as an overlay on the screen.

[1905] Step 6:

[1906] Users enter their questions using the in-app chat function.

[1907] Input: User's question text.

[1908] Output: The question text is sent to the server.

[1909] Specific operation: The user types a question in the chat window and presses the send button. The application sends the question text to the server.

[1910] Step 7:

[1911] The server uses a generated AI model to produce an answer to the question and sends it back to the user's terminal.

[1912] Input: User's question text.

[1913] Output: Generated answer text.

[1914] Specific operation: The server inputs the question text into an AI model, which generates an appropriate answer. The generated answer is then supplemented with information retrieved from a database and sent back to the user's terminal.

[1915] Step 8:

[1916] The user's terminal displays the generated response in the chat window.

[1917] Input: Generated response text.

[1918] Output: Chat response displayed on the user's device.

[1919] Specific operation: The application displays the received response text in the chat window for the user to review.

[1920] Step 9:

[1921] The device captures the user's facial expressions and voice, analyzes them using an emotion engine, and sends the emotional information to the server.

[1922] Input: User's facial expressions and voice data.

[1923] Output: Analyzed emotion information.

[1924] Specific operation: The smart device's camera and microphone are used to capture the user's facial expressions and voice, and the emotion engine analyzes this data. The resulting emotion information is then sent to the server.

[1925] Step 10:

[1926] The server generates customized information based on emotional data and sends it back to the user's terminal.

[1927] Input: Analyzed emotion information.

[1928] Output: Customized informational data.

[1929] Specific operation: The server analyzes the received emotional information and generates optimal product and promotional information based on the user's emotional state. The generated information is then sent back to the user's terminal.

[1930] Step 11:

[1931] The device provides users with customized information.

[1932] Input: Customized information data.

[1933] Output: Customization information displayed to the user.

[1934] Specific operation: The application displays the customization information it has received and provides the user with appropriate product suggestions and promotions.

[1935] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1936] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1937] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1938] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1939] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. In the upper and lower directions of the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. Also, the upper side of the concentric circles is where "pleasant" emotions are located, and the lower side is where "unpleasant" emotions are located. In this way, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1940] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1941] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1942] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1943] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1944] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1945] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1946] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1947] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1948] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1949] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1950] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1951] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1952] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1953] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1954] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1955] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[1956] The following is further disclosed regarding the embodiments described above.

[1957] (Claim 1)

[1958] In a system that provides information via a user terminal,

[1959] A means for mobile devices to obtain information about players and the pitch in real time,

[1960] A method by which a server retrieves player and match information from a database and sends it back to a mobile device,

[1961] A means by which a user terminal captures images of players and the pitch using its camera function and sends the identification results to a server,

[1962] A means for the server to acquire information on identified players and areas and send it back to the mobile device,

[1963] A means by which a user terminal receives questions in a chat format and sends those questions to a server,

[1964] A means by which a server generates an answer to a question using a generative AI and sends it back to the mobile device,

[1965] A system that includes this.

[1966] (Claim 2)

[1967] The system according to claim 1, comprising a means for a server to acquire live status data in real time and send it back to the mobile terminal of a television viewer.

[1968] (Claim 3)

[1969] The system according to claim 1, wherein the mobile terminal has means for overlaying player information onto the camera image.

[1970] "Example 1"

[1971] (Claim 1)

[1972] In a system that provides information via a user terminal,

[1973] A means for mobile devices to obtain information about matches and players in real time,

[1974] A method by which the server retrieves match and player information from a database and sends it back to the mobile device,

[1975] A means by which a user terminal captures video of the match or players using its camera function and sends the identification results to a server,

[1976] A means for the server to acquire information on identified players and areas and send it back to the mobile device,

[1977] A means by which a user terminal receives questions in a chat format and sends those questions to a server,

[1978] A means by which a server generates an answer to a question using a generative AI model and sends it back to the mobile device,

[1979] A means for generating prompt sentences to be input to a generative AI model,

[1980] A system that includes this.

[1981] (Claim 2)

[1982] The system according to claim 1, comprising a means for a server to acquire live data in real time and send it back to the viewer's mobile device.

[1983] (Claim 3)

[1984] The system according to claim 1, comprising a means for a mobile terminal to overlay identified player information onto camera footage.

[1985] "Application Example 1"

[1986] (Claim 1)

[1987] In a system that provides information via a user terminal,

[1988] A means for mobile devices to acquire information about players and the field in real time,

[1989] A method by which a server retrieves player and match information from a database and sends it back to a mobile device,

[1990] A means by which a user terminal uses its camera function to capture video of players and the field, and sends the identification results to a server,

[1991] A means for the server to acquire information on identified players and areas and send it back to the mobile device,

[1992] A means by which a user terminal receives questions in a chat format and sends those questions to a server,

[1993] A means by which a server generates an answer to a question using a ...

Claims

1. In a system that provides information via a user terminal, A means for mobile devices to obtain information about players and the pitch in real time, A method by which a server retrieves player and match information from a database and sends it back to a mobile device, A means by which a user terminal uses its camera function to capture images of players and the pitch, and sends the identification results to a server, A means for the server to acquire information on identified players and areas and send it back to the mobile device, A means by which a user terminal receives questions in a chat format and sends those questions to a server, A means by which a server generates an answer to a question using a generative AI and sends it back to the mobile device, A system that includes this.

2. The system according to claim 1, further comprising means for a server to acquire live status data in real time and send it back to the mobile terminal of a television viewer.

3. The system according to claim 1, wherein the mobile terminal has means for overlaying player information onto the camera image.

Images