system

Abstract

We provide the system. [Solution] A means for receiving event information or shopping information entered from a spectator's or customer's terminal, A means for analyzing data to generate explanatory information or product information to be provided to spectators or customers based on the aforementioned information, Means for transmitting the aforementioned explanatory information or product information to the terminal of a spectator or customer, and presenting it by display or sound, A means of predicting the status of events at facilities and the congestion levels of services, A means of proposing the optimal timing for using the service or purchasing the service to the spectator or customer based on the aforementioned prediction results, A means of analyzing traffic conditions or travel conditions and providing the optimal return route or travel path for spectators or customers, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In sports watching, spectators with little knowledge of the sport find it difficult to understand the on-site play and rules, and as a result, cannot fully enjoy the pleasure of watching. Also, in the use of events and food and beverage services in stadiums and arenas, it is difficult to determine the optimal timing, and congestion is often unavoidable. Furthermore, when returning home, the traffic situation may not be properly grasped, and one may be caught in the congestion on the way back. Thus, there are problems in improving the overall sports watching experience.

Means for Solving the Problems

[0005] This invention proposes a system that analyzes gameplay in real time based on sports event information received from spectators' devices, and generates and provides easy-to-understand commentary information to spectators. Furthermore, it incorporates a function that uses AI technology to predict event schedules and the congestion levels of food and beverage facilities, suggesting the optimal timing for spectators to use these facilities. Additionally, it analyzes traffic conditions upon departure and suggests the most suitable route home for spectators. This comprehensively supports the sports viewing experience, enabling spectators to enjoy a more comfortable and fulfilling time.

[0006] A "spectator" refers to an individual who directly watches a sporting event at a stadium, arena, or other venue.

[0007] "Devices" refers to portable electronic devices such as smartphones and tablets used by spectators.

[0008] "Sports event information" refers to the type of sport and detailed information that spectators submit in connection with their viewing experience.

[0009] "Explanatory information" refers to information that includes explanations about the progress and rules of the sporting event being watched.

[0010] "Match data" refers to data collected during a sporting event regarding the movements of players and the progress of the match.

[0011] "Event status" refers to information about events and activities other than matches that take place inside stadiums and arenas.

[0012] "Food and beverage establishment congestion status" refers to information regarding the level of congestion at food and beverage outlets within stadiums and arenas.

[0013] "Optimal service usage timing" refers to the appropriate time for spectators to comfortably use the event and food and beverage facilities.

[0014] "Traffic conditions" refers to information regarding the degree of congestion on roads and transportation facilities when returning home.

[0015] "Return route" refers to the movement route selected by spectators when returning home after watching a game.

Brief Explanation of Drawings

[0016] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of a data processing system in Application Example 2 when a sentiment engine is combined.

Mode for Carrying Out the Invention

[0017] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0020] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0021] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0022] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0024] [First Embodiment]

[0025] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0026] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0029] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0032] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0036] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0037] This invention is a system designed to enhance the sports viewing experience for spectators, and is realized through the use of spectator terminals, servers, and AI technology. In this system, spectators can receive necessary commentary and facility usage information in real time by sending information about sports events from their terminals to the server.

[0038] First, the user enters information about the sporting event using their device. This includes the type of sport they will watch and details about the match. The user's device then sends this input data to the server. The server uses a dedicated application on a smartphone or tablet to provide the information through an intuitive user interface.

[0039] The server analyzes video feeds and related data of the match based on the received sports event information. Artificial intelligence then generates commentary information in real time based on this data and provides it to the user. For example, the server can identify important plays during a match and provide detailed explanations of their tactical significance and the rules governing them.

[0040] Furthermore, the server collects and analyzes information on event status within stadiums and arenas, as well as congestion at food and beverage facilities. This allows users to receive suggestions for the optimal timing to use the services. Specifically, the server uses AI to predict congestion levels, identify the best time to use the services, and notifies the user.

[0041] After the event, the server analyzes traffic conditions to match the user's expected return time. This allows the optimal return route to be sent to the user's device, enabling them to avoid congestion and return home smoothly. In this way, the system comprehensively supports the user's viewing experience and provides new value to sports viewing.

[0042] The following describes the processing flow.

[0043] Step 1:

[0044] The user uses a device to enter information about the sporting event they wish to attend. The device accepts data, including details such as the type of match and the date of the event, through a dedicated application.

[0045] Step 2:

[0046] The device sends the entered sports event information to the server. The device uses an encrypted network connection to securely transmit data.

[0047] Step 3:

[0048] The server analyzes the received sports event information and collects relevant match video feeds and historical data. The server uses AI algorithms to generate explanatory information on important plays and rules of the match.

[0049] Step 4:

[0050] The server generates commentary information and provides it to users watching the game in a timely manner. The commentary information is sent to the device and displayed as audio or text.

[0051] Step 5:

[0052] The server collects information on event status and congestion levels at food and beverage facilities within the stadium, and analyzes the information in real time using a predictive model. Based on the results, the server notifies spectators of the optimal time to use the services.

[0053] Step 6:

[0054] When a user begins preparing to go home after the game ends, the server investigates traffic conditions and analyzes the optimal route home. The server sends the analysis results to the user's device to help them get home efficiently.

[0055] (Example 1)

[0056] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0057] For spectators to enjoy sporting events more comfortably and meaningfully, real-time supplementary information about the matches and information about crowd conditions at the venue are necessary. However, current technology does not adequately provide this information in real time, which degrades the quality of the viewing experience. Furthermore, existing systems have limitations in providing customized information tailored to the interests and understanding of individual spectators. This invention aims to solve these problems.

[0058] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0059] In this invention, the server includes means for receiving sports event information input from a spectator's information processing device, means for analyzing competition data, and means for providing real-time commentary on important events during the match using a generative AI model. This allows spectators to gain a deep understanding of the current match situation and receive real-time information on crowd conditions and traffic at the venue.

[0060] A "spectator information processing device" refers to an electronic device used by users to input sports event information and transmit it to a server. Typically, this includes smartphones and tablets.

[0061] "Sports event information" refers to data that includes information such as the type of sport a user will be watching, details of the match, location, and date and time.

[0062] "Competition data" refers to information that records the progress and content of a match in digital format, such as video feeds related to the match, player statistics, and statistical information.

[0063] A "generative AI model" refers to artificial intelligence technology that analyzes data during a match and generates value-added information, such as commentary, in real time.

[0064] "Explanatory information" refers to detailed information provided to users about the rules of the match, the tactical significance of plays, and the backgrounds of the players.

[0065] "Crowding status" refers to information about the level of crowding at event venues, restaurants, and other facilities, and is obtained using sensors and other data collection methods.

[0066] "Traffic conditions" refers to information regarding congestion, accidents, and other issues on roads and public transportation used by spectators on their way home.

[0067] This invention is a system for improving the sports viewing experience for spectators, and is realized by using an information processing device, a central scoring device, and a generative AI model.

[0068] First, users input detailed information about the sporting event they wish to attend using an information processing device such as a smartphone or tablet. The information processing device then securely transmits this input data to a central tallying device using the HTTPS protocol. This allows users to provide information with intuitive operation.

[0069] The central data aggregation system analyzes competition data in real time based on received sports event information. Using a generative AI model, it detects important events from the in-game data and generates commentary information based on these events. In particular, the generative AI model generates and notifies users of structured information in real time, providing commentary on important plays and their strategic significance.

[0070] Furthermore, the central scoring system analyzes the congestion situation within the venue using sensors and other information gathering methods, and notifies users of the optimal timing for using the service. In addition, after the match ends, it analyzes traffic conditions in real time and optimizes and sends guidance to users on their return journey to their terminals.

[0071] A concrete example would be a user entering information such as "Soccer, FC Tokyo vs. Kawasaki Frontale, April 10th, Ajinomoto Stadium." The central scoring system would then analyze any unusual plays during the match based on this information, explain the details to the user, and inform them of the best time to avoid congestion.

[0072] An example of a prompt message is: "Generate explanatory information for the sports viewing system. The type of match is soccer, and the match is between FC Tokyo and Kawasaki Frontale. What information will you provide to the customer?"

[0073] This invention utilizes these technologies to provide spectators with real-time and personalized information about sporting events, thereby improving the viewing experience.

[0074] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0075] Step 1:

[0076] Users enter detailed information about the sporting event they are watching using their smartphones or tablets. This includes the type of match, date and time, location, and participating teams. The entered information is transmitted by the information processing unit to the central tallying unit. This transmission is secure, using the HTTPS protocol. The output is the event information received by the central tallying unit.

[0077] Step 2:

[0078] The server matches and collects relevant competition data based on sports event information received from users. During this process, video feeds and statistical information for the relevant matches are retrieved from the database. Inputs are user information and a dynamic database, while output is a competition dataset that serves as the basis for analysis.

[0079] Step 3:

[0080] The server inputs the collected competition dataset into a generating AI model, which analyzes important match events in real time. During this process, the AI ​​model analyzes the data and detects tactically important plays and match highlights. Based on this, detailed commentary information is generated. The output is commentary information for the user.

[0081] Step 4:

[0082] The server sends the generated explanatory information to the user's information processing device. This information is presented to the user visually or audibly. Specifically, the explanatory information is provided immediately via push notifications through the notification API. The output is the explanatory information displayed on the device.

[0083] Step 5:

[0084] The server collects and analyzes real-time congestion data from sensors and other sources within the match venue. This data is used with machine learning algorithms to predict congestion and analyze the current situation. The output is congestion prediction information, which is used to calculate the optimal time to use the service.

[0085] Step 6:

[0086] The server uses congestion forecast information to notify users of the optimal time to use the service. This includes the best times to avoid congestion at food and beverage facilities and restrooms within the stadium. Specifically, an alert is sent to the user's terminal. The output is the timing suggestion information presented to the terminal.

[0087] Step 7:

[0088] After the match ends, the server collects traffic information and optimizes the return routes for spectators. It analyzes real-time traffic data to generate routes that avoid congestion. Based on this, return route guidance is sent to the user's device. The output is optimized return route information.

[0089] (Application Example 1)

[0090] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0091] In recent years, providing individually optimized information and less crowded experiences to spectators and shoppers has become a crucial challenge. In particular, providing real-time, customized information and efficient route suggestions during sports events and in-store purchases is difficult and a major source of stress. This invention aims to solve these problems and provide spectators and customers with a more fulfilling experience.

[0092] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0093] In this invention, the server includes means for receiving event information or shopping information input from a spectator's or customer's terminal, means for analyzing data to generate explanatory information or product information based on the information, and means for transmitting and presenting the explanatory information or product information to the spectator's or customer's terminal. This enables the provision of individually optimized information and efficient service use for spectators and customers.

[0094] A "spectator" is someone who watches a sporting event or other form of entertainment, and is also the recipient of information about it.

[0095] A "terminal" is a device used for inputting or displaying information, and includes smartphones, tablets, and other similar devices.

[0096] "Event information" refers to information related to a specific sports event or activity, including the date, time, location, and details of the competition.

[0097] "Explanatory information" refers to information that provides a detailed explanation of the content of a sporting event, including the situation of the match and its tactical implications.

[0098] "Product information" refers to information about the location and promotion of products in physical stores, and is intended to guide customers when they make a purchase.

[0099] "Data" refers to a collection of information obtained from spectators or customers, as well as various other pieces of information related to the event.

[0100] "Analyzing" is the process of evaluating the information obtained and understanding trends and characteristics.

[0101] "To send" refers to the act of delivering information to another device or server.

[0102] "To present" refers to the act of showing or explaining information to a user through a device.

[0103] "Congestion level" refers to the extent to which a particular area or service is being used.

[0104] "Predicting" is the act of estimating future conditions based on existing data.

[0105] "Service utilization timing" refers to the time when spectators or customers can most effectively utilize the service.

[0106] "Purchase timing" refers to the most opportune time for a customer to buy a product.

[0107] "Traffic conditions" refers to information that indicates the degree of congestion and flow of traffic on roads and public transportation.

[0108] "Movement conditions" refer to the environment and circumstances when spectators or customers move from one point to another.

[0109] A "generative AI model" is an artificial intelligence technology that extracts features from data and performs analysis and predictions.

[0110] A "prompt statement" is an instruction statement used to perform specific information processing on a generating AI model.

[0111] The system for implementing this invention is for receiving information transmitted from spectators' or customers' terminals and providing optimized information based on that information. First, the user inputs event information or purchase information using a terminal such as a smartphone or tablet. This information is transmitted to the server.

[0112] The server analyzes the received information using a generative AI model. This AI model is built using frameworks such as TENSORFLOW® and PyTorch, extracting features from the information and generating commentary for spectators and product information for customers. The information is analyzed in real time and transmitted to the terminal. This information is either displayed visually on a screen or output as audio.

[0113] Furthermore, the server predicts congestion levels within the facility and generates prompts to inform spectators and customers of the optimal time to use the service. This prediction utilizes machine learning algorithms and is updated in real time. For example, a user visiting a shopping mall on the weekend can use this app to be guided to less crowded routes or receive information on items on sale.

[0114] Furthermore, the server analyzes traffic conditions or movement patterns within the store to provide indicators for offering the optimal return route or travel path for spectators and customers. This information is sent to the user as a prompt.

[0115] Example of a prompt:

[0116] "Analyze the in-store camera feeds to predict congestion levels in front of product shelves."

[0117] "Please take into account the current customer traffic patterns and suggest the optimal checkout wait time."

[0118] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0119] Step 1:

[0120] The user uses a terminal to enter event or purchase information. This input data includes the type and date of the event, and a list of items the user wishes to purchase. The terminal then pre-processes this data and prepares it for transmission to the server.

[0121] Step 2:

[0122] The server retrieves user input data received from the terminal. At this time, it checks the validity of the data to ensure that there is no inaccurate information. The received data is then passed to the generation AI model, which analyzes the necessary information and prepares to generate explanatory and product information.

[0123] Step 3:

[0124] The generative AI model analyzes the received data. A pre-trained model is used for data analysis, generating optimal information based on the user's individual interests and behavior. The output includes explanatory and product information, which is managed on the server.

[0125] Step 4:

[0126] The server sends the generated explanatory or product information to the user's device. The user's device displays or outputs the received information as audio. This allows the user to receive the necessary information in real time.

[0127] Step 5:

[0128] The server uses data from sensors and cameras within the facility to predict congestion levels and traffic flow. Machine learning algorithms analyze this data in real time and prepare to generate prompt messages to avoid congestion.

[0129] Step 6:

[0130] Based on congestion levels and predicted information, the server generates prompt messages and suggests the optimal time for the user to use the service. The terminal receives these suggestions and notifies the user. The user can then use the service efficiently by following the suggestions.

[0131] Step 7:

[0132] The server analyzes traffic conditions or in-store movement data to determine the optimal return route or travel path. The generated results are sent to the user's terminal and presented as navigation.

[0133] Step 8:

[0134] Users follow the on-screen navigation to return home or travel safely and smoothly. This promotes efficient travel with minimal congestion and waiting times.

[0135] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0136] This invention is a system designed to enhance the experience of sports spectators, providing dynamic content based on the spectator's real-time emotions and interests. This system, which incorporates an emotion engine, is realized by integrating spectator terminals, servers, and emotion recognition technology.

[0137] First, the user selects a sporting event to watch via their device, and the device sends this information to the server. At this time, the device uses its camera and microphone to detect the user's emotions. The emotion engine analyzes the user's facial expressions and tone of voice to recognize emotions such as joy and surprise.

[0138] The server analyzes the information based on the sports event selected by the user and the detected sentiment data, and generates explanatory information according to the user's interest and level of understanding. For example, if the user expresses surprise, the server will provide a detailed explanation of the importance and background of that play.

[0139] Furthermore, the server predicts the status of events and the congestion levels of restaurants and bars, and the emotion engine considers the user's stress and level of concentration to suggest the optimal time to visit. For example, if a user is intently watching a game, it will suggest postponing their dining or drinking time to avoid congestion.

[0140] After the event, the server analyzes the user's emotional history through an emotion engine and provides the device with the optimal route home, taking traffic conditions into consideration. This feedback loop continuously improves the user's sports viewing experience.

[0141] In this way, this system changes and adjusts the service according to the emotions of the spectators, making it an enjoyable and easy-to-understand experience as part of the viewing experience.

[0142] The following describes the processing flow.

[0143] Step 1:

[0144] The user uses their device to input information about the sporting event they want to watch. When the device sends the entered event information to the server, it simultaneously collects the user's facial expressions and voice using its camera and microphone.

[0145] Step 2:

[0146] The device sends the user's emotional data collected to the server. The device's emotion engine performs initial analysis to recognize emotions from facial expressions and voice.

[0147] Step 3:

[0148] Based on the sports event information and sentiment data received by the server, AI is used to generate real-time commentary on the match. The server adjusts the commentary content, taking into account the user's level of interest and understanding.

[0149] Step 4:

[0150] The server generates explanatory information and sends it to the user's device, which then presents the information to the user via audio or text display. In particular, if the user's reaction is one of surprise, the server will provide a more detailed explanation of the background and specifics of the play.

[0151] Step 5:

[0152] The server predicts the status of events taking place in the stadium and the congestion levels of food and beverage facilities, and uses an emotion engine to assess the user's stress level. The server calculates the best time for the user to use the service comfortably and notifies the device.

[0153] Step 6:

[0154] After the viewing session ends, the server analyzes the user's emotional history and calculates the optimal route home. The terminal then provides this route information to the user, helping them to return home efficiently while avoiding congestion.

[0155] (Example 2)

[0156] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0157] Providing personalized commentary tailored to the emotions and interests of spectators in real time during sports events is difficult. Furthermore, optimal service utilization and return-home suggestions that take into account event crowds and traffic information are insufficient. This can lead to inconvenience for spectators and detract from the overall viewing experience.

[0158] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0159] In this invention, the server includes means for processing spectator behavior and audio signals and identifying emotions, means for predicting the status of the event and the congestion of food and beverage facilities, and means for analyzing traffic conditions and providing spectators with the optimal route home. This enables the provision of personalized commentary tailored to the emotions of spectators, as well as the suggestion of optimal services and routes during and after the event.

[0160] An "information processing device" is a device used for inputting, processing, and outputting data, and refers to terminal equipment used by spectators.

[0161] "Sports competition information" refers to a collection of data and information related to sports matches and competitions.

[0162] "Explanatory information" refers to the commentary and background information provided to spectators, including detailed information related to the competition.

[0163] "Competition information" refers to data and statistics related to a specific match or competition, including tactics, player data, and scores.

[0164] "Emotional analysis means" refers to techniques or methods for identifying emotional states from the facial expressions and voices of spectators.

[0165] "Presenting visually or aurally" refers to providing information through a device as a screen display or audio output.

[0166] "Event status" refers to information related to the ongoing status and schedule of an event or match.

[0167] "Crowding situation" refers to information about how crowded a place or facility is where spectators gather.

[0168] "Return route" refers to the paths or routes that spectators can choose to take when returning home after the event has ended.

[0169] A "generative AI model" refers to a model that uses artificial intelligence technology to generate useful information and predictions from large amounts of data.

[0170] This invention provides a system that enhances the viewing experience by offering content such as commentary that dynamically changes in response to the emotions of the spectators. The embodiments thereof are described in detail below.

[0171] Users select the sport they wish to watch using a mobile device or computer. This device receives input from the spectator and transmits the sport information to a server. The device also captures the spectator's facial expressions and voice through input devices such as cameras and microphones, and uses this information for emotion analysis. This analysis uses emotion analysis tools to identify the spectator's reactions (joy, surprise, etc.) in real time.

[0172] The server uses a generative AI model to generate the most relevant explanatory information for the spectator based on the received athletic competition information and spectator emotion data. This provides relevant background information and tactical commentary according to the emotions the spectator expresses in response to specific moments in the match. For example, if a spectator expresses surprise at a crucial scoring moment in the match, a detailed explanation of the background leading up to that goal and the strategic movements of the players will be notified to the terminal.

[0173] Furthermore, the server analyzes historical data and real-time information to predict the event's progress and the congestion of food and beverage facilities. This allows it to suggest optimal timings for eating and drinking, as well as event viewing plans, to spectators. After the event ends, it also considers current traffic conditions and provides spectators with ideal routes home. The terminal visually displays the route home, supporting spectators in moving smoothly.

[0174] An example of a prompt would be, "Explain how to provide a user who is surprised by a soccer goal with a detailed explanation of the player's background and similar past tactics." This prompt allows the generative AI model to provide viewers with appropriate information, resulting in a more enriching viewing experience.

[0175] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0176] Step 1:

[0177] The user selects the sports they wish to watch using their device. The user's selection information is retrieved as input, and the device sends this information to the server. This allows the server to prepare data on the sports that the spectator is interested in.

[0178] Step 2:

[0179] The device uses its built-in camera and microphone to collect the user's facial expressions and voice in real time. Video and audio data are acquired as input, and this data is used for analysis by emotion analysis tools. The emotion engine analyzes this data to identify the user's emotional state (joy, surprise, etc.) in real time. Identified emotion data is generated as output.

[0180] Step 3:

[0181] The server receives athletic competition information from the user and emotional data acquired from the device as input. Using a generative AI model, the server generates explanatory information tailored to the spectator based on the input information. This process dynamically generates commentary that responds to the match situation and the spectator's emotions. Customized explanatory information is generated as output.

[0182] Step 4:

[0183] The server sends the explanatory information generated above to the terminal. The terminal receives this and presents it to the user visually or audibly. The explanatory information, as input, is provided in a format that is easy for the user to understand and includes specific explanations to enhance the viewing experience. As output, the information is presented to the user in a format that is easy for them to understand.

[0184] Step 5:

[0185] The server analyzes the progress of events and the congestion levels of food and beverage facilities in real time. Using historical data and current schedule information as input, it predicts the optimal timing for service utilization. This process enables a function that suggests the best visit timing and break plans to the user. The output generates suggestions for the optimal service timing for the user.

[0186] Step 6:

[0187] After the event ends, the server collects and analyzes traffic information to calculate the optimal route home for the user. Traffic data is used as input, and the server performs calculations to determine the shortest or most efficient route for the user. As output, a specific route home is sent to the terminal, providing the user with visual instructions.

[0188] (Application Example 2)

[0189] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0190] Conventional sports viewing systems have limited personalized explanations and information based on the viewer's emotions and interests, making it difficult to extend and optimize the viewing experience within the home environment. Furthermore, there has been a lack of established means to control home automation devices in real time in response to changes in emotions during viewing, thereby providing a comfortable viewing environment. As a result, there has been a challenge in providing an environment where viewers can enjoy themselves to the fullest.

[0191] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0192] In this invention, the server includes means for receiving sports activity information input from a spectator's information terminal, means for analyzing the activity data and generating explanatory information to be provided to the spectator, means for transmitting the explanatory information to the spectator's information terminal and presenting it by display or audio, means for predicting the progress of the event and the congestion of food and beverage facilities and suggesting the optimal timing for using the services, means for analyzing traffic conditions and providing the optimal route home, and means for controlling a home automation device for adjusting the home environment based on the spectator's emotions. This makes it possible to optimize the home environment according to the spectator's emotions and provide a personalized viewing experience.

[0193] A "spectator information terminal" is a device equipped with the function of receiving sports activity information and displaying or audio-presenting the provided explanatory information.

[0194] "Sports activity information" refers to information about events that spectators are watching, including the status of the matches.

[0195] "Descriptive information" refers to data generated to provide spectators with details and related information about the match.

[0196] "Activity data" refers to various data related to sports activities, mainly including the progress of matches and statistical data.

[0197] "Event progress" refers to information showing the current status and timeline of ongoing sports activities.

[0198] "Food and beverage facility congestion status" refers to information regarding the usage status of food and beverage spaces at sports event venues.

[0199] "Optimal service usage timing" refers to the time when spectators can use events and facilities efficiently and comfortably.

[0200] "Traffic conditions" refers to information regarding traffic flow and congestion around the event venue and on the return route.

[0201] "Home automation devices" are devices used to control the environment within a home, and include lighting, air conditioning, robots, and other similar devices.

[0202] "Emotion-based adjustment" refers to analyzing the emotional state of spectators and automatically changing the lighting and temperature in the home based on that analysis.

[0203] This invention is a system that dynamically utilizes spectator emotional data to optimize the in-home sports viewing experience. The server receives sports activity information from the spectator's information terminal, analyzes the match activity data based on that information, and generates explanatory information to provide to the spectator. This explanatory information is transmitted to the spectator's information terminal and presented via display or audio.

[0204] Furthermore, the server predicts the progress of the event and the congestion levels of food and beverage facilities, and based on this data, suggests the optimal timing for spectators to use the services. It also analyzes traffic conditions and provides the best routes home for spectators, ensuring smooth travel.

[0205] In addition, by controlling home automation devices, the home environment can be dynamically adjusted based on the spectator's emotions. For example, if a spectator becomes excited, the brightness of smart lighting can be increased and the air conditioning adjusted to create a comfortable environment. Home robots can also provide beverages at the appropriate time. This creates an environment where spectators can enjoy the game with greater concentration.

[0206] In summary, this invention aims to provide a comfortable and efficient viewing environment through spectator emotional data, and its implementation utilizes open-source facial recognition libraries (e.g., OpenCV), home automation devices, information terminals, and other hardware and software.

[0207] By utilizing generative AI models, it is possible to improve the accuracy of analyzing and predicting emotional data. An example of a prompt would be: "Suggest ways to optimize the home environment based on the user's emotions while watching sports. For example, come up with ideas on how to adjust the lighting and air conditioning when the user is excited, and what a robot should provide."

[0208] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0209] Step 1:

[0210] The terminal sends sports activity information as input to the server. This allows information about matches that the user is interested in to be accumulated on the server, preparing the data that will serve as the basis for subsequent processing.

[0211] Step 2:

[0212] The server analyzes match activity data based on sports activity information received from terminals. This includes match progress and player statistics. The analyzed data is generated as explanatory information and customized to the user's interests.

[0213] Step 3:

[0214] The server sends the generated explanatory information to the user's device. The device receives this information and presents it on the screen or as audio. The user can view this information and deepen their understanding of the match.

[0215] Step 4:

[0216] The server uses machine learning algorithms to analyze collected data in order to predict the progress of events and the congestion levels of food and beverage facilities. The prediction results are used as a basis for deciding when to suggest the optimal time for users to use the services.

[0217] Step 5:

[0218] The server analyzes traffic conditions in real time and prepares data to guide users home via the most suitable route after watching a game. This information provides the optimal route, taking into account traffic flow and congestion.

[0219] Step 6:

[0220] The server analyzes user emotion data and generates instructions to control home automation devices. These instructions then adjust lighting, change air conditioning settings, and provide services via home robots, all in response to the user's emotions. The use of a generative AI model improves the accuracy of emotion analysis, further optimizing the user experience.

[0221] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0222] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0223] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0224] [Second Embodiment]

[0225] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0226] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0227] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0228] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0229] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0230] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0231] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0232] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0233] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0234] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0235] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0236] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0237] This invention is a system designed to enhance the sports viewing experience for spectators, and is realized through the use of spectator terminals, servers, and AI technology. In this system, spectators can receive necessary commentary and facility usage information in real time by sending information about sports events from their terminals to the server.

[0238] First, the user enters information about the sporting event using their device. This includes the type of sport they will watch and details about the match. The user's device then sends this input data to the server. The server uses a dedicated application on a smartphone or tablet to provide the information through an intuitive user interface.

[0239] The server analyzes video feeds and related data of the match based on the received sports event information. Artificial intelligence then generates commentary information in real time based on this data and provides it to the user. For example, the server can identify important plays during a match and provide detailed explanations of their tactical significance and the rules governing them.

[0240] Furthermore, the server collects and analyzes information on event status within stadiums and arenas, as well as congestion at food and beverage facilities. This allows users to receive suggestions for the optimal timing to use the services. Specifically, the server uses AI to predict congestion levels, identify the best time to use the services, and notifies the user.

[0241] After the event, the server analyzes traffic conditions to match the user's expected return time. This allows the optimal return route to be sent to the user's device, enabling them to avoid congestion and return home smoothly. In this way, the system comprehensively supports the user's viewing experience and provides new value to sports viewing.

[0242] The following describes the processing flow.

[0243] Step 1:

[0244] The user uses a device to enter information about the sporting event they wish to attend. The device accepts data, including details such as the type of match and the date of the event, through a dedicated application.

[0245] Step 2:

[0246] The device sends the entered sports event information to the server. The device uses an encrypted network connection to securely transmit data.

[0247] Step 3:

[0248] The server analyzes the received sports event information and collects relevant match video feeds and historical data. The server uses AI algorithms to generate explanatory information on important plays and rules of the match.

[0249] Step 4:

[0250] The server generates commentary information and provides it to users watching the game in a timely manner. The commentary information is sent to the device and displayed as audio or text.

[0251] Step 5:

[0252] The server collects information on event status and congestion levels at food and beverage facilities within the stadium, and analyzes the information in real time using a predictive model. Based on the results, the server notifies spectators of the optimal time to use the services.

[0253] Step 6:

[0254] When a user begins preparing to go home after the game ends, the server investigates traffic conditions and analyzes the optimal route home. The server sends the analysis results to the user's device to help them get home efficiently.

[0255] (Example 1)

[0256] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0257] For spectators to enjoy sporting events more comfortably and meaningfully, real-time supplementary information about the matches and information about crowd conditions at the venue are necessary. However, current technology does not adequately provide this information in real time, which degrades the quality of the viewing experience. Furthermore, existing systems have limitations in providing customized information tailored to the interests and understanding of individual spectators. This invention aims to solve these problems.

[0258] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0259] In this invention, the server includes means for receiving sports event information input from a spectator's information processing device, means for analyzing competition data, and means for providing real-time commentary on important events during the match using a generative AI model. This allows spectators to gain a deep understanding of the current match situation and receive real-time information on crowd conditions and traffic at the venue.

[0260] A "spectator information processing device" refers to an electronic device used by users to input sports event information and transmit it to a server. Typically, this includes smartphones and tablets.

[0261] "Sports event information" refers to data that includes information such as the type of sport a user will be watching, details of the match, location, and date and time.

[0262] "Competition data" refers to information that records the progress and content of a match in digital format, such as video feeds related to the match, player statistics, and statistical information.

[0263] A "generative AI model" refers to artificial intelligence technology that analyzes data during a match and generates value-added information, such as commentary, in real time.

[0264] "Explanatory information" refers to detailed information provided to users about the rules of the match, the tactical significance of plays, and the backgrounds of the players.

[0265] "Crowding status" refers to information about the level of crowding at event venues, restaurants, and other facilities, and is obtained using sensors and other data collection methods.

[0266] "Traffic conditions" refers to information regarding congestion, accidents, and other issues on roads and public transportation used by spectators on their way home.

[0267] This invention is a system for improving the sports viewing experience for spectators, and is realized by using an information processing device, a central tabulation device, and a generative AI model.

[0268] First, users input detailed information about the sporting event they wish to attend using an information processing device such as a smartphone or tablet. The information processing device then securely transmits this input data to a central tallying device using the HTTPS protocol. This allows users to provide information with intuitive operation.

[0269] The central data aggregation system analyzes competition data in real time based on received sports event information. Using a generative AI model, it detects important events from the in-game data and generates commentary information based on these events. In particular, the generative AI model generates and notifies users of structured information in real time, providing commentary on important plays and their strategic significance.

[0270] Furthermore, the central scoring system analyzes the congestion situation within the venue using sensors and other information gathering methods, and notifies users of the optimal timing for using the service. In addition, after the match ends, it analyzes traffic conditions in real time and optimizes and sends guidance to users on their return journey to their terminals.

[0271] A concrete example would be a user entering information such as "Soccer, FC Tokyo vs. Kawasaki Frontale, April 10th, Ajinomoto Stadium." The central scoring system would then analyze any unusual plays during the match based on this information, explain the details to the user, and inform them of the best time to avoid congestion.

[0272] An example of a prompt message is: "Generate explanatory information for the sports viewing system. The type of match is soccer, and the match is between FC Tokyo and Kawasaki Frontale. What information will you provide to the customer?"

[0273] This invention utilizes these technologies to provide spectators with real-time and personalized information about sporting events, thereby improving the viewing experience.

[0274] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0275] Step 1:

[0276] Users enter detailed information about the sporting event they are watching using their smartphones or tablets. This includes the type of match, date and time, location, and participating teams. The entered information is transmitted by the information processing unit to the central tallying unit. This transmission is secure, using the HTTPS protocol. The output is the event information received by the central tallying unit.

[0277] Step 2:

[0278] The server matches and collects relevant competition data based on sports event information received from users. During this process, video feeds and statistical information for the relevant matches are retrieved from the database. Inputs are user information and a dynamic database, while output is a competition dataset that serves as the basis for analysis.

[0279] Step 3:

[0280] The server inputs the collected competition dataset into a generating AI model, which analyzes important match events in real time. During this process, the AI ​​model analyzes the data and detects tactically important plays and match highlights. Based on this, detailed commentary information is generated. The output is commentary information for the user.

[0281] Step 4:

[0282] The server transmits the generated explanatory information to the user's information processing device. This information is presented to the user visually or audibly. As a specific operation, the explanatory information is provided immediately by push notification via the notification API. The output is the explanatory information displayed on the terminal.

[0283] Step 5:

[0284] The server collects and analyzes the congestion situation in the game venue in real time from sensors and the like. This data is used to analyze the congestion prediction and the current situation using machine learning algorithms. The output is the congestion prediction information, and the optimal service utilization timing is calculated.

[0285] Step 6:

[0286] The server uses the congestion prediction information to notify the user of the optimal service utilization timing. This includes the optimal time zone to avoid congestion in food and beverage facilities and toilets in the stadium. As a specific operation, an alert is sent to the user terminal. The output is the timing proposal information presented on the terminal.

[0287] Step 7:

[0288] After the game ends, the server collects traffic information and optimizes the home route of the spectators. The real-time traffic data is analyzed, and a route to avoid traffic jams is generated. Based on this, a home route guide is sent to the user's terminal. The output is the optimized home route information.

[0289] (Application Example 1)

[0290] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0291] In recent years, providing individually optimized information and less crowded experiences to spectators and shoppers has become a crucial challenge. In particular, providing real-time, customized information and efficient route suggestions during sports events and in-store purchases is difficult and a major source of stress. This invention aims to solve these problems and provide spectators and customers with a more fulfilling experience.

[0292] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0293] In this invention, the server includes means for receiving event information or shopping information input from a spectator's or customer's terminal, means for analyzing data to generate explanatory information or product information based on the information, and means for transmitting and presenting the explanatory information or product information to the spectator's or customer's terminal. This enables the provision of individually optimized information and efficient service use for spectators and customers.

[0294] A "spectator" is someone who watches a sporting event or other form of entertainment, and is also the recipient of information about it.

[0295] A "terminal" is a device used for inputting or displaying information, and includes smartphones, tablets, and other similar devices.

[0296] "Event information" refers to information related to a specific sports event or activity, including the date, time, location, and details of the competition.

[0297] "Explanatory information" refers to information that provides a detailed explanation of the content of a sporting event, including the situation of the match and its tactical implications.

[0298] "Product information" refers to information about the location and promotion of products in physical stores, and is intended to guide customers when they make a purchase.

[0299] "Data" refers to a collection of various information obtained from spectators or customers and information related to events.

[0300] "Analyze" refers to the process of evaluating the information obtained and grasping trends and characteristics.

[0301] "Transmit" refers to the act of sending information to other devices or servers.

[0302] "Present" refers to the act of showing or explaining information to users through a device.

[0303] "Congestion situation" refers to the state indicating how much a specific area or service is utilized.

[0304] "Predict" refers to the act of estimating future situations based on existing data.

[0305] "Service usage timing" refers to the time when spectators or customers can use the service most effectively.

[0306] "Purchase timing" refers to the time most suitable for customers to purchase goods.

[0307] "Traffic situation" refers to information indicating the congestion level and flow of roads and public transportation.

[0308] "Moving conditions" refer to the environment and situation when spectators or customers move from one location to another.

[0309] "Generative AI model" refers to artificial intelligence technology that extracts features from data and performs analysis and prediction.

[0310] "Prompt text" refers to an instruction text for performing specific information processing on a generative AI model.

[0311] The system for implementing this invention is for receiving information transmitted from spectators' or customers' terminals and providing optimized information based on that information. First, the user inputs event information or purchase information using a terminal such as a smartphone or tablet. This information is transmitted to the server.

[0312] The server analyzes the received information using a generative AI model. This AI model is built using frameworks such as TensorFlow and PyTorch, extracts features from the information, and generates commentary for spectators and product information for customers. The information is analyzed in real time and transmitted to the terminal. This information is either displayed visually on a screen or output as audio.

[0313] Furthermore, the server predicts congestion levels within the facility and generates prompts to inform spectators and customers of the optimal time to use the service. This prediction utilizes machine learning algorithms and is updated in real time. For example, a user visiting a shopping mall on the weekend can use this app to be guided to less crowded routes or receive information on items on sale.

[0314] Furthermore, the server analyzes traffic conditions or movement patterns within the store to provide indicators for offering the optimal return route or travel path for spectators and customers. This information is sent to the user as a prompt.

[0315] Example of a prompt:

[0316] "Analyze the in-store camera feeds to predict congestion levels in front of product shelves."

[0317] "Please take into account the current customer traffic patterns and suggest the optimal checkout wait time."

[0318] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0319] Step 1:

[0320] The user uses a terminal to enter event or purchase information. This input data includes the type and date of the event, and a list of items the user wishes to purchase. The terminal then pre-processes this data and prepares it for transmission to the server.

[0321] Step 2:

[0322] The server retrieves user input data received from the terminal. At this time, it checks the validity of the data to ensure that there is no inaccurate information. The received data is then passed to the generation AI model, which analyzes the necessary information and prepares to generate explanatory and product information.

[0323] Step 3:

[0324] The generative AI model analyzes the received data. A pre-trained model is used for data analysis, generating optimal information based on the user's individual interests and behavior. The output includes explanatory and product information, which is managed on the server.

[0325] Step 4:

[0326] The server sends the generated explanatory or product information to the user's device. The user's device displays or outputs the received information as audio. This allows the user to receive the necessary information in real time.

[0327] Step 5:

[0328] The server uses data from sensors and cameras within the facility to predict congestion levels and traffic flow. Machine learning algorithms analyze this data in real time and prepare to generate prompt messages to avoid congestion.

[0329] Step 6:

[0330] Based on congestion levels and predicted information, the server generates prompt messages and suggests the optimal time for the user to use the service. The terminal receives these suggestions and notifies the user. The user can then use the service efficiently by following the suggestions.

[0331] Step 7:

[0332] The server analyzes traffic conditions or in-store movement data to determine the optimal return route or travel path. The generated results are sent to the user's terminal and presented as navigation.

[0333] Step 8:

[0334] Users follow the on-screen navigation to return home or travel safely and smoothly. This promotes efficient travel with minimal congestion and waiting times.

[0335] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0336] This invention is a system designed to enhance the experience of sports spectators, providing dynamic content based on the spectator's real-time emotions and interests. This system, which incorporates an emotion engine, is realized by integrating spectator terminals, servers, and emotion recognition technology.

[0337] First, the user selects a sporting event to watch via their device, and the device sends this information to the server. At this time, the device uses its camera and microphone to detect the user's emotions. The emotion engine analyzes the user's facial expressions and tone of voice to recognize emotions such as joy and surprise.

[0338] The server analyzes the information based on the sports event selected by the user and the detected sentiment data, and generates explanatory information according to the user's interest and level of understanding. For example, if the user expresses surprise, the server will provide a detailed explanation of the importance and background of that play.

[0339] Furthermore, the server predicts the status of events and the congestion levels of restaurants and bars, and the emotion engine considers the user's stress and level of concentration to suggest the optimal time to visit. For example, if a user is intently watching a game, it will suggest postponing their dining or drinking time to avoid congestion.

[0340] After the event, the server analyzes the user's emotional history through an emotion engine and provides the device with the optimal route home, taking traffic conditions into consideration. This feedback loop continuously improves the user's sports viewing experience.

[0341] In this way, this system changes and adjusts the service according to the emotions of the spectators, making it an enjoyable and easy-to-understand experience as part of the viewing experience.

[0342] The following describes the processing flow.

[0343] Step 1:

[0344] The user uses their device to input information about the sporting event they want to watch. When the device sends the entered event information to the server, it simultaneously collects the user's facial expressions and voice using its camera and microphone.

[0345] Step 2:

[0346] The device sends the user's emotional data collected to the server. The device's emotion engine performs initial analysis to recognize emotions from facial expressions and voice.

[0347] Step 3:

[0348] Based on the sports event information and sentiment data received by the server, AI is used to generate real-time commentary on the match. The server adjusts the commentary content, taking into account the user's level of interest and understanding.

[0349] Step 4:

[0350] The server generates explanatory information and sends it to the user's device, which then presents the information to the user via audio or text display. In particular, if the user's reaction is one of surprise, the server will provide a more detailed explanation of the background and specifics of the play.

[0351] Step 5:

[0352] The server predicts the status of events taking place in the stadium and the congestion levels of food and beverage facilities, and uses an emotion engine to assess the user's stress level. The server calculates the best time for the user to use the service comfortably and notifies the device.

[0353] Step 6:

[0354] After the viewing session ends, the server analyzes the user's emotional history and calculates the optimal route home. The terminal then provides this route information to the user, helping them to return home efficiently while avoiding congestion.

[0355] (Example 2)

[0356] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0357] Providing personalized commentary tailored to the emotions and interests of spectators in real time during sports events is difficult. Furthermore, optimal service utilization and return-home suggestions that take into account event crowds and traffic information are insufficient. This can lead to inconvenience for spectators and detract from the overall viewing experience.

[0358] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0359] In this invention, the server includes means for processing spectator behavior and audio signals and identifying emotions, means for predicting the status of the event and the congestion of food and beverage facilities, and means for analyzing traffic conditions and providing spectators with the optimal route home. This enables the provision of personalized commentary tailored to the emotions of spectators, as well as the suggestion of optimal services and routes during and after the event.

[0360] An "information processing device" is a device used for inputting, processing, and outputting data, and refers to terminal equipment used by spectators.

[0361] "Sports competition information" refers to a collection of data and information related to sports matches and competitions.

[0362] "Explanatory information" refers to the commentary and background information provided to spectators, including detailed information related to the competition.

[0363] "Competition information" refers to data and statistics related to a specific match or competition, including tactics, player data, and scores.

[0364] "Emotional analysis means" refers to techniques or methods for identifying emotional states from the facial expressions and voices of spectators.

[0365] "Presenting visually or aurally" refers to providing information through a device as a screen display or audio output.

[0366] "Event status" refers to information related to the ongoing status and schedule of an event or match.

[0367] "Crowding situation" refers to information about how crowded a place or facility is where spectators gather.

[0368] "Return route" refers to the paths or routes that spectators can choose to take when returning home after the event has ended.

[0369] A "generative AI model" refers to a model that uses artificial intelligence technology to generate useful information and predictions from large amounts of data.

[0370] This invention provides a system that enhances the viewing experience by offering content such as commentary that dynamically changes in response to the emotions of the spectators. The embodiments thereof are described in detail below.

[0371] Users select the sport they wish to watch using a mobile device or computer. This device receives input from the spectator and transmits the sport information to a server. The device also captures the spectator's facial expressions and voice through input devices such as cameras and microphones, and uses this information for emotion analysis. This analysis uses emotion analysis tools to identify the spectator's reactions (joy, surprise, etc.) in real time.

[0372] The server uses a generative AI model to generate the most relevant explanatory information for the spectator based on the received athletic competition information and spectator emotion data. This provides relevant background information and tactical commentary according to the emotions the spectator expresses in response to specific moments in the match. For example, if a spectator expresses surprise at a crucial scoring moment in the match, a detailed explanation of the background leading up to that goal and the strategic movements of the players will be notified to the terminal.

[0373] Furthermore, the server analyzes historical data and real-time information to predict the event's progress and the congestion of food and beverage facilities. This allows it to suggest optimal timings for eating and drinking, as well as event viewing plans, to spectators. After the event ends, it also considers current traffic conditions and provides spectators with ideal routes home. The terminal visually displays the route home, supporting spectators in moving smoothly.

[0374] An example of a prompt would be, "Explain how to provide a user who is surprised by a soccer goal with a detailed explanation of the player's background and similar past tactics." This prompt allows the generative AI model to provide viewers with appropriate information, resulting in a more enriching viewing experience.

[0375] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0376] Step 1:

[0377] The user selects the sports they wish to watch using their device. The user's selection information is retrieved as input, and the device sends this information to the server. This allows the server to prepare data on the sports that the spectator is interested in.

[0378] Step 2:

[0379] The device uses its built-in camera and microphone to collect the user's facial expressions and voice in real time. Video and audio data are acquired as input, and this data is used for analysis by emotion analysis tools. The emotion engine analyzes this data to identify the user's emotional state (joy, surprise, etc.) in real time. Identified emotion data is generated as output.

[0380] Step 3:

[0381] The server receives athletic competition information from the user and emotional data acquired from the device as input. Using a generative AI model, the server generates explanatory information tailored to the spectator based on the input information. This process dynamically generates commentary that responds to the match situation and the spectator's emotions. Customized explanatory information is generated as output.

[0382] Step 4:

[0383] The server sends the explanatory information generated above to the terminal. The terminal receives this and presents it to the user visually or audibly. The explanatory information, as input, is provided in a format that is easy for the user to understand and includes specific explanations to enhance the viewing experience. As output, the information is presented to the user in a format that is easy for them to understand.

[0384] Step 5:

[0385] The server analyzes the progress of events and the congestion levels of food and beverage facilities in real time. Using historical data and current schedule information as input, it predicts the optimal timing for service utilization. This process enables a function that suggests the best visit timing and break plans to the user. The output generates suggestions for the optimal service timing for the user.

[0386] Step 6:

[0387] After the event ends, the server collects and analyzes traffic information to calculate the optimal route home for the user. Traffic data is used as input, and the server performs calculations to determine the shortest or most efficient route for the user. As output, a specific route home is sent to the terminal, providing the user with visual instructions.

[0388] (Application Example 2)

[0389] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0390] Conventional sports viewing systems have limited personalized explanations and information based on the viewer's emotions and interests, making it difficult to extend and optimize the viewing experience within the home environment. Furthermore, there has been a lack of established means to control home automation devices in real time in response to changes in emotions during viewing, thereby providing a comfortable viewing environment. As a result, there has been a challenge in providing an environment where viewers can enjoy themselves to the fullest.

[0391] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0392] In this invention, the server includes means for receiving sports activity information input from a spectator's information terminal, means for analyzing the activity data and generating explanatory information to be provided to the spectator, means for transmitting the explanatory information to the spectator's information terminal and presenting it by display or audio, means for predicting the progress of the event and the congestion of food and beverage facilities and suggesting the optimal timing for using the services, means for analyzing traffic conditions and providing the optimal route home, and means for controlling a home automation device for adjusting the home environment based on the spectator's emotions. This makes it possible to optimize the home environment according to the spectator's emotions and provide a personalized viewing experience.

[0393] A "spectator information terminal" is a device equipped with the function of receiving sports activity information and displaying or audio-presenting the provided explanatory information.

[0394] "Sports activity information" refers to information about events that spectators are watching, including the status of the matches.

[0395] "Descriptive information" refers to data generated to provide spectators with details and related information about the match.

[0396] "Activity data" refers to various data related to sports activities, mainly including the progress of matches and statistical data.

[0397] "Event progress" refers to information showing the current status and timeline of ongoing sports activities.

[0398] "Food and beverage facility congestion status" refers to information regarding the usage status of food and beverage spaces at sports event venues.

[0399] "Optimal service usage timing" refers to the time when spectators can use events and facilities efficiently and comfortably.

[0400] "Traffic conditions" refers to information regarding traffic flow and congestion around the event venue and on the return route.

[0401] "Home automation devices" are devices used to control the environment within a home, and include lighting, air conditioning, robots, and other similar devices.

[0402] "Emotion-based adjustment" refers to analyzing the emotional state of spectators and automatically changing the lighting and temperature in the home based on that analysis.

[0403] This invention is a system that dynamically utilizes spectator emotional data to optimize the in-home sports viewing experience. The server receives sports activity information from the spectator's information terminal, analyzes the match activity data based on that information, and generates explanatory information to provide to the spectator. This explanatory information is transmitted to the spectator's information terminal and presented via display or audio.

[0404] Furthermore, the server predicts the progress of the event and the congestion levels of food and beverage facilities, and based on this data, suggests the optimal timing for spectators to use the services. It also analyzes traffic conditions and provides the best routes home for spectators, ensuring smooth travel.

[0405] In addition, by controlling home automation devices, the home environment can be dynamically adjusted based on the spectator's emotions. For example, if a spectator becomes excited, the brightness of smart lighting can be increased and the air conditioning adjusted to create a comfortable environment. Home robots can also provide beverages at the appropriate time. This creates an environment where spectators can enjoy the game with greater concentration.

[0406] In summary, this invention aims to provide a comfortable and efficient viewing environment through spectator emotional data, and its implementation utilizes open-source facial recognition libraries (e.g., OpenCV), home automation devices, information terminals, and other hardware and software.

[0407] By utilizing generative AI models, it is possible to improve the accuracy of analyzing and predicting emotional data. An example of a prompt would be: "Suggest ways to optimize the home environment based on the user's emotions while watching sports. For example, come up with ideas on how to adjust the lighting and air conditioning when the user is excited, and what a robot should provide."

[0408] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0409] Step 1:

[0410] The terminal sends sports activity information as input to the server. This allows information about matches that the user is interested in to be accumulated on the server, preparing the data that will serve as the basis for subsequent processing.

[0411] Step 2:

[0412] The server analyzes match activity data based on sports activity information received from terminals. This includes match progress and player statistics. The analyzed data is generated as explanatory information and customized to the user's interests.

[0413] Step 3:

[0414] The server sends the generated explanatory information to the user's device. The device receives this information and presents it on the screen or as audio. The user can view this information and deepen their understanding of the match.

[0415] Step 4:

[0416] The server uses machine learning algorithms to analyze collected data in order to predict the progress of events and the congestion levels of food and beverage facilities. The prediction results are used as a basis for deciding when to suggest the optimal time for users to use the services.

[0417] Step 5:

[0418] The server analyzes traffic conditions in real time and prepares data to guide users home via the most suitable route after watching a game. This information provides the optimal route, taking into account traffic flow and congestion.

[0419] Step 6:

[0420] The server analyzes user emotion data and generates instructions to control home automation devices. These instructions then adjust lighting, change air conditioning settings, and provide services via home robots, all in response to the user's emotions. The use of a generative AI model improves the accuracy of emotion analysis, further optimizing the user experience.

[0421] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0422] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0423] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0424] [Third Embodiment]

[0425] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0426] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0427] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0428] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0429] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0430] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0431] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0432] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0433] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0434] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0435] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0436] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0437] This invention is a system designed to enhance the sports viewing experience for spectators, and is realized through the use of spectator terminals, servers, and AI technology. In this system, spectators can receive necessary commentary and facility usage information in real time by sending information about sports events from their terminals to the server.

[0438] First, the user enters information about the sporting event using their device. This includes the type of sport they will watch and details about the match. The user's device then sends this input data to the server. The server uses a dedicated application on a smartphone or tablet to provide the information through an intuitive user interface.

[0439] The server analyzes video feeds and related data of the match based on the received sports event information. Artificial intelligence then generates commentary information in real time based on this data and provides it to the user. For example, the server can identify important plays during a match and provide detailed explanations of their tactical significance and the rules governing them.

[0440] Furthermore, the server collects and analyzes information on event status within stadiums and arenas, as well as congestion at food and beverage facilities. This allows users to receive suggestions for the optimal timing to use the services. Specifically, the server uses AI to predict congestion levels, identify the best time to use the services, and notifies the user.

[0441] After the event, the server analyzes traffic conditions to match the user's expected return time. This allows the optimal return route to be sent to the user's device, enabling them to avoid congestion and return home smoothly. In this way, the system comprehensively supports the user's viewing experience and provides new value to sports viewing.

[0442] The following describes the processing flow.

[0443] Step 1:

[0444] The user uses a device to enter information about the sporting event they wish to attend. The device accepts data, including details such as the type of match and the date of the event, through a dedicated application.

[0445] Step 2:

[0446] The device sends the entered sports event information to the server. The device uses an encrypted network connection to securely transmit data.

[0447] Step 3:

[0448] The server analyzes the received sports event information and collects relevant match video feeds and historical data. The server uses AI algorithms to generate explanatory information on important plays and rules of the match.

[0449] Step 4:

[0450] The server generates commentary information and provides it to users watching the game in a timely manner. The commentary information is sent to the device and displayed as audio or text.

[0451] Step 5:

[0452] The server collects information on event status and congestion levels at food and beverage facilities within the stadium, and analyzes the information in real time using a predictive model. Based on the results, the server notifies spectators of the optimal time to use the services.

[0453] Step 6:

[0454] When a user begins preparing to go home after the game ends, the server investigates traffic conditions and analyzes the optimal route home. The server sends the analysis results to the user's device to help them get home efficiently.

[0455] (Example 1)

[0456] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0457] For spectators to enjoy sporting events more comfortably and meaningfully, real-time supplementary information about the matches and information about crowd conditions at the venue are necessary. However, current technology does not adequately provide this information in real time, which degrades the quality of the viewing experience. Furthermore, existing systems have limitations in providing customized information tailored to the interests and understanding of individual spectators. This invention aims to solve these problems.

[0458] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0459] In this invention, the server includes means for receiving sports event information input from a spectator's information processing device, means for analyzing competition data, and means for providing real-time commentary on important events during the match using a generative AI model. This allows spectators to gain a deep understanding of the current match situation and receive real-time information on crowd conditions and traffic at the venue.

[0460] A "spectator information processing device" refers to an electronic device used by users to input sports event information and transmit it to a server. Typically, this includes smartphones and tablets.

[0461] "Sports event information" refers to data that includes information such as the type of sport a user will be watching, details of the match, location, and date and time.

[0462] "Competition data" refers to information that records the progress and content of a match in digital format, such as video feeds related to the match, player statistics, and statistical information.

[0463] A "generative AI model" refers to artificial intelligence technology that analyzes data during a match and generates value-added information, such as commentary, in real time.

[0464] "Explanatory information" refers to detailed information provided to users about the rules of the match, the tactical significance of plays, and the backgrounds of the players.

[0465] "Crowding status" refers to information about the level of crowding at event venues, restaurants, and other facilities, and is obtained using sensors and other data collection methods.

[0466] "Traffic conditions" refers to information regarding congestion, accidents, and other issues on roads and public transportation used by spectators on their way home.

[0467] This invention is a system for improving the sports viewing experience for spectators, and is realized by using an information processing device, a central tabulation device, and a generative AI model.

[0468] First, users input detailed information about the sporting event they wish to attend using an information processing device such as a smartphone or tablet. The information processing device then securely transmits this input data to a central tallying device using the HTTPS protocol. This allows users to provide information with intuitive operation.

[0469] The central data aggregation system analyzes competition data in real time based on received sports event information. Using a generative AI model, it detects important events from the in-game data and generates commentary information based on these events. In particular, the generative AI model generates and notifies users of structured information in real time, providing commentary on important plays and their strategic significance.

[0470] Furthermore, the central scoring system analyzes the congestion situation within the venue using sensors and other information gathering methods, and notifies users of the optimal timing for using the service. In addition, after the match ends, it analyzes traffic conditions in real time and optimizes and sends guidance to users on their return journey to their terminals.

[0471] A concrete example would be a user entering information such as "Soccer, FC Tokyo vs. Kawasaki Frontale, April 10th, Ajinomoto Stadium." The central scoring system would then analyze any unusual plays during the match based on this information, explain the details to the user, and inform them of the best time to avoid congestion.

[0472] An example of a prompt message is: "Generate explanatory information for the sports viewing system. The type of match is soccer, and the match is between FC Tokyo and Kawasaki Frontale. What information will you provide to the customer?"

[0473] This invention utilizes these technologies to provide spectators with real-time and personalized information about sporting events, thereby improving the viewing experience.

[0474] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0475] Step 1:

[0476] Users enter detailed information about the sporting event they are watching using their smartphones or tablets. This includes the type of match, date and time, location, and participating teams. The entered information is transmitted by the information processing unit to the central tallying unit. This transmission is secure, using the HTTPS protocol. The output is the event information received by the central tallying unit.

[0477] Step 2:

[0478] The server matches and collects relevant competition data based on sports event information received from users. During this process, video feeds and statistical information for the relevant matches are retrieved from the database. Inputs are user information and a dynamic database, while output is a competition dataset that serves as the basis for analysis.

[0479] Step 3:

[0480] The server inputs the collected competition dataset into a generating AI model, which analyzes important match events in real time. During this process, the AI ​​model analyzes the data and detects tactically important plays and match highlights. Based on this, detailed commentary information is generated. The output is commentary information for the user.

[0481] Step 4:

[0482] The server sends the generated explanatory information to the user's information processing device. This information is presented to the user visually or audibly. Specifically, the explanatory information is provided immediately via push notifications through the notification API. The output is the explanatory information displayed on the device.

[0483] Step 5:

[0484] The server collects and analyzes real-time congestion data from sensors and other sources within the match venue. This data is used with machine learning algorithms to predict congestion and analyze the current situation. The output is congestion prediction information, which is used to calculate the optimal time to use the service.

[0485] Step 6:

[0486] The server uses congestion forecast information to notify users of the optimal time to use the service. This includes the best times to avoid congestion at food and beverage facilities and restrooms within the stadium. Specifically, an alert is sent to the user's terminal. The output is the timing suggestion information presented to the terminal.

[0487] Step 7:

[0488] After the match ends, the server collects traffic information and optimizes the return routes for spectators. It analyzes real-time traffic data to generate routes that avoid congestion. Based on this, return route guidance is sent to the user's device. The output is optimized return route information.

[0489] (Application Example 1)

[0490] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0491] In recent years, providing individually optimized information and less crowded experiences to spectators and shoppers has become a crucial challenge. In particular, providing real-time, customized information and efficient route suggestions during sports events and in-store purchases is difficult and a major source of stress. This invention aims to solve these problems and provide spectators and customers with a more fulfilling experience.

[0492] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0493] In this invention, the server includes means for receiving event information or shopping information input from a spectator's or customer's terminal, means for analyzing data to generate explanatory information or product information based on the information, and means for transmitting and presenting the explanatory information or product information to the spectator's or customer's terminal. This enables the provision of individually optimized information and efficient service use for spectators and customers.

[0494] A "spectator" is someone who watches a sporting event or other form of entertainment, and is also the recipient of information about it.

[0495] A "terminal" is a device used for inputting or displaying information, and includes smartphones, tablets, and other similar devices.

[0496] "Event information" refers to information related to a specific sports event or activity, including the date, time, location, and details of the competition.

[0497] "Explanatory information" refers to information that provides a detailed explanation of the content of a sporting event, including the situation of the match and its tactical implications.

[0498] "Product information" refers to information about the location and promotion of products in physical stores, and is intended to guide customers when they make a purchase.

[0499] "Data" refers to a collection of information obtained from spectators or customers, as well as various other pieces of information related to the event.

[0500] "Analyzing" is the process of evaluating the information obtained and understanding trends and characteristics.

[0501] "To send" refers to the act of delivering information to another device or server.

[0502] "To present" refers to the act of showing or explaining information to a user through a device.

[0503] "Congestion level" refers to the extent to which a particular area or service is being used.

[0504] "Predicting" is the act of estimating future conditions based on existing data.

[0505] "Service utilization timing" refers to the time when spectators or customers can most effectively utilize the service.

[0506] "Purchase timing" refers to the most opportune time for a customer to buy a product.

[0507] "Traffic conditions" refers to information that indicates the degree of congestion and flow of traffic on roads and public transportation.

[0508] "Movement conditions" refer to the environment and circumstances when spectators or customers move from one point to another.

[0509] A "generative AI model" is an artificial intelligence technology that extracts features from data and performs analysis and predictions.

[0510] A "prompt statement" is an instruction statement used to perform specific information processing on a generating AI model.

[0511] The system for implementing this invention is for receiving information transmitted from spectators' or customers' terminals and providing optimized information based on that information. First, the user inputs event information or purchase information using a terminal such as a smartphone or tablet. This information is transmitted to the server.

[0512] The server analyzes the received information using a generative AI model. This AI model is built using frameworks such as TensorFlow and PyTorch, extracts features from the information, and generates commentary for spectators and product information for customers. The information is analyzed in real time and transmitted to the terminal. This information is either displayed visually on a screen or output as audio.

[0513] Furthermore, the server predicts congestion levels within the facility and generates prompts to inform spectators and customers of the optimal time to use the service. This prediction utilizes machine learning algorithms and is updated in real time. For example, a user visiting a shopping mall on the weekend can use this app to be guided to less crowded routes or receive information on items on sale.

[0514] Furthermore, the server analyzes traffic conditions or movement patterns within the store to provide indicators for offering the optimal return route or travel path for spectators and customers. This information is sent to the user as a prompt.

[0515] Example of a prompt:

[0516] "Analyze the in-store camera feeds to predict congestion levels in front of product shelves."

[0517] "Please take into account the current customer traffic patterns and suggest the optimal checkout wait time."

[0518] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0519] Step 1:

[0520] The user uses a terminal to enter event or purchase information. This input data includes the type and date of the event, and a list of items the user wishes to purchase. The terminal then pre-processes this data and prepares it for transmission to the server.

[0521] Step 2:

[0522] The server retrieves user input data received from the terminal. At this time, it checks the validity of the data to ensure that there is no inaccurate information. The received data is then passed to the generation AI model, which analyzes the necessary information and prepares to generate explanatory and product information.

[0523] Step 3:

[0524] The generative AI model analyzes the received data. A pre-trained model is used for data analysis, generating optimal information based on the user's individual interests and behavior. The output includes explanatory and product information, which is managed on the server.

[0525] Step 4:

[0526] The server sends the generated explanatory or product information to the user's device. The user's device displays or outputs the received information as audio. This allows the user to receive the necessary information in real time.

[0527] Step 5:

[0528] The server uses data from sensors and cameras within the facility to predict congestion levels and traffic flow. Machine learning algorithms analyze this data in real time and prepare to generate prompt messages to avoid congestion.

[0529] Step 6:

[0530] Based on congestion levels and predicted information, the server generates prompt messages and suggests the optimal time for the user to use the service. The terminal receives these suggestions and notifies the user. The user can then use the service efficiently by following the suggestions.

[0531] Step 7:

[0532] The server analyzes traffic conditions or in-store movement data to determine the optimal return route or travel path. The generated results are sent to the user's terminal and presented as navigation.

[0533] Step 8:

[0534] Users follow the on-screen navigation to return home or travel safely and smoothly. This promotes efficient travel with minimal congestion and waiting times.

[0535] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0536] This invention is a system designed to enhance the experience of sports spectators, providing dynamic content based on the spectator's real-time emotions and interests. This system, which incorporates an emotion engine, is realized by integrating spectator terminals, servers, and emotion recognition technology.

[0537] First, the user selects a sporting event to watch via their device, and the device sends this information to the server. At this time, the device uses its camera and microphone to detect the user's emotions. The emotion engine analyzes the user's facial expressions and tone of voice to recognize emotions such as joy and surprise.

[0538] The server analyzes the information based on the sports event selected by the user and the detected sentiment data, and generates explanatory information according to the user's interest and level of understanding. For example, if the user expresses surprise, the server will provide a detailed explanation of the importance and background of that play.

[0539] Furthermore, the server predicts the status of events and the congestion levels of restaurants and bars, and the emotion engine considers the user's stress and level of concentration to suggest the optimal time to visit. For example, if a user is intently watching a game, it will suggest postponing their dining or drinking time to avoid congestion.

[0540] After the event, the server analyzes the user's emotional history through an emotion engine and provides the device with the optimal route home, taking traffic conditions into consideration. This feedback loop continuously improves the user's sports viewing experience.

[0541] In this way, this system changes and adjusts the service according to the emotions of the spectators, making it an enjoyable and easy-to-understand experience as part of the viewing experience.

[0542] The following describes the processing flow.

[0543] Step 1:

[0544] The user uses their device to input information about the sporting event they want to watch. When the device sends the entered event information to the server, it simultaneously collects the user's facial expressions and voice using its camera and microphone.

[0545] Step 2:

[0546] The device sends the user's emotional data collected to the server. The device's emotion engine performs initial analysis to recognize emotions from facial expressions and voice.

[0547] Step 3:

[0548] Based on the sports event information and sentiment data received by the server, AI is used to generate real-time commentary on the match. The server adjusts the commentary content, taking into account the user's level of interest and understanding.

[0549] Step 4:

[0550] The server generates explanatory information and sends it to the user's device, which then presents the information to the user via audio or text display. In particular, if the user's reaction is one of surprise, the server will provide a more detailed explanation of the background and specifics of the play.

[0551] Step 5:

[0552] The server predicts the status of events taking place in the stadium and the congestion levels of food and beverage facilities, and uses an emotion engine to assess the user's stress level. The server calculates the best time for the user to use the service comfortably and notifies the device.

[0553] Step 6:

[0554] After the viewing session ends, the server analyzes the user's emotional history and calculates the optimal route home. The terminal then provides this route information to the user, helping them to return home efficiently while avoiding congestion.

[0555] (Example 2)

[0556] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0557] Providing personalized commentary tailored to the emotions and interests of spectators in real time during sports events is difficult. Furthermore, optimal service utilization and return-home suggestions that take into account event crowds and traffic information are insufficient. This can lead to inconvenience for spectators and detract from the overall viewing experience.

[0558] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0559] In this invention, the server includes means for processing spectator behavior and audio signals and identifying emotions, means for predicting the status of the event and the congestion of food and beverage facilities, and means for analyzing traffic conditions and providing spectators with the optimal route home. This enables the provision of personalized commentary tailored to the emotions of spectators, as well as the suggestion of optimal services and routes during and after the event.

[0560] An "information processing device" is a device used for inputting, processing, and outputting data, and refers to terminal equipment used by spectators.

[0561] "Sports competition information" refers to a collection of data and information related to sports matches and competitions.

[0562] "Explanatory information" refers to the commentary and background information provided to spectators, including detailed information related to the competition.

[0563] "Competition information" refers to data and statistics related to a specific match or competition, including tactics, player data, and scores.

[0564] "Emotional analysis means" refers to techniques or methods for identifying emotional states from the facial expressions and voices of spectators.

[0565] "Presenting visually or aurally" refers to providing information through a device as a screen display or audio output.

[0566] "Event status" refers to information related to the ongoing status and schedule of an event or match.

[0567] "Crowding situation" refers to information about how crowded a place or facility is where spectators gather.

[0568] "Return route" refers to the paths or routes that spectators can choose to take when returning home after the event has ended.

[0569] A "generative AI model" refers to a model that uses artificial intelligence technology to generate useful information and predictions from large amounts of data.

[0570] This invention provides a system that enhances the viewing experience by offering content such as commentary that dynamically changes in response to the emotions of the spectators. The embodiments thereof are described in detail below.

[0571] Users select the sport they wish to watch using a mobile device or computer. This device receives input from the spectator and transmits the sport information to a server. The device also captures the spectator's facial expressions and voice through input devices such as cameras and microphones, and uses this information for emotion analysis. This analysis uses emotion analysis tools to identify the spectator's reactions (joy, surprise, etc.) in real time.

[0572] The server uses a generative AI model to generate the most relevant explanatory information for the spectator based on the received athletic competition information and spectator emotion data. This provides relevant background information and tactical commentary according to the emotions the spectator expresses in response to specific moments in the match. For example, if a spectator expresses surprise at a crucial scoring moment in the match, a detailed explanation of the background leading up to that goal and the strategic movements of the players will be notified to the terminal.

[0573] Furthermore, the server analyzes historical data and real-time information to predict the event's progress and the congestion of food and beverage facilities. This allows it to suggest optimal timings for eating and drinking, as well as event viewing plans, to spectators. After the event ends, it also considers current traffic conditions and provides spectators with ideal routes home. The terminal visually displays the route home, supporting spectators in moving smoothly.

[0574] An example of a prompt would be, "Explain how to provide a user who is surprised by a soccer goal with a detailed explanation of the player's background and similar past tactics." This prompt allows the generative AI model to provide viewers with appropriate information, resulting in a more enriching viewing experience.

[0575] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0576] Step 1:

[0577] The user selects the sports they wish to watch using their device. The user's selection information is retrieved as input, and the device sends this information to the server. This allows the server to prepare data on the sports that the spectator is interested in.

[0578] Step 2:

[0579] The device uses its built-in camera and microphone to collect the user's facial expressions and voice in real time. Video and audio data are acquired as input, and this data is used for analysis by emotion analysis tools. The emotion engine analyzes this data to identify the user's emotional state (joy, surprise, etc.) in real time. Identified emotion data is generated as output.

[0580] Step 3:

[0581] The server receives athletic competition information from the user and emotional data acquired from the device as input. Using a generative AI model, the server generates explanatory information tailored to the spectator based on the input information. This process dynamically generates commentary that responds to the match situation and the spectator's emotions. Customized explanatory information is generated as output.

[0582] Step 4:

[0583] The server sends the explanatory information generated above to the terminal. The terminal receives this and presents it to the user visually or audibly. The explanatory information, as input, is provided in a format that is easy for the user to understand and includes specific explanations to enhance the viewing experience. As output, the information is presented to the user in a format that is easy for them to understand.

[0584] Step 5:

[0585] The server analyzes the progress of events and the congestion levels of food and beverage facilities in real time. Using historical data and current schedule information as input, it predicts the optimal timing for service utilization. This process enables a function that suggests the best visit timing and break plans to the user. The output generates suggestions for the optimal service timing for the user.

[0586] Step 6:

[0587] After the event ends, the server collects and analyzes traffic information to calculate the optimal route home for the user. Traffic data is used as input, and the server performs calculations to determine the shortest or most efficient route for the user. As output, a specific route home is sent to the terminal, providing the user with visual instructions.

[0588] (Application Example 2)

[0589] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0590] Conventional sports viewing systems have limited personalized explanations and information based on the viewer's emotions and interests, making it difficult to extend and optimize the viewing experience within the home environment. Furthermore, there has been a lack of established means to control home automation devices in real time in response to changes in emotions during viewing, thereby providing a comfortable viewing environment. As a result, there has been a challenge in providing an environment where viewers can enjoy themselves to the fullest.

[0591] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0592] In this invention, the server includes means for receiving sports activity information input from a spectator's information terminal, means for analyzing the activity data and generating explanatory information to be provided to the spectator, means for transmitting the explanatory information to the spectator's information terminal and presenting it by display or audio, means for predicting the progress of the event and the congestion of food and beverage facilities and suggesting the optimal timing for using the services, means for analyzing traffic conditions and providing the optimal route home, and means for controlling a home automation device for adjusting the home environment based on the spectator's emotions. This makes it possible to optimize the home environment according to the spectator's emotions and provide a personalized viewing experience.

[0593] A "spectator information terminal" is a device equipped with the function of receiving sports activity information and displaying or audio-presenting the provided explanatory information.

[0594] "Sports activity information" refers to information about events that spectators are watching, including the status of the matches.

[0595] "Descriptive information" refers to data generated to provide spectators with details and related information about the match.

[0596] "Activity data" refers to various data related to sports activities, mainly including the progress of matches and statistical data.

[0597] "Event progress" refers to information showing the current status and timeline of ongoing sports activities.

[0598] "Food and beverage facility congestion status" refers to information regarding the usage status of food and beverage spaces at sports event venues.

[0599] "Optimal service usage timing" refers to the time when spectators can use events and facilities efficiently and comfortably.

[0600] "Traffic conditions" refers to information regarding traffic flow and congestion around the event venue and on the return route.

[0601] "Home automation devices" are devices used to control the environment within a home, and include lighting, air conditioning, robots, and other similar devices.

[0602] "Emotion-based adjustment" refers to analyzing the emotional state of spectators and automatically changing the lighting and temperature in the home based on that analysis.

[0603] This invention is a system that dynamically utilizes spectator emotional data to optimize the in-home sports viewing experience. The server receives sports activity information from the spectator's information terminal, analyzes the match activity data based on that information, and generates explanatory information to provide to the spectator. This explanatory information is transmitted to the spectator's information terminal and presented via display or audio.

[0604] Furthermore, the server predicts the progress of the event and the congestion levels of food and beverage facilities, and based on this data, suggests the optimal timing for spectators to use the services. It also analyzes traffic conditions and provides the best routes home for spectators, ensuring smooth travel.

[0605] In addition, by controlling home automation devices, the home environment can be dynamically adjusted based on the spectator's emotions. For example, if a spectator becomes excited, the brightness of smart lighting can be increased and the air conditioning adjusted to create a comfortable environment. Home robots can also provide beverages at the appropriate time. This creates an environment where spectators can enjoy the game with greater concentration.

[0606] In summary, this invention aims to provide a comfortable and efficient viewing environment through spectator emotional data, and its implementation utilizes open-source facial recognition libraries (e.g., OpenCV), home automation devices, information terminals, and other hardware and software.

[0607] By utilizing generative AI models, it is possible to improve the accuracy of analyzing and predicting emotional data. An example of a prompt would be: "Suggest ways to optimize the home environment based on the user's emotions while watching sports. For example, come up with ideas on how to adjust the lighting and air conditioning when the user is excited, and what a robot should provide."

[0608] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0609] Step 1:

[0610] The terminal sends sports activity information as input to the server. This allows information about matches that the user is interested in to be accumulated on the server, preparing the data that will serve as the basis for subsequent processing.

[0611] Step 2:

[0612] The server analyzes match activity data based on sports activity information received from terminals. This includes match progress and player statistics. The analyzed data is generated as explanatory information and customized to the user's interests.

[0613] Step 3:

[0614] The server sends the generated explanatory information to the user's device. The device receives this information and presents it on the screen or as audio. The user can view this information and deepen their understanding of the match.

[0615] Step 4:

[0616] The server uses machine learning algorithms to analyze collected data in order to predict the progress of events and the congestion levels of food and beverage facilities. The prediction results are used as a basis for deciding when to suggest the optimal time for users to use the services.

[0617] Step 5:

[0618] The server analyzes traffic conditions in real time and prepares data to guide users home via the most suitable route after watching a game. This information provides the optimal route, taking into account traffic flow and congestion.

[0619] Step 6:

[0620] The server analyzes user emotion data and generates instructions to control home automation devices. These instructions then adjust lighting, change air conditioning settings, and provide services via home robots, all in response to the user's emotions. The use of a generative AI model improves the accuracy of emotion analysis, further optimizing the user experience.

[0621] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0622] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0623] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0624] [Fourth Embodiment]

[0625] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0626] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0627] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0628] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0629] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0630] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0631] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0632] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0633] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0634] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0635] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0636] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0637] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0638] This invention is a system designed to enhance the sports viewing experience for spectators, and is realized through the use of spectator terminals, servers, and AI technology. In this system, spectators can receive necessary commentary and facility usage information in real time by sending information about sports events from their terminals to the server.

[0639] First, the user enters information about the sporting event using their device. This includes the type of sport they will watch and details about the match. The user's device then sends this input data to the server. The server uses a dedicated application on a smartphone or tablet to provide the information through an intuitive user interface.

[0640] The server analyzes video feeds and related data of the match based on the received sports event information. Artificial intelligence then generates commentary information in real time based on this data and provides it to the user. For example, the server can identify important plays during a match and provide detailed explanations of their tactical significance and the rules governing them.

[0641] Furthermore, the server collects and analyzes information on event status within stadiums and arenas, as well as congestion at food and beverage facilities. This allows users to receive suggestions for the optimal timing to use the services. Specifically, the server uses AI to predict congestion levels, identify the best time to use the services, and notifies the user.

[0642] After the event, the server analyzes traffic conditions to match the user's expected return time. This allows the optimal return route to be sent to the user's device, enabling them to avoid congestion and return home smoothly. In this way, the system comprehensively supports the user's viewing experience and provides new value to sports viewing.

[0643] The following describes the processing flow.

[0644] Step 1:

[0645] The user uses a device to enter information about the sporting event they wish to attend. The device accepts data, including details such as the type of match and the date of the event, through a dedicated application.

[0646] Step 2:

[0647] The device sends the entered sports event information to the server. The device uses an encrypted network connection to securely transmit data.

[0648] Step 3:

[0649] The server analyzes the received sports event information and collects relevant match video feeds and historical data. The server uses AI algorithms to generate explanatory information on important plays and rules of the match.

[0650] Step 4:

[0651] The server generates commentary information and provides it to users watching the game in a timely manner. The commentary information is sent to the device and displayed as audio or text.

[0652] Step 5:

[0653] The server collects information on event status and congestion levels at food and beverage facilities within the stadium, and analyzes the information in real time using a predictive model. Based on the results, the server notifies spectators of the optimal time to use the services.

[0654] Step 6:

[0655] When a user begins preparing to go home after the game ends, the server investigates traffic conditions and analyzes the optimal route home. The server sends the analysis results to the user's device to help them get home efficiently.

[0656] (Example 1)

[0657] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0658] For spectators to enjoy sporting events more comfortably and meaningfully, real-time supplementary information about the matches and information about crowd conditions at the venue are necessary. However, current technology does not adequately provide this information in real time, which degrades the quality of the viewing experience. Furthermore, existing systems have limitations in providing customized information tailored to the interests and understanding of individual spectators. This invention aims to solve these problems.

[0659] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0660] In this invention, the server includes means for receiving sports event information input from a spectator's information processing device, means for analyzing competition data, and means for providing real-time commentary on important events during the match using a generative AI model. This allows spectators to gain a deep understanding of the current match situation and receive real-time information on crowd conditions and traffic at the venue.

[0661] A "spectator information processing device" refers to an electronic device used by users to input sports event information and transmit it to a server. Typically, this includes smartphones and tablets.

[0662] "Sports event information" refers to data that includes information such as the type of sport a user will be watching, details of the match, location, and date and time.

[0663] "Competition data" refers to information that records the progress and content of a match in digital format, such as video feeds related to the match, player statistics, and statistical information.

[0664] A "generative AI model" refers to artificial intelligence technology that analyzes data during a match and generates value-added information, such as commentary, in real time.

[0665] "Explanatory information" refers to detailed information provided to users about the rules of the match, the tactical significance of plays, and the backgrounds of the players.

[0666] "Crowding status" refers to information about the level of crowding at event venues, restaurants, and other facilities, and is obtained using sensors and other data collection methods.

[0667] "Traffic conditions" refers to information regarding congestion, accidents, and other issues on roads and public transportation used by spectators on their way home.

[0668] This invention is a system for improving the sports viewing experience for spectators, and is realized by using an information processing device, a central tabulation device, and a generative AI model.

[0669] First, users input detailed information about the sporting event they wish to attend using an information processing device such as a smartphone or tablet. The information processing device then securely transmits this input data to a central tallying device using the HTTPS protocol. This allows users to provide information with intuitive operation.

[0670] The central data aggregation system analyzes competition data in real time based on received sports event information. Using a generative AI model, it detects important events from the in-game data and generates commentary information based on these events. In particular, the generative AI model generates and notifies users of structured information in real time, providing commentary on important plays and their strategic significance.

[0671] Furthermore, the central scoring system analyzes the congestion situation within the venue using sensors and other information gathering methods, and notifies users of the optimal timing for using the service. In addition, after the match ends, it analyzes traffic conditions in real time and optimizes and sends guidance to users on their return journey to their terminals.

[0672] A concrete example would be a user entering information such as "Soccer, FC Tokyo vs. Kawasaki Frontale, April 10th, Ajinomoto Stadium." The central scoring system would then analyze any unusual plays during the match based on this information, explain the details to the user, and inform them of the best time to avoid congestion.

[0673] An example of a prompt message is: "Generate explanatory information for the sports viewing system. The type of match is soccer, and the match is between FC Tokyo and Kawasaki Frontale. What information will you provide to the customer?"

[0674] This invention utilizes these technologies to provide spectators with real-time and personalized information about sporting events, thereby improving the viewing experience.

[0675] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0676] Step 1:

[0677] Users enter detailed information about the sporting event they are watching using their smartphones or tablets. This includes the type of match, date and time, location, and participating teams. The entered information is transmitted by the information processing unit to the central tallying unit. This transmission is secure, using the HTTPS protocol. The output is the event information received by the central tallying unit.

[0678] Step 2:

[0679] The server matches and collects relevant competition data based on sports event information received from users. During this process, video feeds and statistical information for the relevant matches are retrieved from the database. Inputs are user information and a dynamic database, while output is a competition dataset that serves as the basis for analysis.

[0680] Step 3:

[0681] The server inputs the collected competition dataset into a generating AI model, which analyzes important match events in real time. During this process, the AI ​​model analyzes the data and detects tactically important plays and match highlights. Based on this, detailed commentary information is generated. The output is commentary information for the user.

[0682] Step 4:

[0683] The server sends the generated explanatory information to the user's information processing device. This information is presented to the user visually or audibly. Specifically, the explanatory information is provided immediately via push notifications through the notification API. The output is the explanatory information displayed on the device.

[0684] Step 5:

[0685] The server collects and analyzes real-time congestion data from sensors and other sources within the match venue. This data is used with machine learning algorithms to predict congestion and analyze the current situation. The output is congestion prediction information, which is used to calculate the optimal time to use the service.

[0686] Step 6:

[0687] The server uses congestion forecast information to notify users of the optimal time to use the service. This includes the best times to avoid congestion at food and beverage facilities and restrooms within the stadium. Specifically, an alert is sent to the user's terminal. The output is the timing suggestion information presented to the terminal.

[0688] Step 7:

[0689] After the match ends, the server collects traffic information and optimizes the return routes for spectators. It analyzes real-time traffic data to generate routes that avoid congestion. Based on this, return route guidance is sent to the user's device. The output is optimized return route information.

[0690] (Application Example 1)

[0691] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0692] In recent years, providing individually optimized information and less crowded experiences to spectators and shoppers has become a crucial challenge. In particular, providing real-time, customized information and efficient route suggestions during sports events and in-store purchases is difficult and a major source of stress. This invention aims to solve these problems and provide spectators and customers with a more fulfilling experience.

[0693] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0694] In this invention, the server includes means for receiving event information or shopping information input from a spectator's or customer's terminal, means for analyzing data to generate explanatory information or product information based on the information, and means for transmitting and presenting the explanatory information or product information to the spectator's or customer's terminal. This enables the provision of individually optimized information and efficient service use for spectators and customers.

[0695] A "spectator" is someone who watches a sporting event or other form of entertainment, and is also the recipient of information about it.

[0696] A "terminal" is a device used for inputting or displaying information, and includes smartphones, tablets, and other similar devices.

[0697] "Event information" refers to information related to a specific sports event or activity, including the date, time, location, and details of the competition.

[0698] "Explanatory information" refers to information that provides a detailed explanation of the content of a sporting event, including the situation of the match and its tactical implications.

[0699] "Product information" refers to information about the location and promotion of products in physical stores, and is intended to guide customers when they make a purchase.

[0700] "Data" refers to a collection of information obtained from spectators or customers, as well as various other pieces of information related to the event.

[0701] "Analyzing" is the process of evaluating the information obtained and understanding trends and characteristics.

[0702] "To send" refers to the act of delivering information to another device or server.

[0703] "To present" refers to the act of showing or explaining information to a user through a device.

[0704] "Congestion level" refers to the extent to which a particular area or service is being used.

[0705] "Predicting" is the act of estimating future conditions based on existing data.

[0706] "Service utilization timing" refers to the time when spectators or customers can most effectively utilize the service.

[0707] "Purchase timing" refers to the most opportune time for a customer to buy a product.

[0708] "Traffic conditions" refers to information that indicates the degree of congestion and flow of traffic on roads and public transportation.

[0709] "Movement conditions" refer to the environment and circumstances when spectators or customers move from one point to another.

[0710] A "generative AI model" is an artificial intelligence technology that extracts features from data and performs analysis and predictions.

[0711] A "prompt statement" is an instruction statement used to perform specific information processing on a generating AI model.

[0712] The system for implementing this invention is for receiving information transmitted from spectators' or customers' terminals and providing optimized information based on that information. First, the user inputs event information or purchase information using a terminal such as a smartphone or tablet. This information is transmitted to the server.

[0713] The server analyzes the received information using a generative AI model. This AI model is built using frameworks such as TensorFlow and PyTorch, extracts features from the information, and generates commentary for spectators and product information for customers. The information is analyzed in real time and transmitted to the terminal. This information is either displayed visually on a screen or output as audio.

[0714] Furthermore, the server predicts congestion levels within the facility and generates prompts to inform spectators and customers of the optimal time to use the service. This prediction utilizes machine learning algorithms and is updated in real time. For example, a user visiting a shopping mall on the weekend can use this app to be guided to less crowded routes or receive information on items on sale.

[0715] Furthermore, the server analyzes traffic conditions or movement patterns within the store to provide indicators for offering the optimal return route or travel path for spectators and customers. This information is sent to the user as a prompt.

[0716] Example of a prompt:

[0717] "Analyze the in-store camera feeds to predict congestion levels in front of product shelves."

[0718] "Please take into account the current customer traffic patterns and suggest the optimal checkout wait time."

[0719] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0720] Step 1:

[0721] The user uses a terminal to enter event or purchase information. This input data includes the type and date of the event, and a list of items the user wishes to purchase. The terminal then pre-processes this data and prepares it for transmission to the server.

[0722] Step 2:

[0723] The server retrieves user input data received from the terminal. At this time, it checks the validity of the data to ensure that there is no inaccurate information. The received data is then passed to the generation AI model, which analyzes the necessary information and prepares to generate explanatory and product information.

[0724] Step 3:

[0725] The generative AI model analyzes the received data. A pre-trained model is used for data analysis, generating optimal information based on the user's individual interests and behavior. The output includes explanatory and product information, which is managed on the server.

[0726] Step 4:

[0727] The server sends the generated explanatory or product information to the user's device. The user's device displays or outputs the received information as audio. This allows the user to receive the necessary information in real time.

[0728] Step 5:

[0729] The server uses data from sensors and cameras within the facility to predict congestion levels and traffic flow. Machine learning algorithms analyze this data in real time and prepare to generate prompt messages to avoid congestion.

[0730] Step 6:

[0731] Based on congestion levels and predicted information, the server generates prompt messages and suggests the optimal time for the user to use the service. The terminal receives these suggestions and notifies the user. The user can then use the service efficiently by following the suggestions.

[0732] Step 7:

[0733] The server analyzes traffic conditions or in-store movement data to determine the optimal return route or travel path. The generated results are sent to the user's terminal and presented as navigation.

[0734] Step 8:

[0735] Users follow the on-screen navigation to return home or travel safely and smoothly. This promotes efficient travel with minimal congestion and waiting times.

[0736] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0737] This invention is a system designed to enhance the experience of sports spectators, providing dynamic content based on the spectator's real-time emotions and interests. This system, which incorporates an emotion engine, is realized by integrating spectator terminals, servers, and emotion recognition technology.

[0738] First, the user selects a sporting event to watch via their device, and the device sends this information to the server. At this time, the device uses its camera and microphone to detect the user's emotions. The emotion engine analyzes the user's facial expressions and tone of voice to recognize emotions such as joy and surprise.

[0739] The server analyzes the information based on the sports event selected by the user and the detected sentiment data, and generates explanatory information according to the user's interest and level of understanding. For example, if the user expresses surprise, the server will provide a detailed explanation of the importance and background of that play.

[0740] Furthermore, the server predicts the status of events and the congestion levels of restaurants and bars, and the emotion engine considers the user's stress and level of concentration to suggest the optimal time to visit. For example, if a user is intently watching a game, it will suggest postponing their dining or drinking time to avoid congestion.

[0741] After the event, the server analyzes the user's emotional history through an emotion engine and provides the device with the optimal route home, taking traffic conditions into consideration. This feedback loop continuously improves the user's sports viewing experience.

[0742] In this way, this system changes and adjusts the service according to the emotions of the spectators, making it an enjoyable and easy-to-understand experience as part of the viewing experience.

[0743] The following describes the processing flow.

[0744] Step 1:

[0745] The user uses their device to input information about the sporting event they want to watch. When the device sends the entered event information to the server, it simultaneously collects the user's facial expressions and voice using its camera and microphone.

[0746] Step 2:

[0747] The device sends the user's emotional data collected to the server. The device's emotion engine performs initial analysis to recognize emotions from facial expressions and voice.

[0748] Step 3:

[0749] Based on the sports event information and sentiment data received by the server, AI is used to generate real-time commentary on the match. The server adjusts the commentary content, taking into account the user's level of interest and understanding.

[0750] Step 4:

[0751] The server generates explanatory information and sends it to the user's device, which then presents the information to the user via audio or text display. In particular, if the user's reaction is one of surprise, the server will provide a more detailed explanation of the background and specifics of the play.

[0752] Step 5:

[0753] The server predicts the status of events taking place in the stadium and the congestion levels of food and beverage facilities, and uses an emotion engine to assess the user's stress level. The server calculates the best time for the user to use the service comfortably and notifies the device.

[0754] Step 6:

[0755] After the viewing session ends, the server analyzes the user's emotional history and calculates the optimal route home. The terminal then provides this route information to the user, helping them to return home efficiently while avoiding congestion.

[0756] (Example 2)

[0757] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0758] Providing personalized commentary tailored to the emotions and interests of spectators in real time during sports events is difficult. Furthermore, optimal service utilization and return-home suggestions that take into account event crowds and traffic information are insufficient. This can lead to inconvenience for spectators and detract from the overall viewing experience.

[0759] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0760] In this invention, the server includes means for processing spectator behavior and audio signals and identifying emotions, means for predicting the status of the event and the congestion of food and beverage facilities, and means for analyzing traffic conditions and providing spectators with the optimal route home. This enables the provision of personalized commentary tailored to the emotions of spectators, as well as the suggestion of optimal services and routes during and after the event.

[0761] An "information processing device" is a device used for inputting, processing, and outputting data, and refers to terminal equipment used by spectators.

[0762] "Sports competition information" refers to a collection of data and information related to sports matches and competitions.

[0763] "Explanatory information" refers to the commentary and background information provided to spectators, including detailed information related to the competition.

[0764] "Competition information" refers to data and statistics related to a specific match or competition, including tactics, player data, and scores.

[0765] "Emotional analysis means" refers to techniques or methods for identifying emotional states from the facial expressions and voices of spectators.

[0766] "Presenting visually or aurally" refers to providing information through a device as a screen display or audio output.

[0767] "Event status" refers to information related to the ongoing status and schedule of an event or match.

[0768] "Crowding situation" refers to information about how crowded a place or facility is where spectators gather.

[0769] "Return route" refers to the paths or routes that spectators can choose to take when returning home after the event has ended.

[0770] A "generative AI model" refers to a model that uses artificial intelligence technology to generate useful information and predictions from large amounts of data.

[0771] This invention provides a system that enhances the viewing experience by offering content such as commentary that dynamically changes in response to the emotions of the spectators. The embodiments thereof are described in detail below.

[0772] Users select the sport they wish to watch using a mobile device or computer. This device receives input from the spectator and transmits the sport information to a server. The device also captures the spectator's facial expressions and voice through input devices such as cameras and microphones, and uses this information for emotion analysis. This analysis uses emotion analysis tools to identify the spectator's reactions (joy, surprise, etc.) in real time.

[0773] The server uses a generative AI model to generate the most relevant explanatory information for the spectator based on the received athletic competition information and spectator emotion data. This provides relevant background information and tactical commentary according to the emotions the spectator expresses in response to specific moments in the match. For example, if a spectator expresses surprise at a crucial scoring moment in the match, a detailed explanation of the background leading up to that goal and the strategic movements of the players will be notified to the terminal.

[0774] Furthermore, the server analyzes historical data and real-time information to predict the event's progress and the congestion of food and beverage facilities. This allows it to suggest optimal timings for eating and drinking, as well as event viewing plans, to spectators. After the event ends, it also considers current traffic conditions and provides spectators with ideal routes home. The terminal visually displays the route home, supporting spectators in moving smoothly.

[0775] An example of a prompt would be, "Explain how to provide a user who is surprised by a soccer goal with a detailed explanation of the player's background and similar past tactics." This prompt allows the generative AI model to provide viewers with appropriate information, resulting in a more enriching viewing experience.

[0776] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0777] Step 1:

[0778] The user selects the sports they wish to watch using their device. The user's selection information is retrieved as input, and the device sends this information to the server. This allows the server to prepare data on the sports that the spectator is interested in.

[0779] Step 2:

[0780] The device uses its built-in camera and microphone to collect the user's facial expressions and voice in real time. Video and audio data are acquired as input, and this data is used for analysis by emotion analysis tools. The emotion engine analyzes this data to identify the user's emotional state (joy, surprise, etc.) in real time. Identified emotion data is generated as output.

[0781] Step 3:

[0782] The server receives athletic competition information from the user and emotional data acquired from the device as input. Using a generative AI model, the server generates explanatory information tailored to the spectator based on the input information. This process dynamically generates commentary that responds to the match situation and the spectator's emotions. Customized explanatory information is generated as output.

[0783] Step 4:

[0784] The server sends the explanatory information generated above to the terminal. The terminal receives this and presents it to the user visually or audibly. The explanatory information, as input, is provided in a format that is easy for the user to understand and includes specific explanations to enhance the viewing experience. As output, the information is presented to the user in a format that is easy for them to understand.

[0785] Step 5:

[0786] The server analyzes the progress of events and the congestion levels of food and beverage facilities in real time. Using historical data and current schedule information as input, it predicts the optimal timing for service utilization. This process enables a function that suggests the best visit timing and break plans to the user. The output generates suggestions for the optimal service timing for the user.

[0787] Step 6:

[0788] After the event ends, the server collects and analyzes traffic information to calculate the optimal route home for the user. Traffic data is used as input, and the server performs calculations to determine the shortest or most efficient route for the user. As output, a specific route home is sent to the terminal, providing the user with visual instructions.

[0789] (Application Example 2)

[0790] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0791] Conventional sports viewing systems have limited personalized explanations and information based on the viewer's emotions and interests, making it difficult to extend and optimize the viewing experience within the home environment. Furthermore, there has been a lack of established means to control home automation devices in real time in response to changes in emotions during viewing, thereby providing a comfortable viewing environment. As a result, there has been a challenge in providing an environment where viewers can enjoy themselves to the fullest.

[0792] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0793] In this invention, the server includes means for receiving sports activity information input from a spectator's information terminal, means for analyzing the activity data and generating explanatory information to be provided to the spectator, means for transmitting the explanatory information to the spectator's information terminal and presenting it by display or audio, means for predicting the progress of the event and the congestion of food and beverage facilities and suggesting the optimal timing for using the services, means for analyzing traffic conditions and providing the optimal route home, and means for controlling a home automation device for adjusting the home environment based on the spectator's emotions. This makes it possible to optimize the home environment according to the spectator's emotions and provide a personalized viewing experience.

[0794] A "spectator information terminal" is a device equipped with the function of receiving sports activity information and displaying or audio-presenting the provided explanatory information.

[0795] "Sports activity information" refers to information about events that spectators are watching, including the status of the matches.

[0796] "Descriptive information" refers to data generated to provide spectators with details and related information about the match.

[0797] "Activity data" refers to various data related to sports activities, mainly including the progress of matches and statistical data.

[0798] "Event progress" refers to information showing the current status and timeline of ongoing sports activities.

[0799] "Food and beverage facility congestion status" refers to information regarding the usage status of food and beverage spaces at sports event venues.

[0800] "Optimal service usage timing" refers to the time when spectators can use events and facilities efficiently and comfortably.

[0801] "Traffic conditions" refers to information regarding traffic flow and congestion around the event venue and on the return route.

[0802] "Home automation devices" are devices used to control the environment within a home, and include lighting, air conditioning, robots, and other similar devices.

[0803] "Emotion-based adjustment" refers to analyzing the emotional state of spectators and automatically changing the lighting and temperature in the home based on that analysis.

[0804] This invention is a system that dynamically utilizes spectator emotional data to optimize the in-home sports viewing experience. The server receives sports activity information from the spectator's information terminal, analyzes the match activity data based on that information, and generates explanatory information to provide to the spectator. This explanatory information is transmitted to the spectator's information terminal and presented via display or audio.

[0805] Furthermore, the server predicts the progress of the event and the congestion levels of food and beverage facilities, and based on this data, suggests the optimal timing for spectators to use the services. It also analyzes traffic conditions and provides the best routes home for spectators, ensuring smooth travel.

[0806] In addition, by controlling home automation devices, the home environment can be dynamically adjusted based on the spectator's emotions. For example, if a spectator becomes excited, the brightness of smart lighting can be increased and the air conditioning adjusted to create a comfortable environment. Home robots can also provide beverages at the appropriate time. This creates an environment where spectators can enjoy the game with greater concentration.

[0807] In summary, this invention aims to provide a comfortable and efficient viewing environment through spectator emotional data, and its implementation utilizes open-source facial recognition libraries (e.g., OpenCV), home automation devices, information terminals, and other hardware and software.

[0808] By utilizing generative AI models, it is possible to improve the accuracy of analyzing and predicting emotional data. An example of a prompt would be: "Suggest ways to optimize the home environment based on the user's emotions while watching sports. For example, come up with ideas on how to adjust the lighting and air conditioning when the user is excited, and what a robot should provide."

[0809] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0810] Step 1:

[0811] The terminal sends sports activity information as input to the server. This allows information about matches that the user is interested in to be accumulated on the server, preparing the data that will serve as the basis for subsequent processing.

[0812] Step 2:

[0813] The server analyzes match activity data based on sports activity information received from terminals. This includes match progress and player statistics. The analyzed data is generated as explanatory information and customized to the user's interests.

[0814] Step 3:

[0815] The server sends the generated explanatory information to the user's device. The device receives this information and presents it on the screen or as audio. The user can view this information and deepen their understanding of the match.

[0816] Step 4:

[0817] The server uses machine learning algorithms to analyze collected data in order to predict the progress of events and the congestion levels of food and beverage facilities. The prediction results are used as a basis for deciding when to suggest the optimal time for users to use the services.

[0818] Step 5:

[0819] The server analyzes traffic conditions in real time and prepares data to guide users home via the most suitable route after watching a game. This information provides the optimal route, taking into account traffic flow and congestion.

[0820] Step 6:

[0821] The server analyzes user emotion data and generates instructions to control home automation devices. These instructions then adjust lighting, change air conditioning settings, and provide services via home robots, all in response to the user's emotions. The use of a generative AI model improves the accuracy of emotion analysis, further optimizing the user experience.

[0822] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0823] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0824] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0825] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0826] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0827] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0828] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0829] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0830] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0831] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0832] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0833] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0834] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0835] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0836] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0837] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0838] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0839] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0840] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0841] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0842] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[0843] The following is further disclosed regarding the embodiments described above.

[0844] (Claim 1)

[0845] A means of receiving sports event information entered from spectators' devices,

[0846] A means for analyzing match data in order to generate commentary information to be provided to spectators based on the aforementioned sports event information,

[0847] A means for transmitting the aforementioned explanatory information to the spectator's terminal and presenting it on display or by sound,

[0848] A means of predicting the status of events and the congestion levels of food and beverage facilities,

[0849] A means of suggesting the optimal timing for spectators to use the service based on the aforementioned prediction results,

[0850] A means of analyzing traffic conditions and providing spectators with the best route home,

[0851] A system that includes this.

[0852] (Claim 2)

[0853] The system according to claim 1, characterized in that the aforementioned explanatory information is customized based on the viewer's interest and level of understanding.

[0854] (Claim 3)

[0855] The system according to claim 1, characterized in that the prediction results are updated in real time using a machine learning algorithm.

[0856] "Example 1"

[0857] (Claim 1)

[0858] A means for receiving sports event information input from a spectator's information processing device,

[0859] A means for analyzing competition data in order to generate commentary information to be provided to spectators based on the aforementioned sports event information,

[0860] A means for transmitting the aforementioned explanatory information to the spectator's information processing device and presenting it by display or sound,

[0861] A means of predicting the progress of an event and the congestion of food and beverage facilities,

[0862] A means of proposing the optimal time for spectators to use the service based on the aforementioned prediction results,

[0863] A means of analyzing traffic conditions and providing spectators with the best route home,

[0864] A method for providing real-time commentary on important events during a match using generative AI models,

[0865] A means of securely transmitting information using a data communication protocol,

[0866] A system that includes this.

[0867] (Claim 2)

[0868] The system according to claim 1, characterized in that the explanatory information is personalized based on the viewer's interest and level of understanding.

[0869] (Claim 3)

[0870] The system according to claim 1, characterized in that the aforementioned prediction results and explanatory information are optimized in real time using machine learning techniques.

[0871] "Application Example 1"

[0872] (Claim 1)

[0873] A means for receiving event information or shopping information entered from a spectator's or customer's terminal,

[0874] A means for analyzing data to generate explanatory information or product information to be provided to spectators or customers based on the aforementioned information,

[0875] Means for transmitting the aforementioned explanatory information or product information to the terminal of a spectator or customer, and presenting it by display or sound,

[0876] A means of predicting the status of events at facilities and the congestion levels of services,

[0877] A means of proposing the optimal timing for using the service or purchasing the service to the spectator or customer based on the aforementioned prediction results,

[0878] A means of analyzing traffic conditions or travel conditions and providing the optimal return route or travel path for spectators or customers,

[0879] A system that includes this.

[0880] (Claim 2)

[0881] The system according to claim 1, characterized in that the explanatory information or product information is customized based on the interests, understanding, or purchase history of the spectator or customer.

[0882] (Claim 3)

[0883] The system according to claim 1, characterized in that the prediction results are updated in real time using a machine learning algorithm.

[0884] "Example 2 of combining an emotion engine"

[0885] (Claim 1)

[0886] A means for receiving athletic competition information input from a spectator's information processing device,

[0887] A means for analyzing the athletic competition information in order to generate explanatory information to be provided to spectators based on the aforementioned athletic competition information,

[0888] An emotion analysis means that processes the behavior and audio signals of spectators and identifies their emotions,

[0889] Means for transmitting the aforementioned explanatory information to the spectator's information processing device and presenting it visually or aurally,

[0890] A means of predicting the status of events and the congestion levels of food and beverage facilities,

[0891] A means of suggesting the optimal timing for spectators to use the service based on the aforementioned prediction results and the emotional state of the spectators,

[0892] A means of analyzing traffic conditions and providing spectators with the best route home,

[0893] A system that includes this.

[0894] (Claim 2)

[0895] The system according to claim 1, characterized in that the aforementioned explanatory information is adjusted based on the emotions and level of understanding of the spectators.

[0896] (Claim 3)

[0897] The system according to claim 1, characterized in that the prediction results are updated in real time using a generative AI model.

[0898] "Application example 2 when combining with an emotional engine"

[0899] (Claim 1)

[0900] A means of receiving sports activity information entered from spectators' information terminals,

[0901] A means for analyzing activity data in order to generate explanatory information to be provided to spectators based on the aforementioned sports activity information,

[0902] A means for transmitting the aforementioned explanatory information to the spectator's information terminal and presenting it by display or audio,

[0903] A means of predicting the progress of an event and the congestion of food and beverage facilities,

[0904] A means of suggesting the optimal timing for spectators to use the service based on the aforementioned prediction results,

[0905] A means of analyzing traffic conditions and providing spectators with the best route home,

[0906] A means for controlling a home automation device to adjust the home environment based on the emotions of the spectators,

[0907] A system that includes this.

[0908] (Claim 2)

[0909] The system according to claim 1, characterized in that the explanatory information is personalized based on the viewer's interests and level of understanding, and the home environment is further optimized according to the viewer's emotional data.

[0910] (Claim 3)

[0911] The system according to claim 1, characterized in that the prediction results and the control of the home automation device are updated in real time using machine learning methods. [Explanation of Symbols]

[0912] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for receiving event information or shopping information entered from a spectator's or customer's terminal, A means for analyzing data to generate explanatory information or product information to be provided to spectators or customers based on the aforementioned information, Means for transmitting the aforementioned explanatory information or product information to the terminal of a spectator or customer, and presenting it by display or sound, A means of predicting the status of events at facilities and the congestion levels of services, A means of proposing the optimal timing for using the service or purchasing the service to the spectator or customer based on the aforementioned prediction results, A means of analyzing traffic conditions or travel conditions and providing the optimal return route or travel path for spectators or customers, A system that includes this.

2. The system according to claim 1, characterized in that the explanatory information or product information is customized based on the interests, understanding, or purchase history of the spectator or customer.

3. The system according to claim 1, characterized in that the prediction results are updated in real time using a machine learning algorithm.

Images