System

Abstract

A system is provided.SOLUTION: The system includes a bird's-eye camera, a tracking camera, means for processing video data received from the bird's-eye camera and the tracking camera in real time, means for extracting coordinate data of a person and a ball from the video data, means for detecting a movement pattern, a speed, and an event of the person and the ball based on the coordinate data, means for generating statistical data based on the coordinate data and the event detection result, and means for distributing the statistical data to terminals in real time.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Data analysis has become increasingly important in modern sports competitions. However, current methods mainly involve staff manually collecting and analyzing data, which places a heavy burden on the team and is prone to human error. Furthermore, it is difficult to provide real-time data during a match, preventing prompt information provision to spectators and sufficient tactical analysis. To address these issues, there is a need for the development of a system that can automatically collect and analyze data efficiently and accurately. [Means for solving the problem]

[0005] To solve the above problems, the present invention provides the following solutions. First, an overhead camera and a tracking camera are used to capture the entire game and the movements of specific players and the ball. Then, the video data obtained from these cameras is processed in real time to extract coordinate data for people and the ball. Furthermore, a means is provided for detecting player movement patterns and speeds and game events based on this coordinate data. Furthermore, statistical data is generated based on this data and distributed to devices in real time, enabling efficient data analysis and rapid information provision. In addition, motion analysis and tactical analysis functions using skeletal data provide detailed performance analysis and strategic insights. Finally, real-time notifications are sent to spectators based on the event detection results, enhancing the sense of presence of the game and improving the spectator experience.

[0006] A "bird's-eye view camera" is a camera device that captures a wide range of images of the entire game or event from above.

[0007] A "tracking camera" is a camera device that automatically tracks a specific object or person and captures detailed, high-resolution footage.

[0008] "Video data" refers to digital video information acquired from a camera device.

[0009] "Means of processing in real time" refers to technology and equipment that instantly analyzes and processes video data obtained from cameras without delay.

[0010] "Coordinate data" refers to data that expresses the positional information of people and balls extracted from video data in two-dimensional or three-dimensional coordinate format.

[0011] "Movement patterns" refers to data that analyzes and visualizes the movement paths of people and balls over a certain period of time.

[0012] "Speed" is a physical quantity that represents the distance traveled per unit time, and is data used to quantify the movement of a person or a ball.

[0013] An "event" is a specific action or result that occurs during a match, such as a pass, a shot, or a goal.

[0014] "Statistical Data" is a collection of numerical and graphical statistical information generated from the analyzed coordinate data and event information.

[0015] "Means for real-time distribution" refers to the technology and devices that instantly transmit generated data to another terminal or user.

[0016] "Skeletal data" is data that expresses the joint positions of a person in coordinate format and is used for motion analysis.

[0017] "Motion analysis" is a technology that identifies and analyzes specific human movements and movement patterns based on skeletal data.

[0018] "Tactical analysis" is a method of analyzing data for the purpose of evaluating and improving the effectiveness of strategies and tactics based on the analyzed data.

[0019] "Real-time notifications for spectators" refers to the technology and means to provide spectators with immediate information when important events occur during a match. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] The system of the present invention uses a bird's-eye view camera and a tracking camera to collect detailed data on the entire game, players, and the ball in real time, and automatically analyzes and compiles it. Specific embodiments of each processing step are described below.

[0042] This system begins with the placement of overhead cameras and tracking cameras. The overhead cameras are installed in a position that allows for a bird's-eye view of the entire stadium. Meanwhile, the tracking cameras are high-resolution cameras that are placed at key points in the game to track specific players or the ball in detail. These cameras are connected to a server and transmit video data in real time.

[0043] Server Processing

[0044] 1. Video data reception and preprocessing

[0045] The server receives video data from the overhead camera and tracking camera. After preprocessing such as noise reduction and resolution adjustment, the video data is divided into frames.

[0046] 2. Object detection and coordinate data extraction

[0047] The server uses an object detection algorithm (e.g., YOLO or DeepSORT) to detect players and the ball in each frame and extract their coordinate data, which represent the positional information of players and the ball in two or three dimensions.

[0048] 3. Calculating movement patterns and speeds

[0049] The server calculates the movement patterns and speeds of players and the ball from the coordinate data of successive frames, which allows visualization of each player's running distance and speed.

[0050] 4. Event Detection

[0051] The server automatically detects match events such as passes, shots, and goals based on the calculated coordinate and velocity data, and this event data is tagged and used for subsequent analysis.

[0052] 5. Skeletal data extraction and motion analysis

[0053] The server uses a skeletal detection algorithm such as OpenPose or MediaPipe to extract the skeletal data (joint positions) of each player, and analyzes the player's movements (e.g., jump shots and sliding).

[0054] 6. Generation and distribution of statistical data

[0055] The server combines the coordinate data and event data to generate statistical data for each player (e.g., pass success rate, number of successful shots, number of turnovers). The generated statistical data is delivered to the device in real time.

[0056] Terminal handling

[0057] 1. Data Receipt and Analysis

[0058] The terminal receives the real-time data sent from the server, analyzes the received data, and extracts the information required for display.

[0059] 2. Visualization and Notifications

[0060] The device visually displays an overview of the entire game and performance data for each player, and also notifies the user in real time when important events occur.

[0061] User operations

[0062] 1. Viewing real-time data

[0063] Users can check the progress of the match in real time through their devices, and can also view player performance data and match highlights.

[0064] 2. Detailed analysis and visualization

[0065] Users can select detailed analytical data for a specific period or player and view the data graphically visualized, providing insights for tactical analysis and performance improvement.

[0066] Specific examples

[0067] For example, in a basketball game, the server captures camera footage from the start of the game, and the AI ​​model updates the position data of each player and the ball every 0.1 seconds. If the server detects that player A passes the ball and player B receives it, it tags the action as a "pass" and increments the pass success count in real time. The device receives this information and visually displays detailed information about the pass to the user.

[0068] The above is the details of the mode for carrying out the invention. This system can be applied not only to basketball but also to soccer and other sports, so it can meet a variety of needs for sports data analysis.

[0069] The processing flow will be explained below.

[0070] Server Processing

[0071] Step 1:

[0072] The server receives video data from the overhead camera and tracking camera in real time, and the received video data is divided into frames.

[0073] Step 2:

[0074] The server performs pre-processing of the video data, including image pre-processing such as noise reduction, resolution adjustment, and color correction.

[0075] Step 3:

[0076] The server uses an object detection algorithm (e.g., YOLO or DeepSORT) to detect players and the ball for each frame, and extracts the position information of the detected players and the ball as coordinate data.

[0077] Step 4:

[0078] The server calculates the movement patterns of the players and the ball based on the coordinate data of successive frames, including the distance and speed of the players.

[0079] Step 5:

[0080] The server uses the calculated coordinate and velocity data to detect events during the match (e.g., passes, shots, goals), and tags the events.

[0081] Step 6:

[0082] The server uses a skeleton detection algorithm such as OpenPose or MediaPipe to extract the player's skeletal data, which includes the positions of each joint.

[0083] Step 7:

[0084] The server analyzes the player's movements based on the extracted skeletal data, identifying movements such as jump shots and sliding.

[0085] Step 8:

[0086] The server integrates the coordinate data and the event detection results to generate statistical data for each player (e.g., pass success rate, number of successful shots, number of turnovers).

[0087] Step 9:

[0088] The server delivers the generated statistical data to the device in real time, using standard formats such as JSON.

[0089] Terminal handling

[0090] Step 1:

[0091] The terminal receives real-time data sent from the server, and the received data is analyzed immediately.

[0092] Step 2:

[0093] Based on the analyzed data, the device displays a bird's-eye view of the entire match, along with visuals of each player's performance data.

[0094] Step 3:

[0095] The device will notify the user in real time when important match events occur, including pop-up messages and audio alerts.

[0096] Step 4:

[0097] When users click on a notification or select a specific player or time period, they are presented with an interface that displays detailed play data and highlights.

[0098] User operations

[0099] Step 1:

[0100] Users can check the progress of the match in real time through their devices, with overhead camera footage and analysis data displayed simultaneously.

[0101] Step 2:

[0102] Users can select a specific player's performance data and view detailed statistics, such as shooting percentage and distance traveled.

[0103] Step 3:

[0104] When a user selects an important event during a match (e.g., a goal or a foul), a replay of the corresponding play is displayed, allowing spectators to rewatch the important moments of the match.

[0105] Example 1

[0106] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0107] Existing sports data analysis systems have the problem of being unable to grasp the situation of a game in real time, and it takes a long time to analyze the detailed movements of players and the ball. Furthermore, because real-time notification functions for spectators are not fully developed, important moments of the game can be missed. Furthermore, there are issues with the accuracy and reliability of statistical data due to the inefficient fusion and analysis of data obtained from multiple cameras.

[0108] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0109] In this invention, the server includes a means for processing video data received from the overhead camera and tracking camera in real time, a means for extracting coordinate data of people and the ball from the video data using an object detection algorithm, and a means for detecting the movement patterns, speed, and events of people and the ball based on the coordinate data. This allows highly accurate coordinate data to be extracted from the video data in real time, enabling detailed analysis of the movements of players and the ball. Furthermore, by generating and distributing statistical data in real time, users can instantly grasp the situation of the game and watch without missing important moments. Furthermore, data obtained from multiple cameras is efficiently combined to provide highly reliable statistical data.

[0110] A "bird's-eye view camera" is a camera that takes pictures of the entire stadium from above.

[0111] A "tracking camera" is a camera that tracks a specific player or the ball in detail and takes high-resolution images.

[0112] "Real-time processing of video data" means instantly processing video data received from the overhead camera and tracking camera.

[0113] An "object detection algorithm" is an algorithm for identifying specific objects (such as players or the ball) from video data and extracting their location information.

[0114] "Coordinate data" is data that indicates the position of an object in an image, and is usually expressed as two-dimensional or three-dimensional coordinates.

[0115] A "movement pattern" indicates the movement path of an object calculated from continuous coordinate data.

[0116] "Velocity" indicates the amount of change in position per unit time when an object moves.

[0117] An "event" is a specific action or occurrence in a match (e.g., a pass, a shot, a goal) that is tagged when it is detected.

[0118] "Statistical Data" means statistical information about player or team performance that is generated based on extracted and analyzed data.

[0119] "Real-time delivery" refers to the process of instantly transmitting generated data to a user terminal.

[0120] "Skeletal data" is data that indicates the joint positions and structure of a person extracted from video data.

[0121] "Movement analysis" refers to the analysis of a person's specific movements using extracted skeletal data and coordinate data.

[0122] "Tactical analysis" is an analysis that evaluates tactics and playing style based on the results of motion analysis and identifies areas for improvement.

[0123] "Real-time notification" is a function that notifies the user of a specific event immediately when that event is detected.

[0124] A "user interface" is a display screen on a terminal that allows users to visually check the game situation and statistical data.

[0125] The system of the present invention analyzes sports data in real time to detect and analyze the detailed movements of players and the ball. This system uses an overhead camera and a tracking camera to collect detailed data on the entire game, players, and the ball, and automatically analyzes the data and provides it to the user. Specific embodiments of each element are described below.

[0126] Server configuration and processing

[0127] Video data reception and preprocessing

[0128] The server receives video data in real time from the overhead camera and tracking camera. The overhead camera is a high-resolution camera that captures the entire match venue from above, while the tracking camera is a camera that tracks players and the ball in detail. These cameras are connected to the server and transmit video streams in H.264 format. The server performs noise reduction (using Gaussian Blur, for example) and resolution adjustment on the received video data, and then divides it into the required frames.

[0129] Object detection and coordinate data extraction

[0130] The server uses an object detection algorithm (such as YOLO or DeepSORT) to detect players and the ball in the frame and extract their location information, which is then saved as 2D or 3D coordinate data.

[0131] Movement pattern and speed calculations

[0132] The server uses the coordinate data from successive frames to calculate the movement patterns and speeds of players and the ball, which then calculates the distance and speed of each player and stores them in a database.

[0133] Event detection and tagging

[0134] The server automatically detects events such as passes, shots, and goals under certain conditions based on coordinate and velocity data, and tags these events. The event data is stored in a database for subsequent analysis and report generation.

[0135] Skeletal data extraction and motion analysis

[0136] The server uses skeletal detection algorithms such as OpenPose and MediaPipe to extract the player's skeletal data (joint positions) and analyzes their movements (e.g., jump shots and sliding) based on this data.

[0137] Statistical data generation and distribution

[0138] The server integrates the coordinate data and event data to generate statistical data for each player (e.g., pass success rate, number of successful shots, number of turnovers). This statistical data is delivered to the device in real time using communication protocols such as WebSocket and REST API.

[0139] Terminal configuration and handling

[0140] Data reception and analysis

[0141] The device receives real-time data sent from the server, analyzes it, and extracts the necessary information, such as video frames and statistical data, which it stores in memory and reconstructs accordingly.

[0142] Visualization and Notifications

[0143] The device displays a bird's-eye view of the entire game and performance data for each player on the user interface, and notifies the user in real time via pop-up notifications when important events occur (e.g., successful passes or goals).

[0144] User operations

[0145] Viewing real-time data

[0146] Users can check the progress of the match in real time through their device, with match footage and player location information updated continuously, and players' performance data and match highlights visually displayed.

[0147] Detailed Analysis and Visualization

[0148] Users navigate through specific menus on their device to select detailed analytical data for specific periods or players, which are then visualized in graphs and charts to provide tactical analysis and insights for performance improvement.

[0149] Specific examples

[0150] For example, in a basketball game, the server captures video from the overhead camera and tracking camera from the start of the game and updates the position data of each player and the ball every 0.1 seconds using an object detection algorithm. If the server detects that player A passes the ball and player B receives it, it tags the action as a "pass" and increments the pass success count in real time. The device receives this information and visually displays the pass details to the user. For example, a notification pops up saying, "Pass from player A to player B was successful."

[0151] Example prompt for a generative AI model:

[0152] "In a basketball game, collect data on when player A successfully passes the ball to player B, and analyze it in real time."

[0153] The above is the detailed description of the mode for carrying out the invention. This system can be applied not only to basketball but also to soccer and other sports, and can meet a variety of needs for sports data analysis.

[0154] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0155] Server Processing

[0156] Step 1: Receiving and preprocessing video data

[0157] The server receives H.264 video data in real time from the overhead camera and tracking camera. The input is the video stream from the camera, and the output is preprocessed frame data. Specifically, Gaussian Blur is applied to remove noise, and the resolution is adjusted and divided into the required frames.

[0158] Step 2: Object detection and coordinate data extraction

[0159] The server applies an object detection algorithm (such as YOLO or DeepSORT) to the preprocessed frame data as input. The input is the preprocessed frame data, and the output is the coordinate data of the detected players and ball. For each frame, it detects specific objects from the pixel data and extracts their location information as 2D or 3D coordinates.

[0160] Step 3: Calculate movement patterns and speeds

[0161] The server inputs the coordinate data of successive frames and calculates the movement patterns and speeds. The input is the coordinate data for each frame, and the output is the movement patterns and speed data of the players and the ball. Specifically, the server calculates the speeds using the time intervals of the coordinate data and stores the results in a database.

[0162] Step 4: Event detection and tagging

[0163] The server automatically detects specific events (passes, shots, goals, etc.) in a match based on movement patterns and speed data. The input is coordinate data and speed data, and the output is the detected events and their tag data. As each event is detected, it is tagged and stored in a database.

[0164] Step 5: Extracting skeletal data and motion analysis

[0165] The server uses preprocessed frame data as input and performs skeletal detection using OpenPose or MediaPipe. The input is the preprocessed frame data, and the output is the player's skeletal data (joint positions). The extracted skeletal data is analyzed to recognize specific movements (e.g., jump shots and sliding).

[0166] Step 6: Generate and distribute statistical data

[0167] The server combines coordinate data, event data, and motion analysis data to generate statistical data for each player. The input is all of the above data, and the output is statistical data. This statistical data is delivered to the device in real time. WebSocket and REST API are used for delivery.

[0168] Terminal handling

[0169] Step 1: Receiving and analyzing data

[0170] The terminal receives real-time data sent from the server. The input is the real-time data from the server, and the output is the analyzed data required for display. The terminal analyzes the received data, stores e.g., video frames and statistical data in memory, and reconstructs them accordingly.

[0171] Step 2: Visualization and Notification

[0172] The device displays the received data in a user interface. The input is the analyzed data, and the output is a screen that displays an overview of the game and the performance of each player. When an important event occurs, the user is notified in real time by a pop-up notification.

[0173] User operations

[0174] Step 1: View real-time data

[0175] The user can check the progress of the game in real time through the device. The input is the display data from the device, and the output is the user's visual information. The progress of the game and the position information of the players are updated successively.

[0176] Step 2: Detailed analysis and visualization

[0177] Users operate specific menus on the device and select detailed analysis data. The input is the device's menu operations, and the output is the selected analysis data. The selected data is visualized in graphs and charts, providing insights for tactical analysis and performance improvement.

[0178] (Application example 1)

[0179] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0180] In conventional manufacturing sites, it has been difficult to monitor the operation of manufacturing machines in real time and immediately detect and notify abnormal operations. Furthermore, while precise operation analysis technology is required for efficient operation and quality control of manufacturing machines, there has been no system to achieve this. Furthermore, there has been a lack of means for workers to receive analysis results in real time on-site and respond quickly.

[0181] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0182] In this invention, the server includes an overhead camera, a tracking camera, means for processing video data received from the overhead camera and the tracking camera in real time, means for extracting coordinate data of people and objects from the video data, means for detecting movement patterns, speeds, and events of people and objects based on the coordinate data, means for generating statistical data based on the coordinate data and event detection results, means for detecting operation patterns of manufacturing machines and notifying abnormal operations in real time, and means for delivering the statistical data and abnormality detection results to terminals in real time.This enables real-time monitoring of the operation of manufacturing machines at a manufacturing site, immediate detection and notification of abnormal operations, and prompt response by workers on site.

[0183] A "bird's-eye view camera" is a camera that can capture a wide area from above.

[0184] A "tracking camera" is a high-resolution camera that can capture detailed images of a specific object while tracking it.

[0185] "Means for processing video data in real time" refers to technology that instantly processes video data collected from overhead cameras and tracking cameras.

[0186] "Means for extracting coordinate data" refers to technology that extracts the location information of specific people or objects from video data as numerical data.

[0187] "Means for detecting movement patterns, speed, and events" refers to technology that identifies the movement patterns and speed of objects, as well as specific events, based on the extracted coordinate data.

[0188] The "means for generating statistical data" refers to a technique for creating various statistical information using coordinate data and event detection results.

[0189] "Means for notifying abnormal operation in real time" refers to technology that monitors the operation of manufacturing machines and immediately notifies if an abnormality is detected.

[0190] "Means for delivering to terminals in real time" refers to technology that instantly transmits generated statistical data and anomaly detection results to the appropriate terminal.

[0191] The "means for detecting the operation pattern of a manufacturing machine" refers to a technology that monitors the operation of a manufacturing machine and identifies the characteristics and patterns of its movement.

[0192] "Means for conducting tactical analysis based on the results of motion analysis" refers to a technique for conducting detailed analysis from a tactical perspective based on extracted motion data.

[0193] "Means for providing real-time notifications to workers based on data on abnormal operations detected by the abnormality detection means" refers to technology that immediately notifies workers of details of an abnormality when an abnormality is detected.

[0194] This invention is a system that uses a bird's-eye view camera and a tracking camera to monitor and analyze the operation of manufacturing machines in a manufacturing site. The system is composed of a server, a terminal, and a user.

[0195] The server processes video data collected from the overhead camera and tracking camera in real time. The hardware used includes a high-resolution overhead camera, a tracking camera, and a server. The software used includes OpenCV for Python, FFmpeg, YOLO, DeepSORT, and TENSORFLOW (registered trademark). The server removes noise from the received video data, adjusts the resolution, and divides it into frames. It then uses YOLO and DeepSORT to extract coordinate data for the manufacturing machine and work objects from the video data. The movement pattern and speed of the manufacturing machine are calculated from the coordinate data of consecutive frames.

[0196] The server detects abnormal behavior and patterns and generates anomaly detection results and statistical data, including the operational efficiency, operation history, and frequency of failures of the manufacturing machines.

[0197] The server also delivers anomaly detection results and statistical data in real time to the terminal. The terminal is a pair of smart glasses worn by the worker. The terminal uses Three.js and WebGL to visualize and display the received data in 3D space. If a specific anomaly is detected, the terminal notifies the worker in real time.

[0198] The user, a worker, can visually check the operation data of the manufacturing machine in real time through the smart glasses, which allows them to respond immediately if an abnormality occurs.

[0199] As a concrete example, consider a scenario in which the operation of a robot arm in a production line is monitored. An overhead camera monitors the entire production line, while a tracking camera tracks the movement of the robot arm. The server analyzes this video data and determines whether the robot arm is operating normally or if any abnormalities have occurred. If an abnormality is detected, a worker wearing smart glasses is immediately notified, enabling a prompt response.

[0200] An example prompt might look like this:

[0201] Monitor the movement patterns followed by specific manufacturing robots in your factory in real time, get instant notification if any abnormal behavior is detected, view the coordinate data and movement speed of each robot in 2D and 3D, and store a history of any abnormal behavior.

[0202] This system will significantly improve quality control and efficiency at the manufacturing site, and enable rapid response when abnormalities occur.

[0203] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0204] Step 1:

[0205] The server receives video data from the overhead camera and the tracking camera in real time.

[0206] Specific operation: The server receives the video signal from the camera and divides it into frames. The input is the video data captured by the camera, and the output is the data of each frame with noise removal and resolution adjustment. Preprocessing is performed using OpenCV for Python.

[0207] Step 2:

[0208] The server extracts coordinate data of the manufacturing machine and the work object from the pre-processed frame data.

[0209] Specific operation: Using YOLO and DeepSORT, we detect manufacturing machines and objects in the frame and obtain their coordinate data. The input is each preprocessed frame, and the output is the 2D coordinate data of the detected manufacturing machines and objects.

[0210] Step 3:

[0211] The server calculates the movement patterns and speeds of the manufacturing machines and objects based on the coordinate data between successive frames.

[0212] Specific operation: The server uses TensorFlow to analyze the acquired coordinate data and calculate the movement distance and speed. The input is the detected coordinate data, and the output is the calculated movement pattern and speed data.

[0213] Step 4:

[0214] The server detects abnormal behavior based on movement patterns and speed data.

[0215] Specific operation: The server compares the accumulated movement pattern and speed data to determine whether there are any abnormalities. In this step, an algorithm is used to instantly detect abnormal behavior. Its input is the movement pattern and speed data, and its output is the data of the detected abnormal behavior.

[0216] Step 5:

[0217] The server notifies the worker in real time if any abnormal operation is detected.

[0218] Specific operation: When an abnormality is detected, the server immediately sends an abnormal operation notification to the worker's smart glasses using Socket.IO. The input is the detected abnormal operation data, and the output is a real-time notification.

[0219] Step 6:

[0220] The terminal visualizes abnormal behavior data and statistical data received from the server.

[0221] How it works: The smart glasses use Three.js and WebGL to visually display the received data in 3D space. Its input is abnormal behavior data and statistical data from the server, and its output is a visualized interface.

[0222] Step 7:

[0223] Using smart glasses, users can check the operation of manufacturing machines and receive abnormality notifications in real time and respond accordingly.

[0224] Specific Action: The user checks the display of the smart glasses and takes action if necessary. In this step, the input received by the user is the display data of the smart glasses, and the output is the corresponding action.

[0225] This series of processes makes it possible to monitor the operation of manufacturing machines at the manufacturing site in real time and to immediately notify workers if an abnormality occurs.

[0226] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0227] The system of the present invention uses a bird's-eye view camera and a tracking camera to collect detailed data on the entire game, players, and the ball in real time, and automatically analyzes and compiles it. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it aims to improve the user experience. Below, a specific embodiment of each processing step is explained.

[0228] This system begins with the placement of overhead cameras and tracking cameras. The overhead cameras are installed in a position that allows for a bird's-eye view of the entire stadium. Meanwhile, the tracking cameras are high-resolution cameras that are placed at key points in the game to track specific players or the ball in detail. These cameras are connected to a server and transmit video data in real time.

[0229] Server Processing

[0230] 1. Video data reception and preprocessing

[0231] The server receives video data from the overhead camera and tracking camera in real time, and the received video data is divided into frames.

[0232] 2. Object detection and coordinate data extraction

[0233] The server uses an object detection algorithm (e.g., YOLO or DeepSORT) to detect players and the ball in each frame and extract their coordinate data, which represents the positional information of players and the ball in two or three dimensions.

[0234] 3. Calculating movement patterns and speeds

[0235] The server calculates the movement patterns and speeds of players and the ball from the coordinate data of successive frames, which allows visualization of each player's running distance and speed.

[0236] 4. Event Detection

[0237] The server uses the calculated coordinate and velocity data to detect events during the match (e.g., passes, shots, goals), and tags the events.

[0238] 5. Skeletal data extraction and motion analysis

[0239] The server uses a skeletal detection algorithm such as OpenPose or MediaPipe to extract the player's skeletal data (joint positions), which is then used to analyze the player's movements (e.g., jump shots and sliding).

[0240] 6. Generating and distributing statistical data

[0241] The server combines the coordinate data and event detection results to generate statistical data for each player (e.g., pass success rate, number of successful shots, number of turnovers). The generated statistical data is delivered to the device in real time.

[0242] 7. Emotion Recognition with Emotion Engine

[0243] The server analyzes the user's real-time video and audio data using an emotion engine to recognize the user's emotional state (e.g., excitement, stress, interest).

[0244] 8. Integration and Utilization of Emotional Data

[0245] The server integrates the emotion data generated by the emotion engine with statistical data and event data, generates match highlights and other interesting content based on the user's interests and reactions, and delivers them to the device.

[0246] Terminal handling

[0247] 1. Data Receipt and Analysis

[0248] The device receives statistical data and emotion data sent from the server in real time, and the received data is analyzed immediately.

[0249] 2. Visualization and Notifications

[0250] The device visually displays an overview of the entire game and performance data for each player. It also selectively displays interesting highlights and specific plays based on the user's emotions detected by an emotion engine. Furthermore, it notifies the user in real time when important game events occur.

[0251] User operations

[0252] 1. Viewing real-time data

[0253] Users can check the progress of the match in real time through their devices, with overhead camera footage and analysis data displayed simultaneously.

[0254] 2. Content display based on detailed analysis and emotion recognition

[0255] Users can select and view the performance data of specific players or match highlights, and the emotion engine displays content that reflects the user's emotional data, improving the user experience.

[0256] Specific examples

[0257] For example, in a soccer match, the server captures camera footage from the start of the match, and the AL model updates the position data of each player and the ball every 0.1 seconds. If the server detects that player A passes the ball and player B receives it, it tags the action as a "pass" and increments the number of successful passes in real time. If the emotion engine recognizes the user's excitement, the device will visually display highlights of the play according to the user's excitement level.

[0258] The system will enable spectators to feel more at home in the match, gain detailed insights based on player performance analysis, and display content based on emotional data, providing a more personalized viewing experience.

[0259] As described above, the system of the present invention aims to improve the user experience by integrating multi-camera video analysis and emotion recognition technology to realize automatic analysis of match data and real-time notification.

[0260] The processing flow will be explained below.

[0261] Server Processing

[0262] Step 1:

[0263] The server receives video data from the overhead camera and tracking camera in real time and divides the received video data into frames.

[0264] Step 2:

[0265] The server performs pre-processing of the video data, specifically image pre-processing such as noise removal, resolution adjustment, and color correction.

[0266] Step 3:

[0267] The server uses an object detection algorithm (e.g., YOLO or DeepSORT) to detect players and the ball in each frame, and extracts the position information of the detected players and the ball as coordinate data.

[0268] Step 4:

[0269] The server calculates the movement patterns of the players and the ball based on the coordinate data of successive frames, including the distance and speed of the players.

[0270] Step 5:

[0271] The server automatically detects events (e.g., passes, shots, goals) during a match based on the calculated coordinate and velocity data, and tags the detected events.

[0272] Step 6:

[0273] The server uses a skeletal detection algorithm such as OpenPose or MediaPipe to extract skeletal data for each player, including the position information of each joint.

[0274] Step 7:

[0275] The server analyzes the player's movements based on the extracted skeletal data, identifying movements such as jump shots and sliding.

[0276] Step 8:

[0277] The server integrates the coordinate data and the event detection results to generate statistical data for each player (e.g., pass success rate, number of successful shots, number of turnovers).

[0278] Step 9:

[0279] The server delivers the generated statistical data to the device in real time, using standard formats such as JSON.

[0280] Step 10:

[0281] The server analyzes real-time video and audio data acquired from the user using an emotion engine to recognize the user's emotional state (e.g., excitement, stress, interest).

[0282] Step 11:

[0283] The server integrates the emotion data obtained by the emotion engine with statistical and event data, and generates game highlights and other engaging content based on the user's interests and reactions.

[0284] Step 12:

[0285] The server delivers content based on the generated emotion data to the device in real time, which is displayed as a visual feed that reflects the user's emotional state.

[0286] Terminal handling

[0287] Step 1:

[0288] The device receives statistical data and emotional data sent from the server in real time, and the received data is analyzed immediately.

[0289] Step 2:

[0290] Based on the analyzed data, the device visually displays an overview of the entire match and performance data for each player, including the player's movement patterns and speed.

[0291] Step 3:

[0292] The terminal selectively displays interesting highlights or specific plays based on the user's emotional state detected by the emotion engine, thereby providing content that attracts the user's interest.

[0293] Step 4:

[0294] The device will notify the user in real time when important match events occur, including pop-up messages and audio alerts.

[0295] User operations

[0296] Step 1:

[0297] Users can check the progress of the match in real time through their devices, with overhead camera footage and analysis data displayed simultaneously.

[0298] Step 2:

[0299] Users can select a specific player's performance data and view detailed statistics, such as shooting percentage and distance traveled.

[0300] Step 3:

[0301] The device will display highlights and detailed information related to the user's favorite players and plays of interest based on the user's emotional data recognized by the emotion engine, providing a more personalized viewing experience.

[0302] Example 2

[0303] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0304] Conventional match analysis systems can acquire player and ball position data and detect events based on that data. However, these systems do not take into account user emotional data, which means they cannot provide content optimized for each spectator. Furthermore, there is a lack of technology to perform detailed tactical analysis in real time, making it difficult to enhance the sense of realism in live match viewing.

[0305] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0306] In this invention, the server includes an overhead camera, a tracking camera, a means for processing video data received from the overhead camera and the tracking camera in real time, a means for extracting object coordinate data from the video data, a means for calculating the object's movement pattern and speed based on the coordinate data, a means for detecting events from the coordinate data and speed data, a means for generating statistical data based on the coordinate data and the event detection results, a means for distributing the statistical data to a terminal in real time, a means for recognizing a user's emotion, and a means for integrating the emotion data with the statistical data and event data to generate individually customized content. This enables the provision of personalized content that reflects the user's emotion, achieving a more realistic viewing experience. Furthermore, detailed tactical analysis can be performed in real time, enabling the provision of more accurate data.

[0307] A "bird's-eye view camera" is a camera that is installed in a position that allows it to overlook the entire venue from above and captures video data over a wide area.

[0308] A "tracking camera" is a high-resolution camera installed to closely track a specific player or the ball.

[0309] The "means for processing video data in real time" is a function that includes a process for dividing video data received from the overhead camera and tracking camera and extracting objects and coordinates.

[0310] "Means for extracting coordinate data" refers to an algorithm or system for obtaining position information of objects (such as players or the ball) from video data.

[0311] The "means for calculating the movement pattern and speed" is a function for calculating the movement path and speed of an object based on the coordinate data of successive frames.

[0312] The "event detection means" is a system that identifies important actions during a match (e.g., passes, shots, goals) based on movement patterns and speed data.

[0313] The "means for generating statistical data" is a function that integrates coordinate data and event detection results to calculate detailed statistical information for each player (e.g., pass success rate, number of successful shots).

[0314] The "means for delivering to terminals in real time" refers to a process and system for instantly transmitting the generated statistical data to user terminals.

[0315] The "means for recognizing user emotions" is a technology that analyzes the user's facial expressions and vocal tone to identify emotional states such as excitement, stress, interest, etc.

[0316] "Means for integrating emotional data with statistical data and event data to generate individually customized content" refers to a system that combines recognized user emotional data with other match data to create content (e.g., highlights) that is tailored to each individual user.

[0317] The system of this invention uses a bird's-eye view camera and a tracking camera to collect detailed data on the entire game, players, and the ball in real time, and automatically analyzes and compiles it.Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it aims to improve the user experience.

[0318] The system begins with the placement of overhead cameras and tracking cameras. The overhead cameras are installed in a position that allows for a bird's-eye view of the entire stadium, while the tracking cameras are high-resolution cameras that are placed at key points in the game to track specific players or the ball in detail. These cameras are connected to a server and transmit video data in real time.

[0319] Server Processing

[0320] The server first receives video data in real time from the overhead camera and tracking camera. The received video data is divided into frames, and an object detection algorithm (e.g., YOLO or DeepSORT) is used to detect players and the ball in each frame and extract their coordinate data. This coordinate data represents the positional information of players and the ball in two or three dimensions.

[0321] The server calculates the movement patterns and speeds of players and the ball from the coordinate data of consecutive frames. This allows the running distance and speed of each player to be visualized. The calculated coordinate and speed data are then used to detect events during the game (e.g., passes, shots, goals). Detected events are tagged. Furthermore, a skeleton detection algorithm such as OpenPose or MediaPipe is used to extract players' skeletal data (joint positions). Based on this skeletal data, players' movements (e.g., jump shots and sliding) are analyzed.

[0322] The server combines the generated coordinate data and event detection results to generate statistical data for each player (e.g., pass success rate, number of successful shots, number of turnovers). This statistical data is delivered to the device in real time. In addition, the server analyzes the user's real-time video and audio data using an emotion engine to recognize the user's emotional state (e.g., excitement, stress, interest). The server combines the emotion data generated by the emotion engine with the statistical data and event data to generate game highlights and other interesting content based on the user's interests and reactions, and delivers it to the device.

[0323] Terminal handling

[0324] The device receives statistical and emotional data sent from the server in real time and analyzes it instantly. The device visually displays an overview of the entire game and performance data for each player. It also selectively displays interesting highlights and specific plays based on the user's emotions detected by the emotion engine. Furthermore, when important game events occur, the device notifies the user in real time.

[0325] User operations

[0326] Users can check the progress of the game in real time through their devices. Images from a bird's-eye view camera and analytical data are displayed simultaneously, and they can select and check the performance data of specific players or highlights of the game. An emotion engine displays content that reflects the user's emotional data, improving the user experience.

[0327] Specific examples

[0328] For example, in a soccer match, the server captures camera footage from the start of the match, and the AI ​​model updates the position data of each player and the ball every 0.1 seconds. If the server detects that player A passes the ball and player B receives it, it tags the action as a "pass" and increments the number of successful passes in real time. If the emotion engine recognizes the user's excitement, the device will visually display highlights of the play according to the user's excitement level.

[0329] Prompt Sentence Examples

[0330] "Analyze the number of passes and ball possession time of Player A in a soccer match in real time, and display highlights according to the user's excitement level."

[0331] The system will enable spectators to feel more at home in the match, gain detailed insights based on player performance analysis, and display content based on emotional data, providing a more personalized viewing experience.

[0332] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0333] Step 1: Receiving video data from the overhead camera and tracking camera

[0334] The server receives video data in real time from the overhead camera and tracking camera. This video data is divided into frames at high resolution. The input is the video data sent from the camera, and the output is the video data for each frame stored in the server.

[0335] Step 2: Splitting and Preprocessing Frames

[0336] The server divides the received video data into frames and performs preprocessing, which includes noise removal and frame alignment. The input is video data for each frame, and the output is the preprocessed frame data.

[0337] Step 3: Object detection and coordinate data extraction

[0338] The server applies an object detection algorithm (such as YOLO or DeepSORT) to the preprocessed frame data to extract the coordinate data of players and the ball. The input is the preprocessed frame data, and the output is the coordinate data (e.g., x-coordinate, y-coordinate) of players and the ball in each frame.

[0339] Step 4: Calculate movement patterns and speeds

[0340] The server uses the coordinate data of consecutive frames to calculate the movement patterns and speeds of players and the ball. For example, if player A is located at x=100, y=200 in frame 1 and moves to x=120, y=220 in frame 2, the server calculates the movement distance and speed between them. The input is the coordinate data of consecutive frames, and the output is the movement pattern and speed data.

[0341] Step 5: Detecting an event

[0342] The server analyzes movement pattern and speed data to detect events (e.g., passes, shots, goals) during a match. For example, if it detects that player A passes the ball and player B receives it, it tags this action as a "pass." The input is movement pattern and speed data, and the output is event-tagged data.

[0343] Step 6: Extracting skeletal data and motion analysis

[0344] The server uses OpenPose and MediaPipe to extract the player's skeletal data. Then, it analyzes the player's movements (e.g., jumping, sliding) based on this skeletal data. The input is frame data and coordinate data, and the output is skeletal data and the results of the movement analysis.

[0345] Step 7: Generate and distribute statistical data

[0346] The server integrates the coordinate data, event detection results, and motion analysis results to generate statistical data for each player (e.g., pass success rate, number of successful shots). The generated statistical data is delivered to the device in real time. The input is coordinate data, event data, and motion analysis results, and the output is statistical data.

[0347] Step 8: Emotion Recognition with the Emotion Engine

[0348] The server analyzes the user's real-time video and audio data using an emotion engine to recognize the user's emotional state (e.g., excitement, stress, interest). The input is the user's video and audio data, and the output is the user's emotional data.

[0349] Step 9: Integrate and utilize emotion data

[0350] The server integrates the emotion data generated by the emotion engine with statistical data and event data to generate match highlights and engaging content based on the user's interests and reactions. The generated content is delivered to the device. The inputs are statistical data, event data, and emotion data, and the output is customized highlights and content.

[0351] (Application example 2)

[0352] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0353] Traditional sports viewing systems limit the amount of information spectators can obtain in real time, making it difficult for them to gain a deep understanding of the game's movements and tactics. Furthermore, they lack a mechanism for automatically displaying content that matches the spectator's interests and emotional state, resulting in a lack of a personalized viewing experience for each spectator. This often results in a uniform and uninteresting viewing experience for spectators.

[0354] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for processing video data received from the overhead camera and tracking camera in real time, means for extracting coordinate data of people and the ball from the video data, means for detecting the movement patterns, speed, and events of people and the ball based on the coordinate data, means for integrating the statistical data and emotional data, generating game highlights and interesting content based on the user's interests and reactions, and delivering the content to the terminal in real time, and means for recognizing the user's emotional state using an emotion engine. This allows spectators to not only grasp detailed game movements and tactical analysis in real time, but also enjoy personalized content tailored to their individual emotional states.

[0355] A "bird's-eye view camera" is a camera installed in a position that allows it to overlook the entire venue from above, and is a device that captures footage of the entire match.

[0356] A "tracking camera" is a high-resolution camera that is placed at key points in the game to track specific players or the ball in detail.

[0357] "Video data" refers to digital video data received from the overhead camera and tracking camera and processed in real time.

[0358] "Coordinate data" refers to data that indicates the position information of people and balls extracted from video data, and is expressed in two or three dimensions.

[0359] A "movement pattern" indicates the movement path of a person and a ball calculated from continuous coordinate data.

[0360] An "event" is a specific action or occurrence that occurs during a match (e.g., a pass, a shot, a goal) and is tagged accordingly.

[0361] "Statistical data" refers to data generated based on coordinate data and event detection results, and includes performance information for each player (e.g., pass success rate, number of successful shots).

[0362] The "emotion engine" is a system that analyzes the user's real-time video and audio data to recognize the user's emotional state (e.g., excitement, stress, interest).

[0363] "Emotion Data" refers to data that indicates the user's emotional state as analyzed and recognized by the emotion engine.

[0364] "Highlights" is content that highlights important scenes from the game or parts that will interest spectators.

[0365] "User experience" refers to the overall experience a user has through a system, including operability and satisfaction.

[0366] To implement this invention, the following system is required: This system collects video data in real time using a bird's-eye view camera and a tracking camera, and combines it with an emotion engine that recognizes the user's emotions to improve the user experience.

[0367] Hardware Configuration

[0368] 1. Bird's-eye view camera: A camera installed in a position that allows a bird's-eye view of the entire venue. Specifically, general-purpose cameras such as Arducam can be used.

[0369] 2. Tracking camera: A high-resolution camera for tracking a specific player or the ball in detail. For example, a Basler camera can be used.

[0370] 3. Head-mounted display: A device that allows spectators to virtually watch the game. An example of this is the Oculus Quest 2.

[0371] Software Configuration

[0372] 1. Object detection algorithm: An algorithm that extracts coordinate data of people and spheres from video data acquired from overhead cameras and tracking cameras. Specifically, YOLO (You Only Look Once) and DeepSORT can be used.

[0373] 2. Skeleton detection algorithm: An algorithm for analyzing human skeletal data. Examples of algorithms that can be used include OpenPose and MediaPipe.

[0374] 3. Emotion engine: A system that analyzes real-time video and audio data of the user to recognize their emotional state. For example, Affectiva's emotion recognition API can be used.

[0375] 4. 3D modeling software: Software for visualizing the events during the match in 3D. For example, Unity can be used.

[0376] 5. Image processing library: A library for processing camera images. OpenCV can be used as a specific example.

[0377] Data calculation and processing

[0378] The server processes video data received from the overhead camera and tracking camera in real time. First, it divides the video data into frames and uses an object detection algorithm to detect people and the ball in each frame and extract their coordinate data. Next, it calculates the movement patterns and speed of people and the ball from the coordinate data of consecutive frames to detect events during the game. This data is integrated and delivered to the device in real time. It also uses an emotion engine to recognize the user's emotions, and combines the emotion data with statistical data to generate game highlights and interesting content tailored to the user, which is delivered to the device.

[0379] Specific examples

[0380] For example, during a soccer match, the server captures video from an overhead camera and a tracking camera, and the AL model updates the position data of each player and the ball every 0.1 seconds. If the server detects that player A passes the ball and player B receives it, it tags the action as a "pass" and increments the number of successful passes in real time. If the emotion engine recognizes the user's excitement, the device will visually display highlights of the play according to the user's excitement level.

[0381] Prompt Sentence Examples

[0382] A scenario where you are watching a soccer match in real time. Accurately track player movements and ball position and highlight key plays. Automatically generate and present relevant highlights based on the user's changing interests and emotions. The emotion engine should recognize the user's emotions and provide personalized match analysis and notifications based on those emotions.

[0383] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0384] Step 1:

[0385] The server receives video data from the overhead camera and tracking camera in real time and splits it into frames. The input is the video stream from the camera, and the output is the split frame data. This processing is performed using OpenCV.

[0386] Step 2:

[0387] The server applies an object detection algorithm (such as YOLO or DeepSORT) to each divided frame to detect players and the ball. The input is the frame data, and the output is the coordinate data of the detected objects (players and the ball). At this time, the coordinate data is recorded along two or three dimensions.

[0388] Step 3:

[0389] The server calculates the movement patterns and speeds of players and the ball based on coordinate data extracted from consecutive frames. The input is a continuous set of coordinate data, and the output is movement pattern data and speed data. In this calculation, the speed is calculated based on the rate of change of position information.

[0390] Step 4:

[0391] The server detects events (e.g., passes, shots, goals) during a match based on movement patterns and velocity data. The input is movement patterns and velocity data, and the output is event-tagged data. The event detection algorithm identifies events based on specific movement actions and velocity changes.

[0392] Step 5:

[0393] The server uses an object detection algorithm (OpenPose or MediaPipe) to extract the player's skeletal data. The input is a video frame, and the output is the player's skeletal coordinate data. This data is recorded as joint position information.

[0394] Step 6:

[0395] The server performs motion and tactical analysis of players based on their skeletal and coordinate data. The input is skeletal and coordinate data, and the output is motion analysis data and tactical analysis data. This processing identifies specific motion patterns and visualizes tactical performance.

[0396] Step 7:

[0397] The server generates statistics for each player based on the statistical data and event detection results. The input is a series of event data and coordinate data, and the output is a statistical data feed. In this step, player performance indicators (e.g., pass success rate, shot success rate) are calculated.

[0398] Step 8:

[0399] The server uses an emotion engine to analyze the user's real-time video and audio data to recognize the user's emotional state. The input is the user's video and audio data, and the output is emotional data. This process is performed using Affectiva's emotion recognition API.

[0400] Step 9:

[0401] The server integrates the emotional data and statistical data to generate highlights and other engaging content based on the user's interests and reactions, and delivers them to the device in real time. The input is emotional data and statistical data, and the output is personalized highlight content. The device receives this data and displays it visually.

[0402] Step 10:

[0403] Users can check the progress of the game in real time through their devices and enjoy personalized content. The input is data received from the server, and the output is an improved user experience, allowing users to experience a truly immersive game experience.

[0404] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0405] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0406] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0407] [Second embodiment]

[0408] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0409] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0410] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0411] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0412] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0413] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0414] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0415] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0416] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0417] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0418] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0419] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0420] The system of the present invention uses a bird's-eye view camera and a tracking camera to collect detailed data on the entire game, players, and the ball in real time, and automatically analyzes and compiles it. Specific embodiments of each processing step are described below.

[0421] This system begins with the placement of overhead cameras and tracking cameras. The overhead cameras are installed in a position that allows for a bird's-eye view of the entire stadium. Meanwhile, the tracking cameras are high-resolution cameras that are placed at key points in the game to track specific players or the ball in detail. These cameras are connected to a server and transmit video data in real time.

[0422] Server Processing

[0423] 1. Video data reception and preprocessing

[0424] The server receives video data from the overhead camera and tracking camera. After preprocessing such as noise reduction and resolution adjustment, the video data is divided into frames.

[0425] 2. Object detection and coordinate data extraction

[0426] The server uses an object detection algorithm (e.g., YOLO or DeepSORT) to detect players and the ball in each frame and extract their coordinate data, which represent the positional information of players and the ball in two or three dimensions.

[0427] 3. Calculating movement patterns and speeds

[0428] The server calculates the movement patterns and speeds of players and the ball from the coordinate data of successive frames, which allows visualization of each player's running distance and speed.

[0429] 4. Event Detection

[0430] The server automatically detects match events such as passes, shots, and goals based on the calculated coordinate and velocity data, and this event data is tagged and used for subsequent analysis.

[0431] 5. Skeletal data extraction and motion analysis

[0432] The server uses a skeletal detection algorithm such as OpenPose or MediaPipe to extract the skeletal data (joint positions) of each player, and analyzes the player's movements (e.g., jump shots and sliding).

[0433] 6. Generation and distribution of statistical data

[0434] The server combines the coordinate data and event data to generate statistical data for each player (e.g., pass success rate, number of successful shots, number of turnovers). The generated statistical data is delivered to the device in real time.

[0435] Terminal handling

[0436] 1. Data Receipt and Analysis

[0437] The terminal receives the real-time data sent from the server, analyzes the received data, and extracts the information required for display.

[0438] 2. Visualization and Notifications

[0439] The device visually displays an overview of the entire game and performance data for each player, and also notifies the user in real time when important events occur.

[0440] User operations

[0441] 1. Viewing real-time data

[0442] Users can check the progress of the match in real time through their devices, and can also view player performance data and match highlights.

[0443] 2. Detailed analysis and visualization

[0444] Users can select detailed analytical data for a specific period or player and view the data graphically visualized, providing insights for tactical analysis and performance improvement.

[0445] Specific examples

[0446] For example, in a basketball game, the server captures camera footage from the start of the game, and the AI ​​model updates the position data of each player and the ball every 0.1 seconds. If the server detects that player A passes the ball and player B receives it, it tags the action as a "pass" and increments the pass success count in real time. The device receives this information and visually displays detailed information about the pass to the user.

[0447] The above is the details of the mode for carrying out the invention. This system can be applied not only to basketball but also to soccer and other sports, so it can meet a variety of needs for sports data analysis.

[0448] The processing flow will be explained below.

[0449] Server Processing

[0450] Step 1:

[0451] The server receives video data from the overhead camera and tracking camera in real time, and the received video data is divided into frames.

[0452] Step 2:

[0453] The server performs pre-processing of the video data, including image pre-processing such as noise reduction, resolution adjustment, and color correction.

[0454] Step 3:

[0455] The server uses an object detection algorithm (e.g., YOLO or DeepSORT) to detect players and the ball for each frame, and extracts the position information of the detected players and the ball as coordinate data.

[0456] Step 4:

[0457] The server calculates the movement patterns of the players and the ball based on the coordinate data of successive frames, including the distance and speed of the players.

[0458] Step 5:

[0459] The server uses the calculated coordinate and velocity data to detect events during the match (e.g., passes, shots, goals), and tags the events.

[0460] Step 6:

[0461] The server uses a skeleton detection algorithm such as OpenPose or MediaPipe to extract the player's skeletal data, which includes the positions of each joint.

[0462] Step 7:

[0463] The server analyzes the player's movements based on the extracted skeletal data, identifying movements such as jump shots and sliding.

[0464] Step 8:

[0465] The server integrates the coordinate data and the event detection results to generate statistical data for each player (e.g., pass success rate, number of successful shots, number of turnovers).

[0466] Step 9:

[0467] The server delivers the generated statistical data to the device in real time, using standard formats such as JSON.

[0468] Terminal handling

[0469] Step 1:

[0470] The terminal receives real-time data sent from the server, and the received data is analyzed immediately.

[0471] Step 2:

[0472] Based on the analyzed data, the device displays a bird's-eye view of the entire match, along with visuals of each player's performance data.

[0473] Step 3:

[0474] The device will notify the user in real time when important match events occur, including pop-up messages and audio alerts.

[0475] Step 4:

[0476] When users click on a notification or select a specific player or time period, they are presented with an interface that displays detailed play data and highlights.

[0477] User operations

[0478] Step 1:

[0479] Users can check the progress of the match in real time through their devices, with overhead camera footage and analysis data displayed simultaneously.

[0480] Step 2:

[0481] Users can select a specific player's performance data and view detailed statistics, such as shooting percentage and distance traveled.

[0482] Step 3:

[0483] When a user selects an important event during a match (e.g., a goal or a foul), a replay of the corresponding play is displayed, allowing spectators to rewatch the important moments of the match.

[0484] Example 1

[0485] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0486] Existing sports data analysis systems have the problem of being unable to grasp the situation of a game in real time, and it takes a long time to analyze the detailed movements of players and the ball. Furthermore, because real-time notification functions for spectators are not fully developed, important moments of the game can be missed. Furthermore, there are issues with the accuracy and reliability of statistical data due to the inefficient fusion and analysis of data obtained from multiple cameras.

[0487] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0488] In this invention, the server includes a means for processing video data received from the overhead camera and tracking camera in real time, a means for extracting coordinate data of people and the ball from the video data using an object detection algorithm, and a means for detecting the movement patterns, speed, and events of people and the ball based on the coordinate data. This allows highly accurate coordinate data to be extracted from the video data in real time, enabling detailed analysis of the movements of players and the ball. Furthermore, by generating and distributing statistical data in real time, users can instantly grasp the situation of the game and watch without missing important moments. Furthermore, data obtained from multiple cameras is efficiently combined to provide highly reliable statistical data.

[0489] A "bird's-eye view camera" is a camera that takes pictures of the entire stadium from above.

[0490] A "tracking camera" is a camera that tracks a specific player or the ball in detail and takes high-resolution images.

[0491] "Real-time processing of video data" means instantly processing video data received from the overhead camera and tracking camera.

[0492] An "object detection algorithm" is an algorithm for identifying specific objects (such as players or the ball) from video data and extracting their location information.

[0493] "Coordinate data" is data that indicates the position of an object in an image, and is usually expressed as two-dimensional or three-dimensional coordinates.

[0494] A "movement pattern" indicates the movement path of an object calculated from continuous coordinate data.

[0495] "Velocity" indicates the amount of change in position per unit time when an object moves.

[0496] An "event" is a specific action or occurrence in a match (e.g., a pass, a shot, a goal) that is tagged when it is detected.

[0497] "Statistical Data" means statistical information about player or team performance that is generated based on extracted and analyzed data.

[0498] "Real-time delivery" refers to the process of instantly transmitting generated data to a user terminal.

[0499] "Skeletal data" is data that indicates the joint positions and structure of a person extracted from video data.

[0500] "Movement analysis" refers to the analysis of a person's specific movements using extracted skeletal data and coordinate data.

[0501] "Tactical analysis" is an analysis that evaluates tactics and playing style based on the results of motion analysis and identifies areas for improvement.

[0502] "Real-time notification" is a function that notifies the user of a specific event immediately when that event is detected.

[0503] A "user interface" is a display screen on a terminal that allows users to visually check the game situation and statistical data.

[0504] The system of the present invention analyzes sports data in real time to detect and analyze the detailed movements of players and the ball. This system uses an overhead camera and a tracking camera to collect detailed data on the entire game, players, and the ball, and automatically analyzes the data and provides it to the user. Specific embodiments of each element are described below.

[0505] Server configuration and processing

[0506] Video data reception and preprocessing

[0507] The server receives video data in real time from the overhead camera and tracking camera. The overhead camera is a high-resolution camera that captures the entire match venue from above, while the tracking camera is a camera that tracks players and the ball in detail. These cameras are connected to the server and transmit video streams in H.264 format. The server performs noise reduction (using Gaussian Blur, for example) and resolution adjustment on the received video data, and then divides it into the required frames.

[0508] Object detection and coordinate data extraction

[0509] The server uses an object detection algorithm (such as YOLO or DeepSORT) to detect players and the ball in the frame and extract their location information, which is then saved as 2D or 3D coordinate data.

[0510] Movement pattern and speed calculations

[0511] The server uses the coordinate data from successive frames to calculate the movement patterns and speeds of players and the ball, which then calculates the distance and speed of each player and stores them in a database.

[0512] Event detection and tagging

[0513] The server automatically detects events such as passes, shots, and goals under certain conditions based on coordinate and velocity data, and tags these events. The event data is stored in a database for subsequent analysis and report generation.

[0514] Skeletal data extraction and motion analysis

[0515] The server uses skeletal detection algorithms such as OpenPose and MediaPipe to extract the player's skeletal data (joint positions) and analyzes their movements (e.g., jump shots and sliding) based on this data.

[0516] Statistical data generation and distribution

[0517] The server integrates the coordinate data and event data to generate statistical data for each player (e.g., pass success rate, number of successful shots, number of turnovers). This statistical data is delivered to the device in real time using communication protocols such as WebSocket and REST API.

[0518] Terminal configuration and handling

[0519] Data reception and analysis

[0520] The device receives real-time data sent from the server, analyzes it, and extracts the necessary information, such as video frames and statistical data, which it stores in memory and reconstructs accordingly.

[0521] Visualization and Notifications

[0522] The device displays a bird's-eye view of the entire game and performance data for each player on the user interface, and notifies the user in real time via pop-up notifications when important events occur (e.g., successful passes or goals).

[0523] User operations

[0524] Viewing real-time data

[0525] Users can check the progress of the match in real time through their device, with match footage and player location information updated continuously, and players' performance data and match highlights visually displayed.

[0526] Detailed Analysis and Visualization

[0527] Users navigate through specific menus on their device to select detailed analytical data for specific periods or players, which are then visualized in graphs and charts to provide tactical analysis and insights for performance improvement.

[0528] Specific examples

[0529] For example, in a basketball game, the server captures video from the overhead camera and tracking camera from the start of the game and updates the position data of each player and the ball every 0.1 seconds using an object detection algorithm. If the server detects that player A passes the ball and player B receives it, it tags the action as a "pass" and increments the pass success count in real time. The device receives this information and visually displays the pass details to the user. For example, a notification pops up saying, "Pass from player A to player B was successful."

[0530] Example prompt for a generative AI model:

[0531] "In a basketball game, collect data on when player A successfully passes the ball to player B, and analyze it in real time."

[0532] The above is the detailed description of the mode for carrying out the invention. This system can be applied not only to basketball but also to soccer and other sports, and can meet a variety of needs for sports data analysis.

[0533] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0534] Server Processing

[0535] Step 1: Receiving and preprocessing video data

[0536] The server receives H.264 video data in real time from the overhead camera and tracking camera. The input is the video stream from the camera, and the output is preprocessed frame data. Specifically, Gaussian Blur is applied to remove noise, and the resolution is adjusted and divided into the required frames.

[0537] Step 2: Object detection and coordinate data extraction

[0538] The server applies an object detection algorithm (such as YOLO or DeepSORT) to the preprocessed frame data as input. The input is the preprocessed frame data, and the output is the coordinate data of the detected players and ball. For each frame, it detects specific objects from the pixel data and extracts their location information as 2D or 3D coordinates.

[0539] Step 3: Calculate movement patterns and speeds

[0540] The server inputs the coordinate data of successive frames and calculates the movement patterns and speeds. The input is the coordinate data for each frame, and the output is the movement patterns and speed data of the players and the ball. Specifically, the server calculates the speeds using the time intervals of the coordinate data and stores the results in a database.

[0541] Step 4: Event detection and tagging

[0542] The server automatically detects specific events (passes, shots, goals, etc.) in a match based on movement patterns and speed data. The input is coordinate data and speed data, and the output is the detected events and their tag data. As each event is detected, it is tagged and stored in a database.

[0543] Step 5: Extracting skeletal data and motion analysis

[0544] The server uses preprocessed frame data as input and performs skeletal detection using OpenPose or MediaPipe. The input is the preprocessed frame data, and the output is the player's skeletal data (joint positions). The extracted skeletal data is analyzed to recognize specific movements (e.g., jump shots and sliding).

[0545] Step 6: Generate and distribute statistical data

[0546] The server combines coordinate data, event data, and motion analysis data to generate statistical data for each player. The input is all of the above data, and the output is statistical data. This statistical data is delivered to the device in real time. WebSocket and REST API are used for delivery.

[0547] Terminal handling

[0548] Step 1: Receiving and analyzing data

[0549] The terminal receives real-time data sent from the server. The input is the real-time data from the server, and the output is the analyzed data required for display. The terminal analyzes the received data, stores e.g., video frames and statistical data in memory, and reconstructs them accordingly.

[0550] Step 2: Visualization and Notification

[0551] The device displays the received data in a user interface. The input is the analyzed data, and the output is a screen that displays an overview of the game and the performance of each player. When an important event occurs, the user is notified in real time by a pop-up notification.

[0552] User operations

[0553] Step 1: View real-time data

[0554] The user can check the progress of the game in real time through the device. The input is the display data from the device, and the output is the user's visual information. The progress of the game and the position information of the players are updated successively.

[0555] Step 2: Detailed analysis and visualization

[0556] Users operate specific menus on the device and select detailed analysis data. The input is the device's menu operations, and the output is the selected analysis data. The selected data is visualized in graphs and charts, providing insights for tactical analysis and performance improvement.

[0557] (Application example 1)

[0558] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0559] In conventional manufacturing sites, it has been difficult to monitor the operation of manufacturing machines in real time and immediately detect and notify abnormal operations. Furthermore, while precise operation analysis technology is required for efficient operation and quality control of manufacturing machines, there has been no system to achieve this. Furthermore, there has been a lack of means for workers to receive analysis results in real time on-site and respond quickly.

[0560] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0561] In this invention, the server includes an overhead camera, a tracking camera, means for processing video data received from the overhead camera and the tracking camera in real time, means for extracting coordinate data of people and objects from the video data, means for detecting movement patterns, speeds, and events of people and objects based on the coordinate data, means for generating statistical data based on the coordinate data and event detection results, means for detecting operation patterns of manufacturing machines and notifying abnormal operations in real time, and means for delivering the statistical data and abnormality detection results to terminals in real time.This enables real-time monitoring of the operation of manufacturing machines at a manufacturing site, immediate detection and notification of abnormal operations, and prompt response by workers on site.

[0562] A "bird's-eye view camera" is a camera that can capture a wide area from above.

[0563] A "tracking camera" is a high-resolution camera that can capture detailed images of a specific object while tracking it.

[0564] "Means for processing video data in real time" refers to technology that instantly processes video data collected from overhead cameras and tracking cameras.

[0565] "Means for extracting coordinate data" refers to technology that extracts the location information of specific people or objects from video data as numerical data.

[0566] "Means for detecting movement patterns, speed, and events" refers to technology that identifies the movement patterns and speed of objects, as well as specific events, based on the extracted coordinate data.

[0567] The "means for generating statistical data" refers to a technique for creating various statistical information using coordinate data and event detection results.

[0568] "Means for notifying abnormal operation in real time" refers to technology that monitors the operation of manufacturing machines and immediately notifies if an abnormality is detected.

[0569] "Means for delivering to terminals in real time" refers to technology that instantly transmits generated statistical data and anomaly detection results to the appropriate terminal.

[0570] The "means for detecting the operation pattern of a manufacturing machine" refers to a technology that monitors the operation of a manufacturing machine and identifies the characteristics and patterns of its movement.

[0571] "Means for conducting tactical analysis based on the results of motion analysis" refers to a technique for conducting detailed analysis from a tactical perspective based on extracted motion data.

[0572] "Means for providing real-time notifications to workers based on data on abnormal operations detected by the abnormality detection means" refers to technology that immediately notifies workers of details of an abnormality when an abnormality is detected.

[0573] This invention is a system that uses a bird's-eye view camera and a tracking camera to monitor and analyze the operation of manufacturing machines in a manufacturing site. The system is composed of a server, a terminal, and a user.

[0574] The server processes video data collected from the overhead camera and tracking camera in real time. The hardware used includes a high-resolution overhead camera, tracking camera, and server. The software includes OpenCV for Python, FFmpeg, YOLO, DeepSORT, and TensorFlow. The server removes noise from the received video data, adjusts the resolution, and divides it into frames. It then uses YOLO and DeepSORT to extract coordinate data for the manufacturing machine and work objects from the video data. The movement pattern and speed of the manufacturing machine are calculated from the coordinate data of consecutive frames.

[0575] The server detects abnormal behavior and patterns and generates anomaly detection results and statistical data, including the operational efficiency, operation history, and frequency of failures of the manufacturing machines.

[0576] The server also delivers anomaly detection results and statistical data in real time to the terminal. The terminal is a pair of smart glasses worn by the worker. The terminal uses Three.js and WebGL to visualize and display the received data in 3D space. If a specific anomaly is detected, the terminal notifies the worker in real time.

[0577] The user, a worker, can visually check the operation data of the manufacturing machine in real time through the smart glasses, which allows them to respond immediately if an abnormality occurs.

[0578] As a concrete example, consider a scenario in which the operation of a robot arm in a production line is monitored. An overhead camera monitors the entire production line, while a tracking camera tracks the movement of the robot arm. The server analyzes this video data and determines whether the robot arm is operating normally or if any abnormalities have occurred. If an abnormality is detected, a worker wearing smart glasses is immediately notified, enabling a prompt response.

[0579] An example prompt might look like this:

[0580] Monitor the movement patterns followed by specific manufacturing robots in your factory in real time, get instant notification if any abnormal behavior is detected, view the coordinate data and movement speed of each robot in 2D and 3D, and store a history of any abnormal behavior.

[0581] This system will significantly improve quality control and efficiency at the manufacturing site, and enable rapid response when abnormalities occur.

[0582] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0583] Step 1:

[0584] The server receives video data from the overhead camera and the tracking camera in real time.

[0585] Specific operation: The server receives the video signal from the camera and divides it into frames. The input is the video data captured by the camera, and the output is the data of each frame with noise removal and resolution adjustment. Preprocessing is performed using OpenCV for Python.

[0586] Step 2:

[0587] The server extracts coordinate data of the manufacturing machine and the work object from the pre-processed frame data.

[0588] Specific operation: Using YOLO and DeepSORT, we detect manufacturing machines and objects in the frame and obtain their coordinate data. The input is each preprocessed frame, and the output is the 2D coordinate data of the detected manufacturing machines and objects.

[0589] Step 3:

[0590] The server calculates the movement patterns and speeds of the manufacturing machines and objects based on the coordinate data between successive frames.

[0591] Specific operation: The server uses TensorFlow to analyze the acquired coordinate data and calculate the movement distance and speed. The input is the detected coordinate data, and the output is the calculated movement pattern and speed data.

[0592] Step 4:

[0593] The server detects abnormal behavior based on movement patterns and speed data.

[0594] Specific operation: The server compares the accumulated movement pattern and speed data to determine whether there are any abnormalities. In this step, an algorithm is used to instantly detect abnormal behavior. Its input is the movement pattern and speed data, and its output is the data of the detected abnormal behavior.

[0595] Step 5:

[0596] The server notifies the worker in real time if any abnormal operation is detected.

[0597] Specific operation: When an abnormality is detected, the server immediately sends an abnormal operation notification to the worker's smart glasses using Socket.IO. The input is the detected abnormal operation data, and the output is a real-time notification.

[0598] Step 6:

[0599] The terminal visualizes abnormal behavior data and statistical data received from the server.

[0600] How it works: The smart glasses use Three.js and WebGL to visually display the received data in 3D space. Its input is abnormal behavior data and statistical data from the server, and its output is a visualized interface.

[0601] Step 7:

[0602] Using smart glasses, users can check the operation of manufacturing machines and receive abnormality notifications in real time and respond accordingly.

[0603] Specific Action: The user checks the display of the smart glasses and takes action if necessary. In this step, the input received by the user is the display data of the smart glasses, and the output is the corresponding action.

[0604] This series of processes makes it possible to monitor the operation of manufacturing machines at the manufacturing site in real time and to immediately notify workers if an abnormality occurs.

[0605] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0606] The system of the present invention uses a bird's-eye view camera and a tracking camera to collect detailed data on the entire game, players, and the ball in real time, and automatically analyzes and compiles it. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it aims to improve the user experience. Below, a specific embodiment of each processing step is explained.

[0607] This system begins with the placement of overhead cameras and tracking cameras. The overhead cameras are installed in a position that allows for a bird's-eye view of the entire stadium. Meanwhile, the tracking cameras are high-resolution cameras that are placed at key points in the game to track specific players or the ball in detail. These cameras are connected to a server and transmit video data in real time.

[0608] Server Processing

[0609] 1. Video data reception and preprocessing

[0610] The server receives video data from the overhead camera and tracking camera in real time, and the received video data is divided into frames.

[0611] 2. Object detection and coordinate data extraction

[0612] The server uses an object detection algorithm (e.g., YOLO or DeepSORT) to detect players and the ball in each frame and extract their coordinate data, which represents the positional information of players and the ball in two or three dimensions.

[0613] 3. Calculating movement patterns and speeds

[0614] The server calculates the movement patterns and speeds of players and the ball from the coordinate data of successive frames, which allows visualization of each player's running distance and speed.

[0615] 4. Event Detection

[0616] The server uses the calculated coordinate and velocity data to detect events during the match (e.g., passes, shots, goals), and tags the events.

[0617] 5. Skeletal data extraction and motion analysis

[0618] The server uses a skeletal detection algorithm such as OpenPose or MediaPipe to extract the player's skeletal data (joint positions), which is then used to analyze the player's movements (e.g., jump shots and sliding).

[0619] 6. Generating and distributing statistical data

[0620] The server combines the coordinate data and event detection results to generate statistical data for each player (e.g., pass success rate, number of successful shots, number of turnovers). The generated statistical data is delivered to the device in real time.

[0621] 7. Emotion Recognition with Emotion Engine

[0622] The server analyzes the user's real-time video and audio data using an emotion engine to recognize the user's emotional state (e.g., excitement, stress, interest).

[0623] 8. Integration and Utilization of Emotional Data

[0624] The server integrates the emotion data generated by the emotion engine with statistical data and event data, generates match highlights and other interesting content based on the user's interests and reactions, and delivers them to the device.

[0625] Terminal handling

[0626] 1. Data Receipt and Analysis

[0627] The device receives statistical data and emotion data sent from the server in real time, and the received data is analyzed immediately.

[0628] 2. Visualization and Notifications

[0629] The device visually displays an overview of the entire game and performance data for each player. It also selectively displays interesting highlights and specific plays based on the user's emotions detected by an emotion engine. Furthermore, it notifies the user in real time when important game events occur.

[0630] User operations

[0631] 1. Viewing real-time data

[0632] Users can check the progress of the match in real time through their devices, with overhead camera footage and analysis data displayed simultaneously.

[0633] 2. Content display based on detailed analysis and emotion recognition

[0634] Users can select and view the performance data of specific players or match highlights, and the emotion engine displays content that reflects the user's emotional data, improving the user experience.

[0635] Specific examples

[0636] For example, in a soccer match, the server captures camera footage from the start of the match, and the AL model updates the position data of each player and the ball every 0.1 seconds. If the server detects that player A passes the ball and player B receives it, it tags the action as a "pass" and increments the number of successful passes in real time. If the emotion engine recognizes the user's excitement, the device will visually display highlights of the play according to the user's excitement level.

[0637] The system will enable spectators to feel more at home in the match, gain detailed insights based on player performance analysis, and display content based on emotional data, providing a more personalized viewing experience.

[0638] As described above, the system of the present invention aims to improve the user experience by integrating multi-camera video analysis and emotion recognition technology to realize automatic analysis of match data and real-time notification.

[0639] The processing flow will be explained below.

[0640] Server Processing

[0641] Step 1:

[0642] The server receives video data from the overhead camera and tracking camera in real time and divides the received video data into frames.

[0643] Step 2:

[0644] The server performs pre-processing of the video data, specifically image pre-processing such as noise removal, resolution adjustment, and color correction.

[0645] Step 3:

[0646] The server uses an object detection algorithm (e.g., YOLO or DeepSORT) to detect players and the ball in each frame, and extracts the position information of the detected players and the ball as coordinate data.

[0647] Step 4:

[0648] The server calculates the movement patterns of the players and the ball based on the coordinate data of successive frames, including the distance and speed of the players.

[0649] Step 5:

[0650] The server automatically detects events (e.g., passes, shots, goals) during a match based on the calculated coordinate and velocity data, and tags the detected events.

[0651] Step 6:

[0652] The server uses a skeletal detection algorithm such as OpenPose or MediaPipe to extract skeletal data for each player, including the position information of each joint.

[0653] Step 7:

[0654] The server analyzes the player's movements based on the extracted skeletal data, identifying movements such as jump shots and sliding.

[0655] Step 8:

[0656] The server integrates the coordinate data and the event detection results to generate statistical data for each player (e.g., pass success rate, number of successful shots, number of turnovers).

[0657] Step 9:

[0658] The server delivers the generated statistical data to the device in real time, using standard formats such as JSON.

[0659] Step 10:

[0660] The server analyzes real-time video and audio data acquired from the user using an emotion engine to recognize the user's emotional state (e.g., excitement, stress, interest).

[0661] Step 11:

[0662] The server integrates the emotion data obtained by the emotion engine with statistical and event data, and generates game highlights and other engaging content based on the user's interests and reactions.

[0663] Step 12:

[0664] The server delivers content based on the generated emotion data to the device in real time, which is displayed as a visual feed that reflects the user's emotional state.

[0665] Terminal handling

[0666] Step 1:

[0667] The device receives statistical data and emotional data sent from the server in real time, and the received data is analyzed immediately.

[0668] Step 2:

[0669] Based on the analyzed data, the device visually displays an overview of the entire match and performance data for each player, including the player's movement patterns and speed.

[0670] Step 3:

[0671] The terminal selectively displays interesting highlights or specific plays based on the user's emotional state detected by the emotion engine, thereby providing content that attracts the user's interest.

[0672] Step 4:

[0673] The device will notify the user in real time when important match events occur, including pop-up messages and audio alerts.

[0674] User operations

[0675] Step 1:

[0676] Users can check the progress of the match in real time through their devices, with overhead camera footage and analysis data displayed simultaneously.

[0677] Step 2:

[0678] Users can select a specific player's performance data and view detailed statistics, such as shooting percentage and distance traveled.

[0679] Step 3:

[0680] The device will display highlights and detailed information related to the user's favorite players and plays of interest based on the user's emotional data recognized by the emotion engine, providing a more personalized viewing experience.

[0681] Example 2

[0682] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0683] Conventional match analysis systems can acquire player and ball position data and detect events based on that data. However, these systems do not take into account user emotional data, which means they cannot provide content optimized for each spectator. Furthermore, there is a lack of technology to perform detailed tactical analysis in real time, making it difficult to enhance the sense of realism in live match viewing.

[0684] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0685] In this invention, the server includes an overhead camera, a tracking camera, a means for processing video data received from the overhead camera and the tracking camera in real time, a means for extracting object coordinate data from the video data, a means for calculating the object's movement pattern and speed based on the coordinate data, a means for detecting events from the coordinate data and speed data, a means for generating statistical data based on the coordinate data and the event detection results, a means for distributing the statistical data to a terminal in real time, a means for recognizing a user's emotion, and a means for integrating the emotion data with the statistical data and event data to generate individually customized content. This enables the provision of personalized content that reflects the user's emotion, achieving a more realistic viewing experience. Furthermore, detailed tactical analysis can be performed in real time, enabling the provision of more accurate data.

[0686] A "bird's-eye view camera" is a camera that is installed in a position that allows it to overlook the entire venue from above and captures video data over a wide area.

[0687] A "tracking camera" is a high-resolution camera installed to closely track a specific player or the ball.

[0688] The "means for processing video data in real time" is a function that includes a process for dividing video data received from the overhead camera and tracking camera and extracting objects and coordinates.

[0689] "Means for extracting coordinate data" refers to an algorithm or system for obtaining position information of objects (such as players or the ball) from video data.

[0690] The "means for calculating the movement pattern and speed" is a function for calculating the movement path and speed of an object based on the coordinate data of successive frames.

[0691] The "event detection means" is a system that identifies important actions during a match (e.g., passes, shots, goals) based on movement patterns and speed data.

[0692] The "means for generating statistical data" is a function that integrates coordinate data and event detection results to calculate detailed statistical information for each player (e.g., pass success rate, number of successful shots).

[0693] The "means for delivering to terminals in real time" refers to a process and system for instantly transmitting the generated statistical data to user terminals.

[0694] The "means for recognizing user emotions" is a technology that analyzes the user's facial expressions and vocal tone to identify emotional states such as excitement, stress, interest, etc.

[0695] "Means for integrating emotional data with statistical data and event data to generate individually customized content" refers to a system that combines recognized user emotional data with other match data to create content (e.g., highlights) that is tailored to each individual user.

[0696] The system of this invention uses a bird's-eye view camera and a tracking camera to collect detailed data on the entire game, players, and the ball in real time, and automatically analyzes and compiles it.Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it aims to improve the user experience.

[0697] The system begins with the placement of overhead cameras and tracking cameras. The overhead cameras are installed in a position that allows for a bird's-eye view of the entire stadium, while the tracking cameras are high-resolution cameras that are placed at key points in the game to track specific players or the ball in detail. These cameras are connected to a server and transmit video data in real time.

[0698] Server Processing

[0699] The server first receives video data in real time from the overhead camera and tracking camera. The received video data is divided into frames, and an object detection algorithm (e.g., YOLO or DeepSORT) is used to detect players and the ball in each frame and extract their coordinate data. This coordinate data represents the positional information of players and the ball in two or three dimensions.

[0700] The server calculates the movement patterns and speeds of players and the ball from the coordinate data of consecutive frames. This allows the running distance and speed of each player to be visualized. The calculated coordinate and speed data are then used to detect events during the game (e.g., passes, shots, goals). Detected events are tagged. Furthermore, a skeleton detection algorithm such as OpenPose or MediaPipe is used to extract players' skeletal data (joint positions). Based on this skeletal data, players' movements (e.g., jump shots and sliding) are analyzed.

[0701] The server combines the generated coordinate data and event detection results to generate statistical data for each player (e.g., pass success rate, number of successful shots, number of turnovers). This statistical data is delivered to the device in real time. In addition, the server analyzes the user's real-time video and audio data using an emotion engine to recognize the user's emotional state (e.g., excitement, stress, interest). The server combines the emotion data generated by the emotion engine with the statistical data and event data to generate game highlights and other interesting content based on the user's interests and reactions, and delivers it to the device.

[0702] Terminal handling

[0703] The device receives statistical and emotional data sent from the server in real time and analyzes it instantly. The device visually displays an overview of the entire game and performance data for each player. It also selectively displays interesting highlights and specific plays based on the user's emotions detected by the emotion engine. Furthermore, when important game events occur, the device notifies the user in real time.

[0704] User operations

[0705] Users can check the progress of the game in real time through their devices. Images from a bird's-eye view camera and analytical data are displayed simultaneously, and they can select and check the performance data of specific players or highlights of the game. An emotion engine displays content that reflects the user's emotional data, improving the user experience.

[0706] Specific examples

[0707] For example, in a soccer match, the server captures camera footage from the start of the match, and the AI ​​model updates the position data of each player and the ball every 0.1 seconds. If the server detects that player A passes the ball and player B receives it, it tags the action as a "pass" and increments the number of successful passes in real time. If the emotion engine recognizes the user's excitement, the device will visually display highlights of the play according to the user's excitement level.

[0708] Prompt Sentence Examples

[0709] "Analyze the number of passes and ball possession time of Player A in a soccer match in real time, and display highlights according to the user's excitement level."

[0710] The system will enable spectators to feel more at home in the match, gain detailed insights based on player performance analysis, and display content based on emotional data, providing a more personalized viewing experience.

[0711] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0712] Step 1: Receiving video data from the overhead camera and tracking camera

[0713] The server receives video data in real time from the overhead camera and tracking camera. This video data is divided into frames at high resolution. The input is the video data sent from the camera, and the output is the video data for each frame stored in the server.

[0714] Step 2: Splitting and Preprocessing Frames

[0715] The server divides the received video data into frames and performs preprocessing, which includes noise removal and frame alignment. The input is video data for each frame, and the output is the preprocessed frame data.

[0716] Step 3: Object detection and coordinate data extraction

[0717] The server applies an object detection algorithm (such as YOLO or DeepSORT) to the preprocessed frame data to extract the coordinate data of players and the ball. The input is the preprocessed frame data, and the output is the coordinate data (e.g., x-coordinate, y-coordinate) of players and the ball in each frame.

[0718] Step 4: Calculate movement patterns and speeds

[0719] The server uses the coordinate data of consecutive frames to calculate the movement patterns and speeds of players and the ball. For example, if player A is located at x=100, y=200 in frame 1 and moves to x=120, y=220 in frame 2, the server calculates the movement distance and speed between them. The input is the coordinate data of consecutive frames, and the output is the movement pattern and speed data.

[0720] Step 5: Detecting an event

[0721] The server analyzes movement pattern and speed data to detect events (e.g., passes, shots, goals) during a match. For example, if it detects that player A passes the ball and player B receives it, it tags this action as a "pass." The input is movement pattern and speed data, and the output is event-tagged data.

[0722] Step 6: Extracting skeletal data and motion analysis

[0723] The server uses OpenPose and MediaPipe to extract the player's skeletal data. Then, it analyzes the player's movements (e.g., jumping, sliding) based on this skeletal data. The input is frame data and coordinate data, and the output is skeletal data and the results of the movement analysis.

[0724] Step 7: Generate and distribute statistical data

[0725] The server integrates the coordinate data, event detection results, and motion analysis results to generate statistical data for each player (e.g., pass success rate, number of successful shots). The generated statistical data is delivered to the device in real time. The input is coordinate data, event data, and motion analysis results, and the output is statistical data.

[0726] Step 8: Emotion Recognition with the Emotion Engine

[0727] The server analyzes the user's real-time video and audio data using an emotion engine to recognize the user's emotional state (e.g., excitement, stress, interest). The input is the user's video and audio data, and the output is the user's emotional data.

[0728] Step 9: Integrate and utilize emotion data

[0729] The server integrates the emotion data generated by the emotion engine with statistical data and event data to generate match highlights and engaging content based on the user's interests and reactions. The generated content is delivered to the device. The inputs are statistical data, event data, and emotion data, and the output is customized highlights and content.

[0730] (Application example 2)

[0731] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0732] Traditional sports viewing systems limit the amount of information spectators can obtain in real time, making it difficult for them to gain a deep understanding of the game's movements and tactics. Furthermore, they lack a mechanism for automatically displaying content that matches the spectator's interests and emotional state, resulting in a lack of a personalized viewing experience for each spectator. This often results in a uniform and uninteresting viewing experience for spectators.

[0733] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for processing video data received from the overhead camera and tracking camera in real time, means for extracting coordinate data of people and the ball from the video data, means for detecting the movement patterns, speed, and events of people and the ball based on the coordinate data, means for integrating the statistical data and emotional data, generating game highlights and interesting content based on the user's interests and reactions, and delivering the content to the terminal in real time, and means for recognizing the user's emotional state using an emotion engine. This allows spectators to not only grasp detailed game movements and tactical analysis in real time, but also enjoy personalized content tailored to their individual emotional states.

[0734] A "bird's-eye view camera" is a camera installed in a position that allows it to overlook the entire venue from above, and is a device that captures footage of the entire match.

[0735] A "tracking camera" is a high-resolution camera that is placed at key points in the game to track specific players or the ball in detail.

[0736] "Video data" refers to digital video data received from the overhead camera and tracking camera and processed in real time.

[0737] "Coordinate data" refers to data that indicates the position information of people and balls extracted from video data, and is expressed in two or three dimensions.

[0738] A "movement pattern" indicates the movement path of a person and a ball calculated from continuous coordinate data.

[0739] An "event" is a specific action or occurrence that occurs during a match (e.g., a pass, a shot, a goal) and is tagged accordingly.

[0740] "Statistical data" refers to data generated based on coordinate data and event detection results, and includes performance information for each player (e.g., pass success rate, number of successful shots).

[0741] The "emotion engine" is a system that analyzes the user's real-time video and audio data to recognize the user's emotional state (e.g., excitement, stress, interest).

[0742] "Emotion Data" refers to data that indicates the user's emotional state as analyzed and recognized by the emotion engine.

[0743] "Highlights" is content that highlights important scenes from the game or parts that will interest spectators.

[0744] "User experience" refers to the overall experience a user has through a system, including operability and satisfaction.

[0745] To implement this invention, the following system is required: This system collects video data in real time using a bird's-eye view camera and a tracking camera, and combines it with an emotion engine that recognizes the user's emotions to improve the user experience.

[0746] Hardware Configuration

[0747] 1. Bird's-eye view camera: A camera installed in a position that allows a bird's-eye view of the entire venue. Specifically, general-purpose cameras such as Arducam can be used.

[0748] 2. Tracking camera: A high-resolution camera for tracking a specific player or the ball in detail. For example, a Basler camera can be used.

[0749] 3. Head-mounted display: A device that allows spectators to virtually watch the game. An example of this is the Oculus Quest 2.

[0750] Software Configuration

[0751] 1. Object detection algorithm: An algorithm that extracts coordinate data of people and spheres from video data acquired from overhead cameras and tracking cameras. Specifically, YOLO (You Only Look Once) and DeepSORT can be used.

[0752] 2. Skeleton detection algorithm: An algorithm for analyzing human skeletal data. Examples of algorithms that can be used include OpenPose and MediaPipe.

[0753] 3. Emotion engine: A system that analyzes real-time video and audio data of the user to recognize their emotional state. For example, Affectiva's emotion recognition API can be used.

[0754] 4. 3D modeling software: Software for visualizing the events during the match in 3D. For example, Unity can be used.

[0755] 5. Image processing library: A library for processing camera images. OpenCV can be used as a specific example.

[0756] Data calculation and processing

[0757] The server processes video data received from the overhead camera and tracking camera in real time. First, it divides the video data into frames and uses an object detection algorithm to detect people and the ball in each frame and extract their coordinate data. Next, it calculates the movement patterns and speed of people and the ball from the coordinate data of consecutive frames to detect events during the game. This data is integrated and delivered to the device in real time. It also uses an emotion engine to recognize the user's emotions, and combines the emotion data with statistical data to generate game highlights and interesting content tailored to the user, which is delivered to the device.

[0758] Specific examples

[0759] For example, during a soccer match, the server captures video from an overhead camera and a tracking camera, and the AL model updates the position data of each player and the ball every 0.1 seconds. If the server detects that player A passes the ball and player B receives it, it tags the action as a "pass" and increments the number of successful passes in real time. If the emotion engine recognizes the user's excitement, the device will visually display highlights of the play according to the user's excitement level.

[0760] Prompt Sentence Examples

[0761] A scenario where you are watching a soccer match in real time. Accurately track player movements and ball position and highlight key plays. Automatically generate and present relevant highlights based on the user's changing interests and emotions. The emotion engine should recognize the user's emotions and provide personalized match analysis and notifications based on those emotions.

[0762] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0763] Step 1:

[0764] The server receives video data from the overhead camera and tracking camera in real time and splits it into frames. The input is the video stream from the camera, and the output is the split frame data. This processing is performed using OpenCV.

[0765] Step 2:

[0766] The server applies an object detection algorithm (such as YOLO or DeepSORT) to each divided frame to detect players and the ball. The input is the frame data, and the output is the coordinate data of the detected objects (players and the ball). At this time, the coordinate data is recorded along two or three dimensions.

[0767] Step 3:

[0768] The server calculates the movement patterns and speeds of players and the ball based on coordinate data extracted from consecutive frames. The input is a continuous set of coordinate data, and the output is movement pattern data and speed data. In this calculation, the speed is calculated based on the rate of change of position information.

[0769] Step 4:

[0770] The server detects events (e.g., passes, shots, goals) during a match based on movement patterns and velocity data. The input is movement patterns and velocity data, and the output is event-tagged data. The event detection algorithm identifies events based on specific movement actions and velocity changes.

[0771] Step 5:

[0772] The server uses an object detection algorithm (OpenPose or MediaPipe) to extract the player's skeletal data. The input is a video frame, and the output is the player's skeletal coordinate data. This data is recorded as joint position information.

[0773] Step 6:

[0774] The server performs motion and tactical analysis of players based on their skeletal and coordinate data. The input is skeletal and coordinate data, and the output is motion analysis data and tactical analysis data. This processing identifies specific motion patterns and visualizes tactical performance.

[0775] Step 7:

[0776] The server generates statistics for each player based on the statistical data and event detection results. The input is a series of event data and coordinate data, and the output is a statistical data feed. In this step, player performance indicators (e.g., pass success rate, shot success rate) are calculated.

[0777] Step 8:

[0778] The server uses an emotion engine to analyze the user's real-time video and audio data to recognize the user's emotional state. The input is the user's video and audio data, and the output is emotional data. This process is performed using Affectiva's emotion recognition API.

[0779] Step 9:

[0780] The server integrates the emotional data and statistical data to generate highlights and other engaging content based on the user's interests and reactions, and delivers them to the device in real time. The input is emotional data and statistical data, and the output is personalized highlight content. The device receives this data and displays it visually.

[0781] Step 10:

[0782] Users can check the progress of the game in real time through their devices and enjoy personalized content. The input is data received from the server, and the output is an improved user experience, allowing users to experience a truly immersive game experience.

[0783] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0784] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0785] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0786] [Third embodiment]

[0787] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0788] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0789] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0790] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0791] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0792] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0793] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0794] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0795] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0796] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0797] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0798] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0799] The system of the present invention uses a bird's-eye view camera and a tracking camera to collect detailed data on the entire game, players, and the ball in real time, and automatically analyzes and compiles it. Specific embodiments of each processing step are described below.

[0800] This system begins with the placement of overhead cameras and tracking cameras. The overhead cameras are installed in a position that allows for a bird's-eye view of the entire stadium. Meanwhile, the tracking cameras are high-resolution cameras that are placed at key points in the game to track specific players or the ball in detail. These cameras are connected to a server and transmit video data in real time.

[0801] Server Processing

[0802] 1. Video data reception and preprocessing

[0803] The server receives video data from the overhead camera and tracking camera. After preprocessing such as noise reduction and resolution adjustment, the video data is divided into frames.

[0804] 2. Object detection and coordinate data extraction

[0805] The server uses an object detection algorithm (e.g., YOLO or DeepSORT) to detect players and the ball in each frame and extract their coordinate data, which represent the positional information of players and the ball in two or three dimensions.

[0806] 3. Calculating movement patterns and speeds

[0807] The server calculates the movement patterns and speeds of players and the ball from the coordinate data of successive frames, which allows visualization of each player's running distance and speed.

[0808] 4. Event Detection

[0809] The server automatically detects match events such as passes, shots, and goals based on the calculated coordinate and velocity data, and this event data is tagged and used for subsequent analysis.

[0810] 5. Skeletal data extraction and motion analysis

[0811] The server uses a skeletal detection algorithm such as OpenPose or MediaPipe to extract the skeletal data (joint positions) of each player, and analyzes the player's movements (e.g., jump shots and sliding).

[0812] 6. Generation and distribution of statistical data

[0813] The server combines the coordinate data and event data to generate statistical data for each player (e.g., pass success rate, number of successful shots, number of turnovers). The generated statistical data is delivered to the device in real time.

[0814] Terminal handling

[0815] 1. Data Receipt and Analysis

[0816] The terminal receives the real-time data sent from the server, analyzes the received data, and extracts the information required for display.

[0817] 2. Visualization and Notifications

[0818] The device visually displays an overview of the entire game and performance data for each player, and also notifies the user in real time when important events occur.

[0819] User operations

[0820] 1. Viewing real-time data

[0821] Users can check the progress of the match in real time through their devices, and can also view player performance data and match highlights.

[0822] 2. Detailed analysis and visualization

[0823] Users can select detailed analytical data for a specific period or player and view the data graphically visualized, providing insights for tactical analysis and performance improvement.

[0824] Specific examples

[0825] For example, in a basketball game, the server captures camera footage from the start of the game, and the AI ​​model updates the position data of each player and the ball every 0.1 seconds. If the server detects that player A passes the ball and player B receives it, it tags the action as a "pass" and increments the pass success count in real time. The device receives this information and visually displays detailed information about the pass to the user.

[0826] The above is the details of the mode for carrying out the invention. This system can be applied not only to basketball but also to soccer and other sports, so it can meet a variety of needs for sports data analysis.

[0827] The processing flow will be explained below.

[0828] Server Processing

[0829] Step 1:

[0830] The server receives video data from the overhead camera and tracking camera in real time, and the received video data is divided into frames.

[0831] Step 2:

[0832] The server performs pre-processing of the video data, including image pre-processing such as noise reduction, resolution adjustment, and color correction.

[0833] Step 3:

[0834] The server uses an object detection algorithm (e.g., YOLO or DeepSORT) to detect players and the ball for each frame, and extracts the position information of the detected players and the ball as coordinate data.

[0835] Step 4:

[0836] The server calculates the movement patterns of the players and the ball based on the coordinate data of successive frames, including the distance and speed of the players.

[0837] Step 5:

[0838] The server uses the calculated coordinate and velocity data to detect events during the match (e.g., passes, shots, goals), and tags the events.

[0839] Step 6:

[0840] The server uses a skeleton detection algorithm such as OpenPose or MediaPipe to extract the player's skeletal data, which includes the positions of each joint.

[0841] Step 7:

[0842] The server analyzes the player's movements based on the extracted skeletal data, identifying movements such as jump shots and sliding.

[0843] Step 8:

[0844] The server integrates the coordinate data and the event detection results to generate statistical data for each player (e.g., pass success rate, number of successful shots, number of turnovers).

[0845] Step 9:

[0846] The server delivers the generated statistical data to the device in real time, using standard formats such as JSON.

[0847] Terminal handling

[0848] Step 1:

[0849] The terminal receives real-time data sent from the server, and the received data is analyzed immediately.

[0850] Step 2:

[0851] Based on the analyzed data, the device displays a bird's-eye view of the entire match, along with visuals of each player's performance data.

[0852] Step 3:

[0853] The device will notify the user in real time when important match events occur, including pop-up messages and audio alerts.

[0854] Step 4:

[0855] When users click on a notification or select a specific player or time period, they are presented with an interface that displays detailed play data and highlights.

[0856] User operations

[0857] Step 1:

[0858] Users can check the progress of the match in real time through their devices, with overhead camera footage and analysis data displayed simultaneously.

[0859] Step 2:

[0860] Users can select a specific player's performance data and view detailed statistics, such as shooting percentage and distance traveled.

[0861] Step 3:

[0862] When a user selects an important event during a match (e.g., a goal or a foul), a replay of the corresponding play is displayed, allowing spectators to rewatch the important moments of the match.

[0863] Example 1

[0864] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0865] Existing sports data analysis systems have the problem of being unable to grasp the situation of a game in real time, and it takes a long time to analyze the detailed movements of players and the ball. Furthermore, because real-time notification functions for spectators are not fully developed, important moments of the game can be missed. Furthermore, there are issues with the accuracy and reliability of statistical data due to the inefficient fusion and analysis of data obtained from multiple cameras.

[0866] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0867] In this invention, the server includes a means for processing video data received from the overhead camera and tracking camera in real time, a means for extracting coordinate data of people and the ball from the video data using an object detection algorithm, and a means for detecting the movement patterns, speed, and events of people and the ball based on the coordinate data. This allows highly accurate coordinate data to be extracted from the video data in real time, enabling detailed analysis of the movements of players and the ball. Furthermore, by generating and distributing statistical data in real time, users can instantly grasp the situation of the game and watch without missing important moments. Furthermore, data obtained from multiple cameras is efficiently combined to provide highly reliable statistical data.

[0868] A "bird's-eye view camera" is a camera that takes pictures of the entire stadium from above.

[0869] A "tracking camera" is a camera that tracks a specific player or the ball in detail and takes high-resolution images.

[0870] "Real-time processing of video data" means instantly processing video data received from the overhead camera and tracking camera.

[0871] An "object detection algorithm" is an algorithm for identifying specific objects (such as players or the ball) from video data and extracting their location information.

[0872] "Coordinate data" is data that indicates the position of an object in an image, and is usually expressed as two-dimensional or three-dimensional coordinates.

[0873] A "movement pattern" indicates the movement path of an object calculated from continuous coordinate data.

[0874] "Velocity" indicates the amount of change in position per unit time when an object moves.

[0875] An "event" is a specific action or occurrence in a match (e.g., a pass, a shot, a goal) that is tagged when it is detected.

[0876] "Statistical Data" means statistical information about player or team performance that is generated based on extracted and analyzed data.

[0877] "Real-time delivery" refers to the process of instantly transmitting generated data to a user terminal.

[0878] "Skeletal data" is data that indicates the joint positions and structure of a person extracted from video data.

[0879] "Movement analysis" refers to the analysis of a person's specific movements using extracted skeletal data and coordinate data.

[0880] "Tactical analysis" is an analysis that evaluates tactics and playing style based on the results of motion analysis and identifies areas for improvement.

[0881] "Real-time notification" is a function that notifies the user of a specific event immediately when that event is detected.

[0882] A "user interface" is a display screen on a terminal that allows users to visually check the game situation and statistical data.

[0883] The system of the present invention analyzes sports data in real time to detect and analyze the detailed movements of players and the ball. This system uses an overhead camera and a tracking camera to collect detailed data on the entire game, players, and the ball, and automatically analyzes the data and provides it to the user. Specific embodiments of each element are described below.

[0884] Server configuration and processing

[0885] Video data reception and preprocessing

[0886] The server receives video data in real time from the overhead camera and tracking camera. The overhead camera is a high-resolution camera that captures the entire match venue from above, while the tracking camera is a camera that tracks players and the ball in detail. These cameras are connected to the server and transmit video streams in H.264 format. The server performs noise reduction (using Gaussian Blur, for example) and resolution adjustment on the received video data, and then divides it into the required frames.

[0887] Object detection and coordinate data extraction

[0888] The server uses an object detection algorithm (such as YOLO or DeepSORT) to detect players and the ball in the frame and extract their location information, which is then saved as 2D or 3D coordinate data.

[0889] Movement pattern and speed calculations

[0890] The server uses the coordinate data from successive frames to calculate the movement patterns and speeds of players and the ball, which then calculates the distance and speed of each player and stores them in a database.

[0891] Event detection and tagging

[0892] The server automatically detects events such as passes, shots, and goals under certain conditions based on coordinate and velocity data, and tags these events. The event data is stored in a database for subsequent analysis and report generation.

[0893] Skeletal data extraction and motion analysis

[0894] The server uses skeletal detection algorithms such as OpenPose and MediaPipe to extract the player's skeletal data (joint positions) and analyzes their movements (e.g., jump shots and sliding) based on this data.

[0895] Statistical data generation and distribution

[0896] The server integrates the coordinate data and event data to generate statistical data for each player (e.g., pass success rate, number of successful shots, number of turnovers). This statistical data is delivered to the device in real time using communication protocols such as WebSocket and REST API.

[0897] Terminal configuration and handling

[0898] Data reception and analysis

[0899] The device receives real-time data sent from the server, analyzes it, and extracts the necessary information, such as video frames and statistical data, which it stores in memory and reconstructs accordingly.

[0900] Visualization and Notifications

[0901] The device displays a bird's-eye view of the entire game and performance data for each player on the user interface, and notifies the user in real time via pop-up notifications when important events occur (e.g., successful passes or goals).

[0902] User operations

[0903] Viewing real-time data

[0904] Users can check the progress of the match in real time through their device, with match footage and player location information updated continuously, and players' performance data and match highlights visually displayed.

[0905] Detailed Analysis and Visualization

[0906] Users navigate through specific menus on their device to select detailed analytical data for specific periods or players, which are then visualized in graphs and charts to provide tactical analysis and insights for performance improvement.

[0907] Specific examples

[0908] For example, in a basketball game, the server captures video from the overhead camera and tracking camera from the start of the game and updates the position data of each player and the ball every 0.1 seconds using an object detection algorithm. If the server detects that player A passes the ball and player B receives it, it tags the action as a "pass" and increments the pass success count in real time. The device receives this information and visually displays the pass details to the user. For example, a notification pops up saying, "Pass from player A to player B was successful."

[0909] Example prompt for a generative AI model:

[0910] "In a basketball game, collect data on when player A successfully passes the ball to player B, and analyze it in real time."

[0911] The above is the detailed description of the mode for carrying out the invention. This system can be applied not only to basketball but also to soccer and other sports, and can meet a variety of needs for sports data analysis.

[0912] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0913] Server Processing

[0914] Step 1: Receiving and preprocessing video data

[0915] The server receives H.264 video data in real time from the overhead camera and tracking camera. The input is the video stream from the camera, and the output is preprocessed frame data. Specifically, Gaussian Blur is applied to remove noise, and the resolution is adjusted and divided into the required frames.

[0916] Step 2: Object detection and coordinate data extraction

[0917] The server applies an object detection algorithm (such as YOLO or DeepSORT) to the preprocessed frame data as input. The input is the preprocessed frame data, and the output is the coordinate data of the detected players and ball. For each frame, it detects specific objects from the pixel data and extracts their location information as 2D or 3D coordinates.

[0918] Step 3: Calculate movement patterns and speeds

[0919] The server inputs the coordinate data of successive frames and calculates the movement patterns and speeds. The input is the coordinate data for each frame, and the output is the movement patterns and speed data of the players and the ball. Specifically, the server calculates the speeds using the time intervals of the coordinate data and stores the results in a database.

[0920] Step 4: Event detection and tagging

[0921] The server automatically detects specific events (passes, shots, goals, etc.) in a match based on movement patterns and speed data. The input is coordinate data and speed data, and the output is the detected events and their tag data. As each event is detected, it is tagged and stored in a database.

[0922] Step 5: Extracting skeletal data and motion analysis

[0923] The server uses preprocessed frame data as input and performs skeletal detection using OpenPose or MediaPipe. The input is the preprocessed frame data, and the output is the player's skeletal data (joint positions). The extracted skeletal data is analyzed to recognize specific movements (e.g., jump shots and sliding).

[0924] Step 6: Generate and distribute statistical data

[0925] The server combines coordinate data, event data, and motion analysis data to generate statistical data for each player. The input is all of the above data, and the output is statistical data. This statistical data is delivered to the device in real time. WebSocket and REST API are used for delivery.

[0926] Terminal handling

[0927] Step 1: Receiving and analyzing data

[0928] The terminal receives real-time data sent from the server. The input is the real-time data from the server, and the output is the analyzed data required for display. The terminal analyzes the received data, stores e.g., video frames and statistical data in memory, and reconstructs them accordingly.

[0929] Step 2: Visualization and Notification

[0930] The device displays the received data in a user interface. The input is the analyzed data, and the output is a screen that displays an overview of the game and the performance of each player. When an important event occurs, the user is notified in real time by a pop-up notification.

[0931] User operations

[0932] Step 1: View real-time data

[0933] The user can check the progress of the game in real time through the device. The input is the display data from the device, and the output is the user's visual information. The progress of the game and the position information of the players are updated successively.

[0934] Step 2: Detailed analysis and visualization

[0935] Users operate specific menus on the device and select detailed analysis data. The input is the device's menu operations, and the output is the selected analysis data. The selected data is visualized in graphs and charts, providing insights for tactical analysis and performance improvement.

[0936] (Application example 1)

[0937] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0938] In conventional manufacturing sites, it has been difficult to monitor the operation of manufacturing machines in real time and immediately detect and notify abnormal operations. Furthermore, while precise operation analysis technology is required for efficient operation and quality control of manufacturing machines, there has been no system to achieve this. Furthermore, there has been a lack of means for workers to receive analysis results in real time on-site and respond quickly.

[0939] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0940] In this invention, the server includes an overhead camera, a tracking camera, means for processing video data received from the overhead camera and the tracking camera in real time, means for extracting coordinate data of people and objects from the video data, means for detecting movement patterns, speeds, and events of people and objects based on the coordinate data, means for generating statistical data based on the coordinate data and event detection results, means for detecting operation patterns of manufacturing machines and notifying abnormal operations in real time, and means for delivering the statistical data and abnormality detection results to terminals in real time.This enables real-time monitoring of the operation of manufacturing machines at a manufacturing site, immediate detection and notification of abnormal operations, and prompt response by workers on site.

[0941] A "bird's-eye view camera" is a camera that can capture a wide area from above.

[0942] A "tracking camera" is a high-resolution camera that can capture detailed images of a specific object while tracking it.

[0943] "Means for processing video data in real time" refers to technology that instantly processes video data collected from overhead cameras and tracking cameras.

[0944] "Means for extracting coordinate data" refers to technology that extracts the location information of specific people or objects from video data as numerical data.

[0945] "Means for detecting movement patterns, speed, and events" refers to technology that identifies the movement patterns and speed of objects, as well as specific events, based on the extracted coordinate data.

[0946] The "means for generating statistical data" refers to a technique for creating various statistical information using coordinate data and event detection results.

[0947] "Means for notifying abnormal operation in real time" refers to technology that monitors the operation of manufacturing machines and immediately notifies if an abnormality is detected.

[0948] "Means for delivering to terminals in real time" refers to technology that instantly transmits generated statistical data and anomaly detection results to the appropriate terminal.

[0949] The "means for detecting the operation pattern of a manufacturing machine" refers to a technology that monitors the operation of a manufacturing machine and identifies the characteristics and patterns of its movement.

[0950] "Means for conducting tactical analysis based on the results of motion analysis" refers to a technique for conducting detailed analysis from a tactical perspective based on extracted motion data.

[0951] "Means for providing real-time notifications to workers based on data on abnormal operations detected by the abnormality detection means" refers to technology that immediately notifies workers of details of an abnormality when an abnormality is detected.

[0952] This invention is a system that uses a bird's-eye view camera and a tracking camera to monitor and analyze the operation of manufacturing machines in a manufacturing site. The system is composed of a server, a terminal, and a user.

[0953] The server processes video data collected from the overhead camera and tracking camera in real time. The hardware used includes a high-resolution overhead camera, tracking camera, and server. The software includes OpenCV for Python, FFmpeg, YOLO, DeepSORT, and TensorFlow. The server removes noise from the received video data, adjusts the resolution, and divides it into frames. It then uses YOLO and DeepSORT to extract coordinate data for the manufacturing machine and work objects from the video data. The movement pattern and speed of the manufacturing machine are calculated from the coordinate data of consecutive frames.

[0954] The server detects abnormal behavior and patterns and generates anomaly detection results and statistical data, including the operational efficiency, operation history, and frequency of failures of the manufacturing machines.

[0955] The server also delivers anomaly detection results and statistical data in real time to the terminal. The terminal is a pair of smart glasses worn by the worker. The terminal uses Three.js and WebGL to visualize and display the received data in 3D space. If a specific anomaly is detected, the terminal notifies the worker in real time.

[0956] The user, a worker, can visually check the operation data of the manufacturing machine in real time through the smart glasses, which allows them to respond immediately if an abnormality occurs.

[0957] As a concrete example, consider a scenario in which the operation of a robot arm in a production line is monitored. An overhead camera monitors the entire production line, while a tracking camera tracks the movement of the robot arm. The server analyzes this video data and determines whether the robot arm is operating normally or if any abnormalities have occurred. If an abnormality is detected, a worker wearing smart glasses is immediately notified, enabling a prompt response.

[0958] An example prompt might look like this:

[0959] Monitor the movement patterns followed by specific manufacturing robots in your factory in real time, get instant notification if any abnormal behavior is detected, view the coordinate data and movement speed of each robot in 2D and 3D, and store a history of any abnormal behavior.

[0960] This system will significantly improve quality control and efficiency at the manufacturing site, and enable rapid response when abnormalities occur.

[0961] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0962] Step 1:

[0963] The server receives video data from the overhead camera and the tracking camera in real time.

[0964] Specific operation: The server receives the video signal from the camera and divides it into frames. The input is the video data captured by the camera, and the output is the data of each frame with noise removal and resolution adjustment. Preprocessing is performed using OpenCV for Python.

[0965] Step 2:

[0966] The server extracts coordinate data of the manufacturing machine and the work object from the pre-processed frame data.

[0967] Specific operation: Using YOLO and DeepSORT, we detect manufacturing machines and objects in the frame and obtain their coordinate data. The input is each preprocessed frame, and the output is the 2D coordinate data of the detected manufacturing machines and objects.

[0968] Step 3:

[0969] The server calculates the movement patterns and speeds of the manufacturing machines and objects based on the coordinate data between successive frames.

[0970] Specific operation: The server uses TensorFlow to analyze the acquired coordinate data and calculate the movement distance and speed. The input is the detected coordinate data, and the output is the calculated movement pattern and speed data.

[0971] Step 4:

[0972] The server detects abnormal behavior based on movement patterns and speed data.

[0973] Specific operation: The server compares the accumulated movement pattern and speed data to determine whether there are any abnormalities. In this step, an algorithm is used to instantly detect abnormal behavior. Its input is the movement pattern and speed data, and its output is the data of the detected abnormal behavior.

[0974] Step 5:

[0975] The server notifies the worker in real time if any abnormal operation is detected.

[0976] Specific operation: When an abnormality is detected, the server immediately sends an abnormal operation notification to the worker's smart glasses using Socket.IO. The input is the detected abnormal operation data, and the output is a real-time notification.

[0977] Step 6:

[0978] The terminal visualizes abnormal behavior data and statistical data received from the server.

[0979] How it works: The smart glasses use Three.js and WebGL to visually display the received data in 3D space. Its input is abnormal behavior data and statistical data from the server, and its output is a visualized interface.

[0980] Step 7:

[0981] Using smart glasses, users can check the operation of manufacturing machines and receive abnormality notifications in real time and respond accordingly.

[0982] Specific Action: The user checks the display of the smart glasses and takes action if necessary. In this step, the input received by the user is the display data of the smart glasses, and the output is the corresponding action.

[0983] This series of processes makes it possible to monitor the operation of manufacturing machines at the manufacturing site in real time and to immediately notify workers if an abnormality occurs.

[0984] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0985] The system of the present invention uses a bird's-eye view camera and a tracking camera to collect detailed data on the entire game, players, and the ball in real time, and automatically analyzes and compiles it. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it aims to improve the user experience. Below, a specific embodiment of each processing step is explained.

[0986] This system begins with the placement of overhead cameras and tracking cameras. The overhead cameras are installed in a position that allows for a bird's-eye view of the entire stadium. Meanwhile, the tracking cameras are high-resolution cameras that are placed at key points in the game to track specific players or the ball in detail. These cameras are connected to a server and transmit video data in real time.

[0987] Server Processing

[0988] 1. Video data reception and preprocessing

[0989] The server receives video data from the overhead camera and tracking camera in real time, and the received video data is divided into frames.

[0990] 2. Object detection and coordinate data extraction

[0991] The server uses an object detection algorithm (e.g., YOLO or DeepSORT) to detect players and the ball in each frame and extract their coordinate data, which represents the positional information of players and the ball in two or three dimensions.

[0992] 3. Calculating movement patterns and speeds

[0993] The server calculates the movement patterns and speeds of players and the ball from the coordinate data of successive frames, which allows visualization of each player's running distance and speed.

[0994] 4. Event Detection

[0995] The server uses the calculated coordinate and velocity data to detect events during the match (e.g., passes, shots, goals), and tags the events.

[0996] 5. Skeletal data extraction and motion analysis

[0997] The server uses a skeletal detection algorithm such as OpenPose or MediaPipe to extract the player's skeletal data (joint positions), which is then used to analyze the player's movements (e.g., jump shots and sliding).

[0998] 6. Generating and distributing statistical data

[0999] The server combines the coordinate data and event detection results to generate statistical data for each player (e.g., pass success rate, number of successful shots, number of turnovers). The generated statistical data is delivered to the device in real time.

[1000] 7. Emotion Recognition with Emotion Engine

[1001] The server analyzes the user's real-time video and audio data using an emotion engine to recognize the user's emotional state (e.g., excitement, stress, interest).

[1002] 8. Integration and Utilization of Emotional Data

[1003] The server integrates the emotion data generated by the emotion engine with statistical data and event data, generates match highlights and other interesting content based on the user's interests and reactions, and delivers them to the device.

[1004] Terminal handling

[1005] 1. Data Receipt and Analysis

[1006] The device receives statistical data and emotion data sent from the server in real time, and the received data is analyzed immediately.

[1007] 2. Visualization and Notifications

[1008] The device visually displays an overview of the entire game and performance data for each player. It also selectively displays interesting highlights and specific plays based on the user's emotions detected by an emotion engine. Furthermore, it notifies the user in real time when important game events occur.

[1009] User operations

[1010] 1. Viewing real-time data

[1011] Users can check the progress of the match in real time through their devices, with overhead camera footage and analysis data displayed simultaneously.

[1012] 2. Content display based on detailed analysis and emotion recognition

[1013] Users can select and view the performance data of specific players or match highlights, and the emotion engine displays content that reflects the user's emotional data, improving the user experience.

[1014] Specific examples

[1015] For example, in a soccer match, the server captures camera footage from the start of the match, and the AL model updates the position data of each player and the ball every 0.1 seconds. If the server detects that player A passes the ball and player B receives it, it tags the action as a "pass" and increments the number of successful passes in real time. If the emotion engine recognizes the user's excitement, the device will visually display highlights of the play according to the user's excitement level.

[1016] The system will enable spectators to feel more at home in the match, gain detailed insights based on player performance analysis, and display content based on emotional data, providing a more personalized viewing experience.

[1017] As described above, the system of the present invention aims to improve the user experience by integrating multi-camera video analysis and emotion recognition technology to realize automatic analysis of match data and real-time notification.

[1018] The processing flow will be explained below.

[1019] Server Processing

[1020] Step 1:

[1021] The server receives video data from the overhead camera and tracking camera in real time and divides the received video data into frames.

[1022] Step 2:

[1023] The server performs pre-processing of the video data, specifically image pre-processing such as noise removal, resolution adjustment, and color correction.

[1024] Step 3:

[1025] The server uses an object detection algorithm (e.g., YOLO or DeepSORT) to detect players and the ball in each frame, and extracts the position information of the detected players and the ball as coordinate data.

[1026] Step 4:

[1027] The server calculates the movement patterns of the players and the ball based on the coordinate data of successive frames, including the distance and speed of the players.

[1028] Step 5:

[1029] The server automatically detects events (e.g., passes, shots, goals) during a match based on the calculated coordinate and velocity data, and tags the detected events.

[1030] Step 6:

[1031] The server uses a skeletal detection algorithm such as OpenPose or MediaPipe to extract skeletal data for each player, including the position information of each joint.

[1032] Step 7:

[1033] The server analyzes the player's movements based on the extracted skeletal data, identifying movements such as jump shots and sliding.

[1034] Step 8:

[1035] The server integrates the coordinate data and the event detection results to generate statistical data for each player (e.g., pass success rate, number of successful shots, number of turnovers).

[1036] Step 9:

[1037] The server delivers the generated statistical data to the device in real time, using standard formats such as JSON.

[1038] Step 10:

[1039] The server analyzes real-time video and audio data acquired from the user using an emotion engine to recognize the user's emotional state (e.g., excitement, stress, interest).

[1040] Step 11:

[1041] The server integrates the emotion data obtained by the emotion engine with statistical and event data, and generates game highlights and other engaging content based on the user's interests and reactions.

[1042] Step 12:

[1043] The server delivers content based on the generated emotion data to the device in real time, which is displayed as a visual feed that reflects the user's emotional state.

[1044] Terminal handling

[1045] Step 1:

[1046] The device receives statistical data and emotional data sent from the server in real time, and the received data is analyzed immediately.

[1047] Step 2:

[1048] Based on the analyzed data, the device visually displays an overview of the entire match and performance data for each player, including the player's movement patterns and speed.

[1049] Step 3:

[1050] The terminal selectively displays interesting highlights or specific plays based on the user's emotional state detected by the emotion engine, thereby providing content that attracts the user's interest.

[1051] Step 4:

[1052] The device will notify the user in real time when important match events occur, including pop-up messages and audio alerts.

[1053] User operations

[1054] Step 1:

[1055] Users can check the progress of the match in real time through their devices, with overhead camera footage and analysis data displayed simultaneously.

[1056] Step 2:

[1057] Users can select a specific player's performance data and view detailed statistics, such as shooting percentage and distance traveled.

[1058] Step 3:

[1059] The device will display highlights and detailed information related to the user's favorite players and plays of interest based on the user's emotional data recognized by the emotion engine, providing a more personalized viewing experience.

[1060] Example 2

[1061] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1062] Conventional match analysis systems can acquire player and ball position data and detect events based on that data. However, these systems do not take into account user emotional data, which means they cannot provide content optimized for each spectator. Furthermore, there is a lack of technology to perform detailed tactical analysis in real time, making it difficult to enhance the sense of realism in live match viewing.

[1063] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1064] In this invention, the server includes an overhead camera, a tracking camera, a means for processing video data received from the overhead camera and the tracking camera in real time, a means for extracting object coordinate data from the video data, a means for calculating the object's movement pattern and speed based on the coordinate data, a means for detecting events from the coordinate data and speed data, a means for generating statistical data based on the coordinate data and the event detection results, a means for distributing the statistical data to a terminal in real time, a means for recognizing a user's emotion, and a means for integrating the emotion data with the statistical data and event data to generate individually customized content. This enables the provision of personalized content that reflects the user's emotion, achieving a more realistic viewing experience. Furthermore, detailed tactical analysis can be performed in real time, enabling the provision of more accurate data.

[1065] A "bird's-eye view camera" is a camera that is installed in a position that allows it to overlook the entire venue from above and captures video data over a wide area.

[1066] A "tracking camera" is a high-resolution camera installed to closely track a specific player or the ball.

[1067] The "means for processing video data in real time" is a function that includes a process for dividing video data received from the overhead camera and tracking camera and extracting objects and coordinates.

[1068] "Means for extracting coordinate data" refers to an algorithm or system for obtaining position information of objects (such as players or the ball) from video data.

[1069] The "means for calculating the movement pattern and speed" is a function for calculating the movement path and speed of an object based on the coordinate data of successive frames.

[1070] The "event detection means" is a system that identifies important actions during a match (e.g., passes, shots, goals) based on movement patterns and speed data.

[1071] The "means for generating statistical data" is a function that integrates coordinate data and event detection results to calculate detailed statistical information for each player (e.g., pass success rate, number of successful shots).

[1072] The "means for delivering to terminals in real time" refers to a process and system for instantly transmitting the generated statistical data to user terminals.

[1073] The "means for recognizing user emotions" is a technology that analyzes the user's facial expressions and vocal tone to identify emotional states such as excitement, stress, interest, etc.

[1074] "Means for integrating emotional data with statistical data and event data to generate individually customized content" refers to a system that combines recognized user emotional data with other match data to create content (e.g., highlights) that is tailored to each individual user.

[1075] The system of this invention uses a bird's-eye view camera and a tracking camera to collect detailed data on the entire game, players, and the ball in real time, and automatically analyzes and compiles it.Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it aims to improve the user experience.

[1076] The system begins with the placement of overhead cameras and tracking cameras. The overhead cameras are installed in a position that allows for a bird's-eye view of the entire stadium, while the tracking cameras are high-resolution cameras that are placed at key points in the game to track specific players or the ball in detail. These cameras are connected to a server and transmit video data in real time.

[1077] Server Processing

[1078] The server first receives video data in real time from the overhead camera and tracking camera. The received video data is divided into frames, and an object detection algorithm (e.g., YOLO or DeepSORT) is used to detect players and the ball in each frame and extract their coordinate data. This coordinate data represents the positional information of players and the ball in two or three dimensions.

[1079] The server calculates the movement patterns and speeds of players and the ball from the coordinate data of consecutive frames. This allows the running distance and speed of each player to be visualized. The calculated coordinate and speed data are then used to detect events during the game (e.g., passes, shots, goals). Detected events are tagged. Furthermore, a skeleton detection algorithm such as OpenPose or MediaPipe is used to extract players' skeletal data (joint positions). Based on this skeletal data, players' movements (e.g., jump shots and sliding) are analyzed.

[1080] The server combines the generated coordinate data and event detection results to generate statistical data for each player (e.g., pass success rate, number of successful shots, number of turnovers). This statistical data is delivered to the device in real time. In addition, the server analyzes the user's real-time video and audio data using an emotion engine to recognize the user's emotional state (e.g., excitement, stress, interest). The server combines the emotion data generated by the emotion engine with the statistical data and event data to generate game highlights and other interesting content based on the user's interests and reactions, and delivers it to the device.

[1081] Terminal handling

[1082] The device receives statistical and emotional data sent from the server in real time and analyzes it instantly. The device visually displays an overview of the entire game and performance data for each player. It also selectively displays interesting highlights and specific plays based on the user's emotions detected by the emotion engine. Furthermore, when important game events occur, the device notifies the user in real time.

[1083] User operations

[1084] Users can check the progress of the game in real time through their devices. Images from a bird's-eye view camera and analytical data are displayed simultaneously, and they can select and check the performance data of specific players or highlights of the game. An emotion engine displays content that reflects the user's emotional data, improving the user experience.

[1085] Specific examples

[1086] For example, in a soccer match, the server captures camera footage from the start of the match, and the AI ​​model updates the position data of each player and the ball every 0.1 seconds. If the server detects that player A passes the ball and player B receives it, it tags the action as a "pass" and increments the number of successful passes in real time. If the emotion engine recognizes the user's excitement, the device will visually display highlights of the play according to the user's excitement level.

[1087] Prompt Sentence Examples

[1088] "Analyze the number of passes and ball possession time of Player A in a soccer match in real time, and display highlights according to the user's excitement level."

[1089] The system will enable spectators to feel more at home in the match, gain detailed insights based on player performance analysis, and display content based on emotional data, providing a more personalized viewing experience.

[1090] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1091] Step 1: Receiving video data from the overhead camera and tracking camera

[1092] The server receives video data in real time from the overhead camera and tracking camera. This video data is divided into frames at high resolution. The input is the video data sent from the camera, and the output is the video data for each frame stored in the server.

[1093] Step 2: Splitting and Preprocessing Frames

[1094] The server divides the received video data into frames and performs preprocessing, which includes noise removal and frame alignment. The input is video data for each frame, and the output is the preprocessed frame data.

[1095] Step 3: Object detection and coordinate data extraction

[1096] The server applies an object detection algorithm (such as YOLO or DeepSORT) to the preprocessed frame data to extract the coordinate data of players and the ball. The input is the preprocessed frame data, and the output is the coordinate data (e.g., x-coordinate, y-coordinate) of players and the ball in each frame.

[1097] Step 4: Calculate movement patterns and speeds

[1098] The server uses the coordinate data of consecutive frames to calculate the movement patterns and speeds of players and the ball. For example, if player A is located at x=100, y=200 in frame 1 and moves to x=120, y=220 in frame 2, the server calculates the movement distance and speed between them. The input is the coordinate data of consecutive frames, and the output is the movement pattern and speed data.

[1099] Step 5: Detecting an event

[1100] The server analyzes movement pattern and speed data to detect events (e.g., passes, shots, goals) during a match. For example, if it detects that player A passes the ball and player B receives it, it tags this action as a "pass." The input is movement pattern and speed data, and the output is event-tagged data.

[1101] Step 6: Extracting skeletal data and motion analysis

[1102] The server uses OpenPose and MediaPipe to extract the player's skeletal data. Then, it analyzes the player's movements (e.g., jumping, sliding) based on this skeletal data. The input is frame data and coordinate data, and the output is skeletal data and the results of the movement analysis.

[1103] Step 7: Generate and distribute statistical data

[1104] The server integrates the coordinate data, event detection results, and motion analysis results to generate statistical data for each player (e.g., pass success rate, number of successful shots). The generated statistical data is delivered to the device in real time. The input is coordinate data, event data, and motion analysis results, and the output is statistical data.

[1105] Step 8: Emotion Recognition with the Emotion Engine

[1106] The server analyzes the user's real-time video and audio data using an emotion engine to recognize the user's emotional state (e.g., excitement, stress, interest). The input is the user's video and audio data, and the output is the user's emotional data.

[1107] Step 9: Integrate and utilize emotion data

[1108] The server integrates the emotion data generated by the emotion engine with statistical data and event data to generate match highlights and engaging content based on the user's interests and reactions. The generated content is delivered to the device. The inputs are statistical data, event data, and emotion data, and the output is customized highlights and content.

[1109] (Application example 2)

[1110] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1111] Traditional sports viewing systems limit the amount of information spectators can obtain in real time, making it difficult for them to gain a deep understanding of the game's movements and tactics. Furthermore, they lack a mechanism for automatically displaying content that matches the spectator's interests and emotional state, resulting in a lack of a personalized viewing experience for each spectator. This often results in a uniform and uninteresting viewing experience for spectators.

[1112] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for processing video data received from the overhead camera and tracking camera in real time, means for extracting coordinate data of people and the ball from the video data, means for detecting the movement patterns, speed, and events of people and the ball based on the coordinate data, means for integrating the statistical data and emotional data, generating game highlights and interesting content based on the user's interests and reactions, and delivering the content to the terminal in real time, and means for recognizing the user's emotional state using an emotion engine. This allows spectators to not only grasp detailed game movements and tactical analysis in real time, but also enjoy personalized content tailored to their individual emotional states.

[1113] A "bird's-eye view camera" is a camera installed in a position that allows it to overlook the entire venue from above, and is a device that captures footage of the entire match.

[1114] A "tracking camera" is a high-resolution camera that is placed at key points in the game to track specific players or the ball in detail.

[1115] "Video data" refers to digital video data received from the overhead camera and tracking camera and processed in real time.

[1116] "Coordinate data" refers to data that indicates the position information of people and balls extracted from video data, and is expressed in two or three dimensions.

[1117] A "movement pattern" indicates the movement path of a person and a ball calculated from continuous coordinate data.

[1118] An "event" is a specific action or occurrence that occurs during a match (e.g., a pass, a shot, a goal) and is tagged accordingly.

[1119] "Statistical data" refers to data generated based on coordinate data and event detection results, and includes performance information for each player (e.g., pass success rate, number of successful shots).

[1120] The "emotion engine" is a system that analyzes the user's real-time video and audio data to recognize the user's emotional state (e.g., excitement, stress, interest).

[1121] "Emotion Data" refers to data that indicates the user's emotional state as analyzed and recognized by the emotion engine.

[1122] "Highlights" is content that highlights important scenes from the game or parts that will interest spectators.

[1123] "User experience" refers to the overall experience a user has through a system, including operability and satisfaction.

[1124] To implement this invention, the following system is required: This system collects video data in real time using a bird's-eye view camera and a tracking camera, and combines it with an emotion engine that recognizes the user's emotions to improve the user experience.

[1125] Hardware Configuration

[1126] 1. Bird's-eye view camera: A camera installed in a position that allows a bird's-eye view of the entire venue. Specifically, general-purpose cameras such as Arducam can be used.

[1127] 2. Tracking camera: A high-resolution camera for tracking a specific player or the ball in detail. For example, a Basler camera can be used.

[1128] 3. Head-mounted display: A device that allows spectators to virtually watch the game. An example of this is the Oculus Quest 2.

[1129] Software Configuration

[1130] 1. Object detection algorithm: An algorithm that extracts coordinate data of people and spheres from video data acquired from overhead cameras and tracking cameras. Specifically, YOLO (You Only Look Once) and DeepSORT can be used.

[1131] 2. Skeleton detection algorithm: An algorithm for analyzing human skeletal data. Examples of algorithms that can be used include OpenPose and MediaPipe.

[1132] 3. Emotion engine: A system that analyzes real-time video and audio data of the user to recognize their emotional state. For example, Affectiva's emotion recognition API can be used.

[1133] 4. 3D modeling software: Software for visualizing the events during the match in 3D. For example, Unity can be used.

[1134] 5. Image processing library: A library for processing camera images. OpenCV can be used as a specific example.

[1135] Data calculation and processing

[1136] The server processes video data received from the overhead camera and tracking camera in real time. First, it divides the video data into frames and uses an object detection algorithm to detect people and the ball in each frame and extract their coordinate data. Next, it calculates the movement patterns and speed of people and the ball from the coordinate data of consecutive frames to detect events during the game. This data is integrated and delivered to the device in real time. It also uses an emotion engine to recognize the user's emotions, and combines the emotion data with statistical data to generate game highlights and interesting content tailored to the user, which is delivered to the device.

[1137] Specific examples

[1138] For example, during a soccer match, the server captures video from an overhead camera and a tracking camera, and the AL model updates the position data of each player and the ball every 0.1 seconds. If the server detects that player A passes the ball and player B receives it, it tags the action as a "pass" and increments the number of successful passes in real time. If the emotion engine recognizes the user's excitement, the device will visually display highlights of the play according to the user's excitement level.

[1139] Prompt Sentence Examples

[1140] A scenario where you are watching a soccer match in real time. Accurately track player movements and ball position and highlight key plays. Automatically generate and present relevant highlights based on the user's changing interests and emotions. The emotion engine should recognize the user's emotions and provide personalized match analysis and notifications based on those emotions.

[1141] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1142] Step 1:

[1143] The server receives video data from the overhead camera and tracking camera in real time and splits it into frames. The input is the video stream from the camera, and the output is the split frame data. This processing is performed using OpenCV.

[1144] Step 2:

[1145] The server applies an object detection algorithm (such as YOLO or DeepSORT) to each divided frame to detect players and the ball. The input is the frame data, and the output is the coordinate data of the detected objects (players and the ball). At this time, the coordinate data is recorded along two or three dimensions.

[1146] Step 3:

[1147] The server calculates the movement patterns and speeds of players and the ball based on coordinate data extracted from consecutive frames. The input is a continuous set of coordinate data, and the output is movement pattern data and speed data. In this calculation, the speed is calculated based on the rate of change of position information.

[1148] Step 4:

[1149] The server detects events (e.g., passes, shots, goals) during a match based on movement patterns and velocity data. The input is movement patterns and velocity data, and the output is event-tagged data. The event detection algorithm identifies events based on specific movement actions and velocity changes.

[1150] Step 5:

[1151] The server uses an object detection algorithm (OpenPose or MediaPipe) to extract the player's skeletal data. The input is a video frame, and the output is the player's skeletal coordinate data. This data is recorded as joint position information.

[1152] Step 6:

[1153] The server performs motion and tactical analysis of players based on their skeletal and coordinate data. The input is skeletal and coordinate data, and the output is motion analysis data and tactical analysis data. This processing identifies specific motion patterns and visualizes tactical performance.

[1154] Step 7:

[1155] The server generates statistics for each player based on the statistical data and event detection results. The input is a series of event data and coordinate data, and the output is a statistical data feed. In this step, player performance indicators (e.g., pass success rate, shot success rate) are calculated.

[1156] Step 8:

[1157] The server uses an emotion engine to analyze the user's real-time video and audio data to recognize the user's emotional state. The input is the user's video and audio data, and the output is emotional data. This process is performed using Affectiva's emotion recognition API.

[1158] Step 9:

[1159] The server integrates the emotional data and statistical data to generate highlights and other engaging content based on the user's interests and reactions, and delivers them to the device in real time. The input is emotional data and statistical data, and the output is personalized highlight content. The device receives this data and displays it visually.

[1160] Step 10:

[1161] Users can check the progress of the game in real time through their devices and enjoy personalized content. The input is data received from the server, and the output is an improved user experience, allowing users to experience a truly immersive game experience.

[1162] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1163] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1164] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1165] [Fourth embodiment]

[1166] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1167] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1168] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1169] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1170] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1171] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1172] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1173] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1174] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1175] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1176] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1177] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1178] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1179] The system of the present invention uses a bird's-eye view camera and a tracking camera to collect detailed data on the entire game, players, and the ball in real time, and automatically analyzes and compiles it. Specific embodiments of each processing step are described below.

[1180] This system begins with the placement of overhead cameras and tracking cameras. The overhead cameras are installed in a position that allows for a bird's-eye view of the entire stadium. Meanwhile, the tracking cameras are high-resolution cameras that are placed at key points in the game to track specific players or the ball in detail. These cameras are connected to a server and transmit video data in real time.

[1181] Server Processing

[1182] 1. Video data reception and preprocessing

[1183] The server receives video data from the overhead camera and tracking camera. After preprocessing such as noise reduction and resolution adjustment, the video data is divided into frames.

[1184] 2. Object detection and coordinate data extraction

[1185] The server uses an object detection algorithm (e.g., YOLO or DeepSORT) to detect players and the ball in each frame and extract their coordinate data, which represent the positional information of players and the ball in two or three dimensions.

[1186] 3. Calculating movement patterns and speeds

[1187] The server calculates the movement patterns and speeds of players and the ball from the coordinate data of successive frames, which allows visualization of each player's running distance and speed.

[1188] 4. Event Detection

[1189] The server automatically detects match events such as passes, shots, and goals based on the calculated coordinate and velocity data, and this event data is tagged and used for subsequent analysis.

[1190] 5. Skeletal data extraction and motion analysis

[1191] The server uses a skeletal detection algorithm such as OpenPose or MediaPipe to extract the skeletal data (joint positions) of each player, and analyzes the player's movements (e.g., jump shots and sliding).

[1192] 6. Generation and distribution of statistical data

[1193] The server combines the coordinate data and event data to generate statistical data for each player (e.g., pass success rate, number of successful shots, number of turnovers). The generated statistical data is delivered to the device in real time.

[1194] Terminal handling

[1195] 1. Data Receipt and Analysis

[1196] The terminal receives the real-time data sent from the server, analyzes the received data, and extracts the information required for display.

[1197] 2. Visualization and Notifications

[1198] The device visually displays an overview of the entire game and performance data for each player, and also notifies the user in real time when important events occur.

[1199] User operations

[1200] 1. Viewing real-time data

[1201] Users can check the progress of the match in real time through their devices, and can also view player performance data and match highlights.

[1202] 2. Detailed analysis and visualization

[1203] Users can select detailed analytical data for a specific period or player and view the data graphically visualized, providing insights for tactical analysis and performance improvement.

[1204] Specific examples

[1205] For example, in a basketball game, the server captures camera footage from the start of the game, and the AI ​​model updates the position data of each player and the ball every 0.1 seconds. If the server detects that player A passes the ball and player B receives it, it tags the action as a "pass" and increments the pass success count in real time. The device receives this information and visually displays detailed information about the pass to the user.

[1206] The above is the details of the mode for carrying out the invention. This system can be applied not only to basketball but also to soccer and other sports, so it can meet a variety of needs for sports data analysis.

[1207] The processing flow will be explained below.

[1208] Server Processing

[1209] Step 1:

[1210] The server receives video data from the overhead camera and tracking camera in real time, and the received video data is divided into frames.

[1211] Step 2:

[1212] The server performs pre-processing of the video data, including image pre-processing such as noise reduction, resolution adjustment, and color correction.

[1213] Step 3:

[1214] The server uses an object detection algorithm (e.g., YOLO or DeepSORT) to detect players and the ball for each frame, and extracts the position information of the detected players and the ball as coordinate data.

[1215] Step 4:

[1216] The server calculates the movement patterns of the players and the ball based on the coordinate data of successive frames, including the distance and speed of the players.

[1217] Step 5:

[1218] The server uses the calculated coordinate and velocity data to detect events during the match (e.g., passes, shots, goals), and tags the events.

[1219] Step 6:

[1220] The server uses a skeleton detection algorithm such as OpenPose or MediaPipe to extract the player's skeletal data, which includes the positions of each joint.

[1221] Step 7:

[1222] The server analyzes the player's movements based on the extracted skeletal data, identifying movements such as jump shots and sliding.

[1223] Step 8:

[1224] The server integrates the coordinate data and the event detection results to generate statistical data for each player (e.g., pass success rate, number of successful shots, number of turnovers).

[1225] Step 9:

[1226] The server delivers the generated statistical data to the device in real time, using standard formats such as JSON.

[1227] Terminal handling

[1228] Step 1:

[1229] The terminal receives real-time data sent from the server, and the received data is analyzed immediately.

[1230] Step 2:

[1231] Based on the analyzed data, the device displays a bird's-eye view of the entire match, along with visuals of each player's performance data.

[1232] Step 3:

[1233] The device will notify the user in real time when important match events occur, including pop-up messages and audio alerts.

[1234] Step 4:

[1235] When users click on a notification or select a specific player or time period, they are presented with an interface that displays detailed play data and highlights.

[1236] User operations

[1237] Step 1:

[1238] Users can check the progress of the match in real time through their devices, with overhead camera footage and analysis data displayed simultaneously.

[1239] Step 2:

[1240] Users can select a specific player's performance data and view detailed statistics, such as shooting percentage and distance traveled.

[1241] Step 3:

[1242] When a user selects an important event during a match (e.g., a goal or a foul), a replay of the corresponding play is displayed, allowing spectators to rewatch the important moments of the match.

[1243] Example 1

[1244] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1245] Existing sports data analysis systems have the problem of being unable to grasp the situation of a game in real time, and it takes a long time to analyze the detailed movements of players and the ball. Furthermore, because real-time notification functions for spectators are not fully developed, important moments of the game can be missed. Furthermore, there are issues with the accuracy and reliability of statistical data due to the inefficient fusion and analysis of data obtained from multiple cameras.

[1246] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1247] In this invention, the server includes a means for processing video data received from the overhead camera and tracking camera in real time, a means for extracting coordinate data of people and the ball from the video data using an object detection algorithm, and a means for detecting the movement patterns, speed, and events of people and the ball based on the coordinate data. This allows highly accurate coordinate data to be extracted from the video data in real time, enabling detailed analysis of the movements of players and the ball. Furthermore, by generating and distributing statistical data in real time, users can instantly grasp the situation of the game and watch without missing important moments. Furthermore, data obtained from multiple cameras is efficiently combined to provide highly reliable statistical data.

[1248] A "bird's-eye view camera" is a camera that takes pictures of the entire stadium from above.

[1249] A "tracking camera" is a camera that tracks a specific player or the ball in detail and takes high-resolution images.

[1250] "Real-time processing of video data" means instantly processing video data received from the overhead camera and tracking camera.

[1251] An "object detection algorithm" is an algorithm for identifying specific objects (such as players or the ball) from video data and extracting their location information.

[1252] "Coordinate data" is data that indicates the position of an object in an image, and is usually expressed as two-dimensional or three-dimensional coordinates.

[1253] A "movement pattern" indicates the movement path of an object calculated from continuous coordinate data.

[1254] "Velocity" indicates the amount of change in position per unit time when an object moves.

[1255] An "event" is a specific action or occurrence in a match (e.g., a pass, a shot, a goal) that is tagged when it is detected.

[1256] "Statistical Data" means statistical information about player or team performance that is generated based on extracted and analyzed data.

[1257] "Real-time delivery" refers to the process of instantly transmitting generated data to a user terminal.

[1258] "Skeletal data" is data that indicates the joint positions and structure of a person extracted from video data.

[1259] "Movement analysis" refers to the analysis of a person's specific movements using extracted skeletal data and coordinate data.

[1260] "Tactical analysis" is an analysis that evaluates tactics and playing style based on the results of motion analysis and identifies areas for improvement.

[1261] "Real-time notification" is a function that notifies the user of a specific event immediately when that event is detected.

[1262] A "user interface" is a display screen on a terminal that allows users to visually check the game situation and statistical data.

[1263] The system of the present invention analyzes sports data in real time to detect and analyze the detailed movements of players and the ball. This system uses an overhead camera and a tracking camera to collect detailed data on the entire game, players, and the ball, and automatically analyzes the data and provides it to the user. Specific embodiments of each element are described below.

[1264] Server configuration and processing

[1265] Video data reception and preprocessing

[1266] The server receives video data in real time from the overhead camera and tracking camera. The overhead camera is a high-resolution camera that captures the entire match venue from above, while the tracking camera is a camera that tracks players and the ball in detail. These cameras are connected to the server and transmit video streams in H.264 format. The server performs noise reduction (using Gaussian Blur, for example) and resolution adjustment on the received video data, and then divides it into the required frames.

[1267] Object detection and coordinate data extraction

[1268] The server uses an object detection algorithm (such as YOLO or DeepSORT) to detect players and the ball in the frame and extract their location information, which is then saved as 2D or 3D coordinate data.

[1269] Movement pattern and speed calculations

[1270] The server uses the coordinate data from successive frames to calculate the movement patterns and speeds of players and the ball, which then calculates the distance and speed of each player and stores them in a database.

[1271] Event detection and tagging

[1272] The server automatically detects events such as passes, shots, and goals under certain conditions based on coordinate and velocity data, and tags these events. The event data is stored in a database for subsequent analysis and report generation.

[1273] Skeletal data extraction and motion analysis

[1274] The server uses skeletal detection algorithms such as OpenPose and MediaPipe to extract the player's skeletal data (joint positions) and analyzes their movements (e.g., jump shots and sliding) based on this data.

[1275] Statistical data generation and distribution

[1276] The server integrates the coordinate data and event data to generate statistical data for each player (e.g., pass success rate, number of successful shots, number of turnovers). This statistical data is delivered to the device in real time using communication protocols such as WebSocket and REST API.

[1277] Terminal configuration and handling

[1278] Data reception and analysis

[1279] The device receives real-time data sent from the server, analyzes it, and extracts the necessary information, such as video frames and statistical data, which it stores in memory and reconstructs accordingly.

[1280] Visualization and Notifications

[1281] The device displays a bird's-eye view of the entire game and performance data for each player on the user interface, and notifies the user in real time via pop-up notifications when important events occur (e.g., successful passes or goals).

[1282] User operations

[1283] Viewing real-time data

[1284] Users can check the progress of the match in real time through their device, with match footage and player location information updated continuously, and players' performance data and match highlights visually displayed.

[1285] Detailed Analysis and Visualization

[1286] Users navigate through specific menus on their device to select detailed analytical data for specific periods or players, which are then visualized in graphs and charts to provide tactical analysis and insights for performance improvement.

[1287] Specific examples

[1288] For example, in a basketball game, the server captures video from the overhead camera and tracking camera from the start of the game and updates the position data of each player and the ball every 0.1 seconds using an object detection algorithm. If the server detects that player A passes the ball and player B receives it, it tags the action as a "pass" and increments the pass success count in real time. The device receives this information and visually displays the pass details to the user. For example, a notification pops up saying, "Pass from player A to player B was successful."

[1289] Example prompt for a generative AI model:

[1290] "In a basketball game, collect data on when player A successfully passes the ball to player B, and analyze it in real time."

[1291] The above is the detailed description of the mode for carrying out the invention. This system can be applied not only to basketball but also to soccer and other sports, and can meet a variety of needs for sports data analysis.

[1292] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1293] Server Processing

[1294] Step 1: Receiving and preprocessing video data

[1295] The server receives H.264 video data in real time from the overhead camera and tracking camera. The input is the video stream from the camera, and the output is preprocessed frame data. Specifically, Gaussian Blur is applied to remove noise, and the resolution is adjusted and divided into the required frames.

[1296] Step 2: Object detection and coordinate data extraction

[1297] The server applies an object detection algorithm (such as YOLO or DeepSORT) to the preprocessed frame data as input. The input is the preprocessed frame data, and the output is the coordinate data of the detected players and ball. For each frame, it detects specific objects from the pixel data and extracts their location information as 2D or 3D coordinates.

[1298] Step 3: Calculate movement patterns and speeds

[1299] The server inputs the coordinate data of successive frames and calculates the movement patterns and speeds. The input is the coordinate data for each frame, and the output is the movement patterns and speed data of the players and the ball. Specifically, the server calculates the speeds using the time intervals of the coordinate data and stores the results in a database.

[1300] Step 4: Event detection and tagging

[1301] The server automatically detects specific events (passes, shots, goals, etc.) in a match based on movement patterns and speed data. The input is coordinate data and speed data, and the output is the detected events and their tag data. As each event is detected, it is tagged and stored in a database.

[1302] Step 5: Extracting skeletal data and motion analysis

[1303] The server uses preprocessed frame data as input and performs skeletal detection using OpenPose or MediaPipe. The input is the preprocessed frame data, and the output is the player's skeletal data (joint positions). The extracted skeletal data is analyzed to recognize specific movements (e.g., jump shots and sliding).

[1304] Step 6: Generate and distribute statistical data

[1305] The server combines coordinate data, event data, and motion analysis data to generate statistical data for each player. The input is all of the above data, and the output is statistical data. This statistical data is delivered to the device in real time. WebSocket and REST API are used for delivery.

[1306] Terminal handling

[1307] Step 1: Receiving and analyzing data

[1308] The terminal receives real-time data sent from the server. The input is the real-time data from the server, and the output is the analyzed data required for display. The terminal analyzes the received data, stores e.g., video frames and statistical data in memory, and reconstructs them accordingly.

[1309] Step 2: Visualization and Notification

[1310] The device displays the received data in a user interface. The input is the analyzed data, and the output is a screen that displays an overview of the game and the performance of each player. When an important event occurs, the user is notified in real time by a pop-up notification.

[1311] User operations

[1312] Step 1: View real-time data

[1313] The user can check the progress of the game in real time through the device. The input is the display data from the device, and the output is the user's visual information. The progress of the game and the position information of the players are updated successively.

[1314] Step 2: Detailed analysis and visualization

[1315] Users operate specific menus on the device and select detailed analysis data. The input is the device's menu operations, and the output is the selected analysis data. The selected data is visualized in graphs and charts, providing insights for tactical analysis and performance improvement.

[1316] (Application example 1)

[1317] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1318] In conventional manufacturing sites, it has been difficult to monitor the operation of manufacturing machines in real time and immediately detect and notify abnormal operations. Furthermore, while precise operation analysis technology is required for efficient operation and quality control of manufacturing machines, there has been no system to achieve this. Furthermore, there has been a lack of means for workers to receive analysis results in real time on-site and respond quickly.

[1319] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1320] In this invention, the server includes an overhead camera, a tracking camera, means for processing video data received from the overhead camera and the tracking camera in real time, means for extracting coordinate data of people and objects from the video data, means for detecting movement patterns, speeds, and events of people and objects based on the coordinate data, means for generating statistical data based on the coordinate data and event detection results, means for detecting operation patterns of manufacturing machines and notifying abnormal operations in real time, and means for delivering the statistical data and abnormality detection results to terminals in real time.This enables real-time monitoring of the operation of manufacturing machines at a manufacturing site, immediate detection and notification of abnormal operations, and prompt response by workers on site.

[1321] A "bird's-eye view camera" is a camera that can capture a wide area from above.

[1322] A "tracking camera" is a high-resolution camera that can capture detailed images of a specific object while tracking it.

[1323] "Means for processing video data in real time" refers to technology that instantly processes video data collected from overhead cameras and tracking cameras.

[1324] "Means for extracting coordinate data" refers to technology that extracts the location information of specific people or objects from video data as numerical data.

[1325] "Means for detecting movement patterns, speed, and events" refers to technology that identifies the movement patterns and speed of objects, as well as specific events, based on the extracted coordinate data.

[1326] The "means for generating statistical data" refers to a technique for creating various statistical information using coordinate data and event detection results.

[1327] "Means for notifying abnormal operation in real time" refers to technology that monitors the operation of manufacturing machines and immediately notifies if an abnormality is detected.

[1328] "Means for delivering to terminals in real time" refers to technology that instantly transmits generated statistical data and anomaly detection results to the appropriate terminal.

[1329] The "means for detecting the operation pattern of a manufacturing machine" refers to a technology that monitors the operation of a manufacturing machine and identifies the characteristics and patterns of its movement.

[1330] "Means for conducting tactical analysis based on the results of motion analysis" refers to a technique for conducting detailed analysis from a tactical perspective based on extracted motion data.

[1331] "Means for providing real-time notifications to workers based on data on abnormal operations detected by the abnormality detection means" refers to technology that immediately notifies workers of details of an abnormality when an abnormality is detected.

[1332] This invention is a system that uses a bird's-eye view camera and a tracking camera to monitor and analyze the operation of manufacturing machines in a manufacturing site. The system is composed of a server, a terminal, and a user.

[1333] The server processes video data collected from the overhead camera and tracking camera in real time. The hardware used includes a high-resolution overhead camera, tracking camera, and server. The software includes OpenCV for Python, FFmpeg, YOLO, DeepSORT, and TensorFlow. The server removes noise from the received video data, adjusts the resolution, and divides it into frames. It then uses YOLO and DeepSORT to extract coordinate data for the manufacturing machine and work objects from the video data. The movement pattern and speed of the manufacturing machine are calculated from the coordinate data of consecutive frames.

[1334] The server detects abnormal behavior and patterns and generates anomaly detection results and statistical data, including the operational efficiency, operation history, and frequency of failures of the manufacturing machines.

[1335] The server also delivers anomaly detection results and statistical data in real time to the terminal. The terminal is a pair of smart glasses worn by the worker. The terminal uses Three.js and WebGL to visualize and display the received data in 3D space. If a specific anomaly is detected, the terminal notifies the worker in real time.

[1336] The user, a worker, can visually check the operation data of the manufacturing machine in real time through the smart glasses, which allows them to respond immediately if an abnormality occurs.

[1337] As a concrete example, consider a scenario in which the operation of a robot arm in a production line is monitored. An overhead camera monitors the entire production line, while a tracking camera tracks the movement of the robot arm. The server analyzes this video data and determines whether the robot arm is operating normally or if any abnormalities have occurred. If an abnormality is detected, a worker wearing smart glasses is immediately notified, enabling a prompt response.

[1338] An example prompt might look like this:

[1339] Monitor the movement patterns followed by specific manufacturing robots in your factory in real time, get instant notification if any abnormal behavior is detected, view the coordinate data and movement speed of each robot in 2D and 3D, and store a history of any abnormal behavior.

[1340] This system will significantly improve quality control and efficiency at the manufacturing site, and enable rapid response when abnormalities occur.

[1341] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1342] Step 1:

[1343] The server receives video data from the overhead camera and the tracking camera in real time.

[1344] Specific operation: The server receives the video signal from the camera and divides it into frames. The input is the video data captured by the camera, and the output is the data of each frame with noise removal and resolution adjustment. Preprocessing is performed using OpenCV for Python.

[1345] Step 2:

[1346] The server extracts coordinate data of the manufacturing machine and the work object from the pre-processed frame data.

[1347] Specific operation: Using YOLO and DeepSORT, we detect manufacturing machines and objects in the frame and obtain their coordinate data. The input is each preprocessed frame, and the output is the 2D coordinate data of the detected manufacturing machines and objects.

[1348] Step 3:

[1349] The server calculates the movement patterns and speeds of the manufacturing machines and objects based on the coordinate data between successive frames.

[1350] Specific operation: The server uses TensorFlow to analyze the acquired coordinate data and calculate the movement distance and speed. The input is the detected coordinate data, and the output is the calculated movement pattern and speed data.

[1351] Step 4:

[1352] The server detects abnormal behavior based on movement patterns and speed data.

[1353] Specific operation: The server compares the accumulated movement pattern and speed data to determine whether there are any abnormalities. In this step, an algorithm is used to instantly detect abnormal behavior. Its input is the movement pattern and speed data, and its output is the data of the detected abnormal behavior.

[1354] Step 5:

[1355] The server notifies the worker in real time if any abnormal operation is detected.

[1356] Specific operation: When an abnormality is detected, the server immediately sends an abnormal operation notification to the worker's smart glasses using Socket.IO. The input is the detected abnormal operation data, and the output is a real-time notification.

[1357] Step 6:

[1358] The terminal visualizes abnormal behavior data and statistical data received from the server.

[1359] How it works: The smart glasses use Three.js and WebGL to visually display the received data in 3D space. Its input is abnormal behavior data and statistical data from the server, and its output is a visualized interface.

[1360] Step 7:

[1361] Using smart glasses, users can check the operation of manufacturing machines and receive abnormality notifications in real time and respond accordingly.

[1362] Specific Action: The user checks the display of the smart glasses and takes action if necessary. In this step, the input received by the user is the display data of the smart glasses, and the output is the corresponding action.

[1363] This series of processes makes it possible to monitor the operation of manufacturing machines at the manufacturing site in real time and to immediately notify workers if an abnormality occurs.

[1364] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1365] The system of the present invention uses a bird's-eye view camera and a tracking camera to collect detailed data on the entire game, players, and the ball in real time, and automatically analyzes and compiles it. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it aims to improve the user experience. Below, a specific embodiment of each processing step is explained.

[1366] This system begins with the placement of overhead cameras and tracking cameras. The overhead cameras are installed in a position that allows for a bird's-eye view of the entire stadium. Meanwhile, the tracking cameras are high-resolution cameras that are placed at key points in the game to track specific players or the ball in detail. These cameras are connected to a server and transmit video data in real time.

[1367] Server Processing

[1368] 1. Video data reception and preprocessing

[1369] The server receives video data from the overhead camera and tracking camera in real time, and the received video data is divided into frames.

[1370] 2. Object detection and coordinate data extraction

[1371] The server uses an object detection algorithm (e.g., YOLO or DeepSORT) to detect players and the ball in each frame and extract their coordinate data, which represents the positional information of players and the ball in two or three dimensions.

[1372] 3. Calculating movement patterns and speeds

[1373] The server calculates the movement patterns and speeds of players and the ball from the coordinate data of successive frames, which allows visualization of each player's running distance and speed.

[1374] 4. Event Detection

[1375] The server uses the calculated coordinate and velocity data to detect events during the match (e.g., passes, shots, goals), and tags the events.

[1376] 5. Skeletal data extraction and motion analysis

[1377] The server uses a skeletal detection algorithm such as OpenPose or MediaPipe to extract the player's skeletal data (joint positions), which is then used to analyze the player's movements (e.g., jump shots and sliding).

[1378] 6. Generating and distributing statistical data

[1379] The server combines the coordinate data and event detection results to generate statistical data for each player (e.g., pass success rate, number of successful shots, number of turnovers). The generated statistical data is delivered to the device in real time.

[1380] 7. Emotion Recognition with Emotion Engine

[1381] The server analyzes the user's real-time video and audio data using an emotion engine to recognize the user's emotional state (e.g., excitement, stress, interest).

[1382] 8. Integration and Utilization of Emotional Data

[1383] The server integrates the emotion data generated by the emotion engine with statistical data and event data, generates match highlights and other interesting content based on the user's interests and reactions, and delivers them to the device.

[1384] Terminal handling

[1385] 1. Data Receipt and Analysis

[1386] The device receives statistical data and emotion data sent from the server in real time, and the received data is analyzed immediately.

[1387] 2. Visualization and Notifications

[1388] The device visually displays an overview of the entire game and performance data for each player. It also selectively displays interesting highlights and specific plays based on the user's emotions detected by an emotion engine. Furthermore, it notifies the user in real time when important game events occur.

[1389] User operations

[1390] 1. Viewing real-time data

[1391] Users can check the progress of the match in real time through their devices, with overhead camera footage and analysis data displayed simultaneously.

[1392] 2. Content display based on detailed analysis and emotion recognition

[1393] Users can select and view the performance data of specific players or match highlights, and the emotion engine displays content that reflects the user's emotional data, improving the user experience.

[1394] Specific examples

[1395] For example, in a soccer match, the server captures camera footage from the start of the match, and the AL model updates the position data of each player and the ball every 0.1 seconds. If the server detects that player A passes the ball and player B receives it, it tags the action as a "pass" and increments the number of successful passes in real time. If the emotion engine recognizes the user's excitement, the device will visually display highlights of the play according to the user's excitement level.

[1396] The system will enable spectators to feel more at home in the match, gain detailed insights based on player performance analysis, and display content based on emotional data, providing a more personalized viewing experience.

[1397] As described above, the system of the present invention aims to improve the user experience by integrating multi-camera video analysis and emotion recognition technology to realize automatic analysis of match data and real-time notification.

[1398] The processing flow will be explained below.

[1399] Server Processing

[1400] Step 1:

[1401] The server receives video data from the overhead camera and tracking camera in real time and divides the received video data into frames.

[1402] Step 2:

[1403] The server performs pre-processing of the video data, specifically image pre-processing such as noise removal, resolution adjustment, and color correction.

[1404] Step 3:

[1405] The server uses an object detection algorithm (e.g., YOLO or DeepSORT) to detect players and the ball in each frame, and extracts the position information of the detected players and the ball as coordinate data.

[1406] Step 4:

[1407] The server calculates the movement patterns of the players and the ball based on the coordinate data of successive frames, including the distance and speed of the players.

[1408] Step 5:

[1409] The server automatically detects events (e.g., passes, shots, goals) during a match based on the calculated coordinate and velocity data, and tags the detected events.

[1410] Step 6:

[1411] The server uses a skeletal detection algorithm such as OpenPose or MediaPipe to extract skeletal data for each player, including the position information of each joint.

[1412] Step 7:

[1413] The server analyzes the player's movements based on the extracted skeletal data, identifying movements such as jump shots and sliding.

[1414] Step 8:

[1415] The server integrates the coordinate data and the event detection results to generate statistical data for each player (e.g., pass success rate, number of successful shots, number of turnovers).

[1416] Step 9:

[1417] The server delivers the generated statistical data to the device in real time, using standard formats such as JSON.

[1418] Step 10:

[1419] The server analyzes real-time video and audio data acquired from the user using an emotion engine to recognize the user's emotional state (e.g., excitement, stress, interest).

[1420] Step 11:

[1421] The server integrates the emotion data obtained by the emotion engine with statistical and event data, and generates game highlights and other engaging content based on the user's interests and reactions.

[1422] Step 12:

[1423] The server delivers content based on the generated emotion data to the device in real time, which is displayed as a visual feed that reflects the user's emotional state.

[1424] Terminal handling

[1425] Step 1:

[1426] The device receives statistical data and emotional data sent from the server in real time, and the received data is analyzed immediately.

[1427] Step 2:

[1428] Based on the analyzed data, the device visually displays an overview of the entire match and performance data for each player, including the player's movement patterns and speed.

[1429] Step 3:

[1430] The terminal selectively displays interesting highlights or specific plays based on the user's emotional state detected by the emotion engine, thereby providing content that attracts the user's interest.

[1431] Step 4:

[1432] The device will notify the user in real time when important match events occur, including pop-up messages and audio alerts.

[1433] User operations

[1434] Step 1:

[1435] Users can check the progress of the match in real time through their devices, with overhead camera footage and analysis data displayed simultaneously.

[1436] Step 2:

[1437] Users can select a specific player's performance data and view detailed statistics, such as shooting percentage and distance traveled.

[1438] Step 3:

[1439] The device will display highlights and detailed information related to the user's favorite players and plays of interest based on the user's emotional data recognized by the emotion engine, providing a more personalized viewing experience.

[1440] Example 2

[1441] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1442] Conventional match analysis systems can acquire player and ball position data and detect events based on that data. However, these systems do not take into account user emotional data, which means they cannot provide content optimized for each spectator. Furthermore, there is a lack of technology to perform detailed tactical analysis in real time, making it difficult to enhance the sense of realism in live match viewing.

[1443] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1444] In this invention, the server includes an overhead camera, a tracking camera, a means for processing video data received from the overhead camera and the tracking camera in real time, a means for extracting object coordinate data from the video data, a means for calculating the object's movement pattern and speed based on the coordinate data, a means for detecting events from the coordinate data and speed data, a means for generating statistical data based on the coordinate data and the event detection results, a means for distributing the statistical data to a terminal in real time, a means for recognizing a user's emotion, and a means for integrating the emotion data with the statistical data and event data to generate individually customized content. This enables the provision of personalized content that reflects the user's emotion, achieving a more realistic viewing experience. Furthermore, detailed tactical analysis can be performed in real time, enabling the provision of more accurate data.

[1445] A "bird's-eye view camera" is a camera that is installed in a position that allows it to overlook the entire venue from above and captures video data over a wide area.

[1446] A "tracking camera" is a high-resolution camera installed to closely track a specific player or the ball.

[1447] The "means for processing video data in real time" is a function that includes a process for dividing video data received from the overhead camera and tracking camera and extracting objects and coordinates.

[1448] "Means for extracting coordinate data" refers to an algorithm or system for obtaining position information of objects (such as players or the ball) from video data.

[1449] The "means for calculating the movement pattern and speed" is a function for calculating the movement path and speed of an object based on the coordinate data of successive frames.

[1450] The "event detection means" is a system that identifies important actions during a match (e.g., passes, shots, goals) based on movement patterns and speed data.

[1451] The "means for generating statistical data" is a function that integrates coordinate data and event detection results to calculate detailed statistical information for each player (e.g., pass success rate, number of successful shots).

[1452] The "means for delivering to terminals in real time" refers to a process and system for instantly transmitting the generated statistical data to user terminals.

[1453] The "means for recognizing user emotions" is a technology that analyzes the user's facial expressions and vocal tone to identify emotional states such as excitement, stress, interest, etc.

[1454] "Means for integrating emotional data with statistical data and event data to generate individually customized content" refers to a system that combines recognized user emotional data with other match data to create content (e.g., highlights) that is tailored to each individual user.

[1455] The system of this invention uses a bird's-eye view camera and a tracking camera to collect detailed data on the entire game, players, and the ball in real time, and automatically analyzes and compiles it.Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it aims to improve the user experience.

[1456] The system begins with the placement of overhead cameras and tracking cameras. The overhead cameras are installed in a position that allows for a bird's-eye view of the entire stadium, while the tracking cameras are high-resolution cameras that are placed at key points in the game to track specific players or the ball in detail. These cameras are connected to a server and transmit video data in real time.

[1457] Server Processing

[1458] The server first receives video data in real time from the overhead camera and tracking camera. The received video data is divided into frames, and an object detection algorithm (e.g., YOLO or DeepSORT) is used to detect players and the ball in each frame and extract their coordinate data. This coordinate data represents the positional information of players and the ball in two or three dimensions.

[1459] The server calculates the movement patterns and speeds of players and the ball from the coordinate data of consecutive frames. This allows the running distance and speed of each player to be visualized. The calculated coordinate and speed data are then used to detect events during the game (e.g., passes, shots, goals). Detected events are tagged. Furthermore, a skeleton detection algorithm such as OpenPose or MediaPipe is used to extract players' skeletal data (joint positions). Based on this skeletal data, players' movements (e.g., jump shots and sliding) are analyzed.

[1460] The server combines the generated coordinate data and event detection results to generate statistical data for each player (e.g., pass success rate, number of successful shots, number of turnovers). This statistical data is delivered to the device in real time. In addition, the server analyzes the user's real-time video and audio data using an emotion engine to recognize the user's emotional state (e.g., excitement, stress, interest). The server combines the emotion data generated by the emotion engine with the statistical data and event data to generate game highlights and other interesting content based on the user's interests and reactions, and delivers it to the device.

[1461] Terminal handling

[1462] The device receives statistical and emotional data sent from the server in real time and analyzes it instantly. The device visually displays an overview of the entire game and performance data for each player. It also selectively displays interesting highlights and specific plays based on the user's emotions detected by the emotion engine. Furthermore, when important game events occur, the device notifies the user in real time.

[1463] User operations

[1464] Users can check the progress of the game in real time through their devices. Images from a bird's-eye view camera and analytical data are displayed simultaneously, and they can select and check the performance data of specific players or highlights of the game. An emotion engine displays content that reflects the user's emotional data, improving the user experience.

[1465] Specific examples

[1466] For example, in a soccer match, the server captures camera footage from the start of the match, and the AI ​​model updates the position data of each player and the ball every 0.1 seconds. If the server detects that player A passes the ball and player B receives it, it tags the action as a "pass" and increments the number of successful passes in real time. If the emotion engine recognizes the user's excitement, the device will visually display highlights of the play according to the user's excitement level.

[1467] Prompt Sentence Examples

[1468] "Analyze the number of passes and ball possession time of Player A in a soccer match in real time, and display highlights according to the user's excitement level."

[1469] The system will enable spectators to feel more at home in the match, gain detailed insights based on player performance analysis, and display content based on emotional data, providing a more personalized viewing experience.

[1470] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1471] Step 1: Receiving video data from the overhead camera and tracking camera

[1472] The server receives video data in real time from the overhead camera and tracking camera. This video data is divided into frames at high resolution. The input is the video data sent from the camera, and the output is the video data for each frame stored in the server.

[1473] Step 2: Splitting and Preprocessing Frames

[1474] The server divides the received video data into frames and performs preprocessing, which includes noise removal and frame alignment. The input is video data for each frame, and the output is the preprocessed frame data.

[1475] Step 3: Object detection and coordinate data extraction

[1476] The server applies an object detection algorithm (such as YOLO or DeepSORT) to the preprocessed frame data to extract the coordinate data of players and the ball. The input is the preprocessed frame data, and the output is the coordinate data (e.g., x-coordinate, y-coordinate) of players and the ball in each frame.

[1477] Step 4: Calculate movement patterns and speeds

[1478] The server uses the coordinate data of consecutive frames to calculate the movement patterns and speeds of players and the ball. For example, if player A is located at x=100, y=200 in frame 1 and moves to x=120, y=220 in frame 2, the server calculates the movement distance and speed between them. The input is the coordinate data of consecutive frames, and the output is the movement pattern and speed data.

[1479] Step 5: Detecting an event

[1480] The server analyzes movement pattern and speed data to detect events (e.g., passes, shots, goals) during a match. For example, if it detects that player A passes the ball and player B receives it, it tags this action as a "pass." The input is movement pattern and speed data, and the output is event-tagged data.

[1481] Step 6: Extracting skeletal data and motion analysis

[1482] The server uses OpenPose and MediaPipe to extract the player's skeletal data. Then, it analyzes the player's movements (e.g., jumping, sliding) based on this skeletal data. The input is frame data and coordinate data, and the output is skeletal data and the results of the movement analysis.

[1483] Step 7: Generate and distribute statistical data

[1484] The server integrates the coordinate data, event detection results, and motion analysis results to generate statistical data for each player (e.g., pass success rate, number of successful shots). The generated statistical data is delivered to the device in real time. The input is coordinate data, event data, and motion analysis results, and the output is statistical data.

[1485] Step 8: Emotion Recognition with the Emotion Engine

[1486] The server analyzes the user's real-time video and audio data using an emotion engine to recognize the user's emotional state (e.g., excitement, stress, interest). The input is the user's video and audio data, and the output is the user's emotional data.

[1487] Step 9: Integrate and utilize emotion data

[1488] The server integrates the emotion data generated by the emotion engine with statistical data and event data to generate match highlights and engaging content based on the user's interests and reactions. The generated content is delivered to the device. The inputs are statistical data, event data, and emotion data, and the output is customized highlights and content.

[1489] (Application example 2)

[1490] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1491] Traditional sports viewing systems limit the amount of information spectators can obtain in real time, making it difficult for them to gain a deep understanding of the game's movements and tactics. Furthermore, they lack a mechanism for automatically displaying content that matches the spectator's interests and emotional state, resulting in a lack of a personalized viewing experience for each spectator. This often results in a uniform and uninteresting viewing experience for spectators.

[1492] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for processing video data received from the overhead camera and tracking camera in real time, means for extracting coordinate data of people and the ball from the video data, means for detecting the movement patterns, speed, and events of people and the ball based on the coordinate data, means for integrating the statistical data and emotional data, generating game highlights and interesting content based on the user's interests and reactions, and delivering the content to the terminal in real time, and means for recognizing the user's emotional state using an emotion engine. This allows spectators to not only grasp detailed game movements and tactical analysis in real time, but also enjoy personalized content tailored to their individual emotional states.

[1493] A "bird's-eye view camera" is a camera installed in a position that allows it to overlook the entire venue from above, and is a device that captures footage of the entire match.

[1494] A "tracking camera" is a high-resolution camera that is placed at key points in the game to track specific players or the ball in detail.

[1495] "Video data" refers to digital video data received from the overhead camera and tracking camera and processed in real time.

[1496] "Coordinate data" refers to data that indicates the position information of people and balls extracted from video data, and is expressed in two or three dimensions.

[1497] A "movement pattern" indicates the movement path of a person and a ball calculated from continuous coordinate data.

[1498] An "event" is a specific action or occurrence that occurs during a match (e.g., a pass, a shot, a goal) and is tagged accordingly.

[1499] "Statistical data" refers to data generated based on coordinate data and event detection results, and includes performance information for each player (e.g., pass success rate, number of successful shots).

[1500] The "emotion engine" is a system that analyzes the user's real-time video and audio data to recognize the user's emotional state (e.g., excitement, stress, interest).

[1501] "Emotion Data" refers to data that indicates the user's emotional state as analyzed and recognized by the emotion engine.

[1502] "Highlights" is content that highlights important scenes from the game or parts that will interest spectators.

[1503] "User experience" refers to the overall experience a user has through a system, including operability and satisfaction.

[1504] To implement this invention, the following system is required: This system collects video data in real time using a bird's-eye view camera and a tracking camera, and combines it with an emotion engine that recognizes the user's emotions to improve the user experience.

[1505] Hardware Configuration

[1506] 1. Bird's-eye view camera: A camera installed in a position that allows a bird's-eye view of the entire venue. Specifically, general-purpose cameras such as Arducam can be used.

[1507] 2. Tracking camera: A high-resolution camera for tracking a specific player or the ball in detail. For example, a Basler camera can be used.

[1508] 3. Head-mounted display: A device that allows spectators to virtually watch the game. An example of this is the Oculus Quest 2.

[1509] Software Configuration

[1510] 1. Object detection algorithm: An algorithm that extracts coordinate data of people and spheres from video data acquired from overhead cameras and tracking cameras. Specifically, YOLO (You Only Look Once) and DeepSORT can be used.

[1511] 2. Skeleton detection algorithm: An algorithm for analyzing human skeletal data. Examples of algorithms that can be used include OpenPose and MediaPipe.

[1512] 3. Emotion engine: A system that analyzes real-time video and audio data of the user to recognize their emotional state. For example, Affectiva's emotion recognition API can be used.

[1513] 4. 3D modeling software: Software for visualizing the events during the match in 3D. For example, Unity can be used.

[1514] 5. Image processing library: A library for processing camera images. OpenCV can be used as a specific example.

[1515] Data calculation and processing

[1516] The server processes video data received from the overhead camera and tracking camera in real time. First, it divides the video data into frames and uses an object detection algorithm to detect people and the ball in each frame and extract their coordinate data. Next, it calculates the movement patterns and speed of people and the ball from the coordinate data of consecutive frames to detect events during the game. This data is integrated and delivered to the device in real time. It also uses an emotion engine to recognize the user's emotions, and combines the emotion data with statistical data to generate game highlights and interesting content tailored to the user, which is delivered to the device.

[1517] Specific examples

[1518] For example, during a soccer match, the server captures video from an overhead camera and a tracking camera, and the AL model updates the position data of each player and the ball every 0.1 seconds. If the server detects that player A passes the ball and player B receives it, it tags the action as a "pass" and increments the number of successful passes in real time. If the emotion engine recognizes the user's excitement, the device will visually display highlights of the play according to the user's excitement level.

[1519] Prompt Sentence Examples

[1520] A scenario where you are watching a soccer match in real time. Accurately track player movements and ball position and highlight key plays. Automatically generate and present relevant highlights based on the user's changing interests and emotions. The emotion engine should recognize the user's emotions and provide personalized match analysis and notifications based on those emotions.

[1521] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1522] Step 1:

[1523] The server receives video data from the overhead camera and tracking camera in real time and splits it into frames. The input is the video stream from the camera, and the output is the split frame data. This processing is performed using OpenCV.

[1524] Step 2:

[1525] The server applies an object detection algorithm (such as YOLO or DeepSORT) to each divided frame to detect players and the ball. The input is the frame data, and the output is the coordinate data of the detected objects (players and the ball). At this time, the coordinate data is recorded along two or three dimensions.

[1526] Step 3:

[1527] The server calculates the movement patterns and speeds of players and the ball based on coordinate data extracted from consecutive frames. The input is a continuous set of coordinate data, and the output is movement pattern data and speed data. In this calculation, the speed is calculated based on the rate of change of position information.

[1528] Step 4:

[1529] The server detects events (e.g., passes, shots, goals) during a match based on movement patterns and velocity data. The input is movement patterns and velocity data, and the output is event-tagged data. The event detection algorithm identifies events based on specific movement actions and velocity changes.

[1530] Step 5:

[1531] The server uses an object detection algorithm (OpenPose or MediaPipe) to extract the player's skeletal data. The input is a video frame, and the output is the player's skeletal coordinate data. This data is recorded as joint position information.

[1532] Step 6:

[1533] The server performs motion and tactical analysis of players based on their skeletal and coordinate data. The input is skeletal and coordinate data, and the output is motion analysis data and tactical analysis data. This processing identifies specific motion patterns and visualizes tactical performance.

[1534] Step 7:

[1535] The server generates statistics for each player based on the statistical data and event detection results. The input is a series of event data and coordinate data, and the output is a statistical data feed. In this step, player performance indicators (e.g., pass success rate, shot success rate) are calculated.

[1536] Step 8:

[1537] The server uses an emotion engine to analyze the user's real-time video and audio data to recognize the user's emotional state. The input is the user's video and audio data, and the output is emotional data. This process is performed using Affectiva's emotion recognition API.

[1538] Step 9:

[1539] The server integrates the emotional data and statistical data to generate highlights and other engaging content based on the user's interests and reactions, and delivers them to the device in real time. The input is emotional data and statistical data, and the output is personalized highlight content. The device receives this data and displays it visually.

[1540] Step 10:

[1541] Users can check the progress of the game in real time through their devices and enjoy personalized content. The input is data received from the server, and the output is an improved user experience, allowing users to experience a truly immersive game experience.

[1542] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1543] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1544] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1545] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1546] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1547] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1548] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1549] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1550] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1551] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1552] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1553] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1554] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1555] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1556] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1557] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1558] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1559] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1560] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1561] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1562] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1563] The following is further disclosed regarding the above embodiment.

[1564] (Claim 1)

[1565] An overhead camera and

[1566] A tracking camera and

[1567] means for processing video data received from the overhead camera and the tracking camera in real time;

[1568] means for extracting coordinate data of a person and a sphere from the video data;

[1569] means for detecting the movement patterns, speeds, and events of people and balls based on the coordinate data;

[1570] means for generating statistical data based on the coordinate data and the event detection results;

[1571] means for delivering the statistical data to a terminal in real time;

[1572] A system including:

[1573] (Claim 2)

[1574] means for analyzing the movement of a person based on the coordinate data and skeletal data;

[1575] a means for performing tactical analysis based on the results of the motion analysis;

[1576] The system of claim 1 further comprising:

[1577] (Claim 3)

[1578] means for providing real-time notifications to spectators based on data on the events detected by the event detection means;

[1579] The system of claim 1 further comprising:

[1580] "Example 1"

[1581] (Claim 1)

[1582] An overhead camera and

[1583] A tracking camera and

[1584] means for processing video data received from the overhead camera and the tracking camera in real time;

[1585] means for extracting coordinate data of a person and a sphere from the video data using an object detection algorithm;

[1586] means for detecting the movement patterns, speeds, and events of people and balls based on the coordinate data;

[1587] means for generating statistical data based on the coordinate data and the event detection results;

[1588] means for delivering the statistical data to a terminal in real time;

[1589] A system including:

[1590] (Claim 2)

[1591] means for analyzing the movement of a person based on the coordinate data and skeletal data;

[1592] a means for performing tactical analysis based on the results of the motion analysis;

[1593] means for extracting joint positions using said skeleton detection algorithm;

[1594] The system of claim 1 further comprising:

[1595] (Claim 3)

[1596] means for providing real-time notifications to spectators based on data on the events detected by the event detection means;

[1597] A means for the terminal to analyze the data, extract information necessary for display, and display the information on a user interface;

[1598] The system of claim 1 further comprising:

[1599] "Application Example 1"

[1600] (Claim 1)

[1601] An overhead camera and

[1602] A tracking camera and

[1603] means for processing video data received from the overhead camera and the tracking camera in real time;

[1604] means for extracting coordinate data of people and objects from the video data;

[1605] means for detecting movement patterns, speeds, and events of people and objects based on the coordinate data;

[1606] means for generating statistical data based on the coordinate data and the event detection results;

[1607] means for detecting operational patterns of the manufacturing machine and notifying abnormal operation in real time;

[1608] means for delivering the statistical data and anomaly detection results to a terminal in real time;

[1609] A system including:

[1610] (Claim 2)

[1611] means for analyzing the movement of a person based on the coordinate data and skeletal data;

[1612] a means for performing tactical analysis based on the results of the motion analysis;

[1613] means for analyzing the operation of the manufacturing machine based on the coordinate data;

[1614] The system of claim 1 further comprising:

[1615] (Claim 3)

[1616] means for providing real-time notifications to spectators based on data on the events detected by the event detection means;

[1617] a means for sending a real-time notification to an operator based on data of abnormal operation detected by the abnormality detection means;

[1618] The system of claim 1 further comprising:

[1619] "Example 2: Combining Emotion Engines"

[1620] (Claim 1)

[1621] An overhead camera and

[1622] A tracking camera and

[1623] means for processing video data received from the overhead camera and the tracking camera in real time;

[1624] means for extracting coordinate data of an object from the video data;

[1625] means for calculating the movement pattern and speed of an object based on the coordinate data;

[1626] means for detecting an event from the coordinate data and velocity data;

[1627] means for generating statistical data based on the coordinate data and the event detection results;

[1628] means for delivering the statistical data to a terminal in real time;

[1629] means for recognizing a user's emotion;

[1630] means for integrating said emotion data with statistical data and event data to generate individually customized content;

[1631] A system including:

[1632] (Claim 2)

[1633] means for analyzing the motion of an object based on the coordinate data and skeletal data;

[1634] a means for performing tactical analysis based on the results of the motion analysis;

[1635] The system of claim 1 further comprising:

[1636] (Claim 3)

[1637] a means for sending real-time notifications to spectators based on the event data and emotion data detected by the event detection means;

[1638] The system of claim 1 further comprising:

[1639] "Application example 2 when combining emotion engines"

[1640] (Claim 1)

[1641] An overhead camera and

[1642] A tracking camera and

[1643] means for processing video data received from the overhead camera and the tracking camera in real time;

[1644] means for extracting coordinate data of a person and a sphere from the video data;

[1645] means for detecting the movement patterns, speeds, and events of people and balls based on the coordinate data;

[1646] means for generating statistical data based on the coordinate data and the event detection results;

[1647] means for integrating the statistical data and emotional data to generate highlights of matches and other engaging content based on the user's interests and reactions, and delivering the content to the terminal in real time;

[1648] means for recognizing an emotional state of a user using an emotion engine;

[1649] A system including:

[1650] (Claim 2)

[1651] means for analyzing the movement of a person based on the coordinate data and skeletal data;

[1652] a means for performing tactical analysis based on the results of the motion analysis;

[1653] a means for adjusting the display content of the match analysis results based on the user's emotion data;

[1654] The system of claim 1 further comprising:

[1655] (Claim 3)

[1656] means for providing real-time notifications to spectators based on data on the events detected by the event detection means;

[1657] a means for automatically displaying highlights that will attract the audience's attention based on the emotion data recognized by the emotion engine;

[1658] The system of claim 1 further comprising: [Explanation of symbols] ...

Claims

1. An overhead camera and A tracking camera and means for processing video data received from the overhead camera and the tracking camera in real time; means for extracting coordinate data of a person and a sphere from the video data; means for detecting the movement patterns, speeds, and events of people and balls based on the coordinate data; means for generating statistical data based on the coordinate data and the event detection results; means for delivering the statistical data to a terminal in real time; A system including:

2. means for analyzing the movement of a person based on the coordinate data and skeletal data; a means for performing tactical analysis based on the results of the motion analysis; The system of claim 1 further comprising:

3. means for providing real-time notifications to spectators based on data on the events detected by the event detection means; The system of claim 1 further comprising:

Images