Device and method for providing customized video for each sports player based on statistics data

Abstract

The embodiments provide a device and method for providing customized video for each sports athlete based on statistical data. The method according to the embodiments may include: receiving original video data from a first external server, dividing the original video data into frames, and normalizing each frame to generate preprocessed data including a plurality of frames; receiving user information from a user terminal and receiving statistical data from an external server, and normalizing and clustering the statistical data based on the user information; the user information includes at least one of the user's name, contact information, date of birth, residence, and gender, and setting a region of interest for each of the plurality of frames based on the statistical data; identifying a first frame among the plurality of frames, identifying a second frame following the first frame, and generating a target motion vector (u,v) based on the pixel brightness difference of the region of interest (L) between the first frame and the second frame; analyzing the direction and speed of movement of a target based on the target motion vector, identifying the target to select a highlight frame, and connecting the selected highlight frames to generate a highlight video.

Description

Technology Field

[0001] Embodiments of the present invention relate to a device and method for providing customized video for each sports player based on statistical data, and more specifically, to a device and method for providing customized video for each sports player based on statistical data that synchronizes original video data and statistical data to set a region of interest based on the player's individual statistical information and shortens the object detection speed based thereon. Background Technology

[0002] Traditional methods for extracting highlight scenes from sports matches or gameplay videos have primarily involved editors manually reviewing the entire footage to select important moments. This manual editing approach requires a massive amount of time and manpower, and suffers from the problem of difficulty in selecting consistent highlights due to heavy reliance on the editor's subjective judgment. Furthermore, despite the existence of statistical data for matches or games, this data is not directly utilized in video editing, frequently resulting in the missed actual crucial plays or turning points. While some automated highlight generation systems have been developed, they suffer from low accuracy due to their reliance on superficial characteristics such as sound volume or screen transition speed, and they are limited in their inability to generate personalized highlights that reflect user interests or preferences. In particular, even for the same match or game video, user interest may vary depending on factors such as age, gender, and region; however, existing systems generate highlights based solely on standardized criteria, leading to low user satisfaction. Moreover, there is a problem in that the technology to accurately track and analyze the movements of specific players or characters (targets) within the video is insufficient, making it difficult to effectively extract important scenes related to those targets. The problem to be solved

[0003] Embodiments of the present invention aim to provide a method and apparatus for detecting abnormal behavior-based sports video events to solve problems such as the limitations of the prior art. Specifically, the invention provides a method and apparatus that can simultaneously enhance the safety and immersion of sports video by detecting abnormal behavior by comparing it with a user's normal exercise pattern and classifying the detected frames as events.

[0004] The technical problems to be solved in the embodiments are not limited to those mentioned above, and other unmentioned technical problems may be considered by those skilled in the art from the various embodiments described below. means of solving the problem

[0005] A method for providing customized video for each sports player based on stat data according to an embodiment of the present invention comprises: receiving original video data from a shooting system by means of an electronic device, dividing the original video data into frame units, and normalizing each frame to generate preprocessed data including a plurality of frames; wherein the original video data is a game video in which both a first user and a second user appear, and wherein the electronic device receives user information regarding the first user from a user terminal and receives first stat data regarding the first user from an external server, and sets a region of interest for each of the plurality of frames based on the first stat data; wherein the user information includes at least one of the first user's name, contact information, date of birth, residence, and gender, and wherein the electronic device specifies a first frame among the plurality of frames, specifies a second frame following the first frame, and generates a target motion vector based on the pixel brightness difference of the region of interest between the first frame and the second frame. The method may include the step of analyzing the direction and speed of movement of a target based on the target motion vector by the electronic device, identifying the target to select highlight frames, and connecting the selected highlight frames to generate a highlight image.

[0006] The method may include the step of matching and storing the target motion vector and the statistical data by the electronic device.

[0007] The method may include the step of adjusting the size of the region of interest based on the ratio of noise pixels to total pixels within the frame by the electronic device.

[0008] The size of the above region of interest can be set based on the ratio of the noise pixel to the total pixels within the plurality of frames.

[0009] The method may include the step of moving the region of interest to detect a change in brightness again when the difference in pixel brightness in the region of interest between the first frame and the second frame is less than or equal to a threshold value by the electronic device. Effects of the invention

[0010] According to embodiments of the present invention, video analysis optimized for the characteristics of a player is possible by setting a region of interest using the player's statistical data. This allows for the automatic generation of customized highlight videos for each player, thereby significantly reducing video editing time. Furthermore, real-time processing is possible by adopting a motion vector generation method based on pixel brightness differences, and stable performance can be maintained in various shooting environments through a function that adjusts the size of the region of interest based on the noise pixel ratio.

[0011] Furthermore, by matching and storing target motion vectors with statistical data, a player's play patterns can be databased, which can be utilized for future player performance analysis and strategy formulation. The function of automatically moving the region of interest to re-detect when a pixel brightness difference below a threshold is detected enables continuous tracking even when the player moves out of the screen or the camera angle changes. Brief explanation of the drawing

[0012] The accompanying drawings, included as part of the detailed description to aid in understanding the embodiments, provide various embodiments and explain the technical features of the various embodiments together with the detailed description. FIG. 1 is a schematic diagram illustrating a sports game video recording system (10) according to one embodiment of the present invention. FIG. 2 is a block diagram for explaining the structure of the electronic device (100) of FIG. 1. Figure 3 is a diagram illustrating the multilayer neural network (122) of Figure 2. FIG. 4 is a block diagram for explaining the structure of the user terminal (200) of FIG. 1. FIG. 5 is a block diagram for explaining the structure of the sensor device (400) of FIG. 1. FIG. 6 is a flowchart illustrating the original image data preprocessing process according to one embodiment of the present invention. Figure 7 is a diagram schematically illustrating the process of dividing multiple original image data into frames. FIG. 8 is a flowchart illustrating a method for generating a highlight image based on stat data according to an embodiment of the present invention. Specific details for implementing the invention

[0013] The following embodiments are combinations of the components and features of the embodiments in a specific form. Each component or feature may be considered optional unless otherwise explicitly stated. Each component or feature may be implemented in a form not combined with other components or features. Additionally, various embodiments may be constructed by combining some components and / or features. The order of operations described in various embodiments may be changed. Some components or features of one embodiment may be included in another embodiment, or may be replaced with corresponding components or features of another embodiment.

[0014] In the description of the drawings, procedures or steps that could obscure the essence of the various embodiments were not described, nor were procedures or steps that could be understood by a person of ordinary knowledge in the relevant technical field described.

[0015] Throughout the specification, when a part is described as "comprising" or "including" a component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Furthermore, terms such as "...part," "...unit," and "module" as used in the specification refer to a unit that performs at least one function or operation, and this may be implemented in hardware, software, or a combination of hardware and software. Additionally, "one (a or an)," "one," "the," and similar related terms may be used in the context describing various embodiments (particularly in the context of the following claims) in both singular and plural forms, unless otherwise indicated in the specification or clearly contradicted by the context.

[0016] Hereinafter, embodiments according to various examples will be described in detail with reference to the accompanying drawings. The detailed description disclosed below, together with the accompanying drawings, is intended to describe exemplary embodiments of various examples and is not intended to represent the only embodiment.

[0017] In addition, specific terms used in various embodiments are provided to aid in understanding the various embodiments, and the use of such specific terms may be modified in other forms within the scope of not departing from the technical concept of the various embodiments.

[0018] FIG. 1 is a schematic diagram illustrating a sports game video recording system (10) according to one embodiment of the present invention.

[0019] Referring to FIG. 1, the electronic device (100) can receive original image data in real time from the shooting system (300).

[0020] The shooting system (300) may include a plurality of cameras. The plurality of cameras may include a first camera (310), a second camera (320), and a third camera (330). As illustrated in FIG. 1, the first camera (310), the second camera (320), and the third camera (330) may each be installed to photograph the user (P) from different directions. Each of the plurality of cameras may capture still images and video of the user (P). In one embodiment, each of the plurality of cameras may include one or more lenses, image sensors, image signal processors, or flashes. Although FIG. 1 illustrates the shooting system (300) as including three cameras, embodiments of the present invention are not limited thereto. If necessary, the shooting system (300) may include four or more cameras, or two or fewer cameras. However, in order to acquire image data of the user (P) from different angles, it is preferable for the shooting system (300) to include three or more cameras.

[0021] Each of the plurality of cameras included in the shooting system (300) can capture the stationary state or operating state of the user (P) to generate original image data (v_raw). The original image data (v_raw) may refer to data in which the stationary state or moving state of the user (P) is optically recorded.

[0022] In one embodiment, the shooting system (300) may include a first camera (310), a second camera (320) installed to photograph a user (P) from a different direction from the first camera (310), and a third camera (330) installed to photograph a user (P) from a different direction from the first camera (310) and the second camera (320).

[0023] The first camera (310) can capture the user (P) to generate the first original image data (v_raw1). The second camera (320) can capture the user (P) to generate the second original image data (v_raw2). The third camera (330) can capture the user (P) to generate the third original image data (v_raw3). Each image data may include metadata including time information corresponding to the time of image capture, a shooting area, and camera coordinate information. The metadata may be stored in formats such as IPTC (International Press Telecommunications Council), XMP (Extensible Metadata Platform from Adobe), and EXIF ​​(Exchangeable Image File Format).

[0024] In one embodiment, the electronic device (100) provides a membership registration interface to a user terminal (200) and receives user information corresponding to the membership registration interface from the user terminal (200), thereby registering the user to a sports multi-content system implemented by the electronic device (100). The sports multi-content system implemented by the electronic device (100) may include at least one of a multi-angle-based highlight video generation service, a social network-based challenge participation competition service, an abnormal behavior-based sports video event detection service, an emotion feedback-based AI sports video recommendation service, an individual sports discipline-specific highlight automatic generation service, a customized challenge video recommendation service based on individual exercise tendencies, an AI-based game prediction result-linked video summary generation service, a user voice command-based real-time highlight search service, a VR environment immersive sports highlight provision service, a real-time motion correction service based on unstructured exercise data, and a youth exercise data-based growth analysis and coaching video provision service.

[0025] In one embodiment, the electronic device (100) can implement a social network-based challenge participation competition system. The electronic device (100) can induce continuous participation by setting clear rules for the challenges in which users participate, establishing an authentication system capable of fairly judging them, and providing substantial rewards to users. The challenges provided on the challenge participation competition platform implemented by the electronic device (100) can be classified into participation-type challenges and challenge-type challenges. Each challenge may have unique participation conditions, challenge success or execution conditions, and success or execution rewards set.

[0026] In one embodiment, the electronic device (100) can determine whether a user participates in a challenge based on video data captured by a user terminal (200) and determine whether to pay a reward for participating in the challenge.

[0027] In one embodiment, the electronic device (100) may collect original image data from a plurality of user terminals (200). In this case, the original image data (v_raw) may refer to data optically recorded by capturing the stationary or operational state of a user (e.g., athlete, training coach, amateur, etc.) or an object by the camera module (260) of the user terminal (200). The original image data may include metadata including time information corresponding to the time of image capture, a shooting area, and camera coordinate information. The metadata may be stored in formats such as IPTC (International Press Telecommunications Council), XMP (Extensible Metadata Platform from Adobe), and EXIF ​​(Exchangeable Image File Format).

[0028] In one embodiment, the electronic device (100) may collect statistical data from an external server (not shown). Statistic data may refer to factual information recording the performance of a user. For example, when a user plays soccer, there are goals, assists, passes, sprints, aerial duels, etc., and in the case of a batter in baseball, there are hits, home runs, ground balls, flies, bunts, singles, doubles, triples, fouls, etc. The statistical data collected by the electronic device (100) from the external server may be collected in a database format determined according to the sport and the player. Here, the external server may be an official website of a sports league such as baseball, basketball, or soccer, or a data provider (API). In this specification, statistical data may refer to data classified by a user (e.g., player, amateur, training coach, etc.) and a sport (e.g., soccer, basketball, baseball, tennis, golf, etc.), and may include one or more factual pieces of information according to the classification.

[0030] FIG. 2 is a block diagram for explaining the structure of the electronic device (100) of FIG. 1. FIG. 3 is a diagram for explaining the multilayer neural network (122) of FIG. 2.

[0031] Referring to FIG. 2, the electronic device (100) may include a user management unit (110), an image discrimination unit (120), a challenge management unit (130), and a reward management unit (140).

[0032] The user management unit (110) can register users to the sports multi-content system through a membership registration process and collect and manage user information. The user management unit (110) can perform user registration and user information input processes. The user management unit (110) can provide a membership registration interface to the user terminal (200) and receive user information from the user terminal (200). Here, the user information may include at least one of the user's name, contact information, date of birth, place of residence, and gender. The user management unit (110) can provide services such as photo uploading, photo editing, payment, and guidance video output to users registered as members.

[0033] The user management unit (110) may further include a ranking calculation module (111) that aggregates and manages rankings based on the performance of users and teams, and a user DB (112) which is a database built by collecting user information received and information regarding the status of users' challenge participation, performance, ranking, etc.

[0034] The ranking calculation module (111) can calculate a ranking based on the reward points obtained from a challenge in which a user or team participated, and can update and aggregate the ranking according to a predetermined period and provide it to the user.

[0035] The user DB (112) may have a general data structure implemented in the storage space (hard disk or memory) of a computer system using a database management program (DBMS). The database may have a data storage form that allows for the free retrieval (extraction), deletion, editing, and addition of data. The database may be implemented in accordance with the purpose of one embodiment of the present disclosure using a relational database management system (RDBMS) such as Oracle, Informix, Sybase, and DB2, an object-oriented database management system (OODBMS) such as Gemston, Orion, and O2, and an XML native database such as Excelon, Tamino, and Sekaiju, and may have appropriate fields or elements to achieve its function.

[0036] The user management unit (110) can classify input user information and build a database. The user management unit (110) can classify input user information and store and manage it in the user DB (112). The user DB (112) can be implemented using server programs provided in various ways depending on operating systems such as DOS, Windows, Linux, UNIX, and Macintosh on general server hardware. Representative examples include Website and IIS (Internet Information Server) used in Windows environments, and CERN, NCSA, and APPACH used in UNIX environments. Additionally, the user DB (112) may be linked with an authentication system and a payment system for user authentication of services provided on a platform according to an embodiment of the present invention or for purchase payment related to services.

[0037] The image discrimination unit (120) collects image data submitted by the user to prove the achievement of the challenge, and can determine whether the user has achieved the challenge by using object detection technology based on the image data.

[0038] The image determination unit (120) may require the user to provide more than a predetermined number of image data to determine whether the user has performed or succeeded in the challenge.

[0039] The image determination unit (120) can analyze the image using an object detection algorithm and determine whether the image submitted by the user satisfies the actual challenge conditions. For example, if the user participates in a basketball challenge, the image determination unit (120) can collect image data from the user and determine whether the user's shot passes accurately through the rim in the image data to determine whether the challenge is successful.

[0040] To this end, the image discrimination unit (120) may further include an image preprocessing unit (121), a multilayer neural network (122), and a learning engine (123).

[0041] The image preprocessing unit (121) can receive original image data and preprocess it to generate preprocessed data. Here, preprocessing may refer to the operation of dividing the original image data into frames and normalizing each frame to generate preprocessed data. The image preprocessing unit (121) can normalize each frame of the original image data by adjusting it to the same pixel size. The original image data can be processed into preprocessed data containing a plurality of normalized frames (f1, f2, f3, …, fn). For example, if the original image data is a 90-minute video captured at 30 frames per second (fps), a total of 162,000 frames can be generated.

[0042] In one embodiment, the image preprocessing unit (121) can preprocess original image data to generate preprocessed data including a plurality of frames. The image preprocessing unit (121) can select one or more frames among the plurality of frames according to a preset period.

[0043] Here, the period may be a predetermined period based on the classification of the original image data. The classification of the original image data may be determined based on the metadata of the original image data. Alternatively, the classification of the original image data may be determined based on statistical data.

[0044] The preset period can vary. For example, the period can be 0.1 seconds, 1 second, 3 seconds, 5 seconds, etc. Or, the period can be in frames. For example, the period can be 10 frames, 20 frames, 30 frames, 120 frames, etc.

[0045] In one embodiment, the image preprocessing unit (121) may apply a color filter to each of one or more selected frames to form a grid. In one embodiment, the color filter may be a filter that divides each of the plurality of frames into a plurality of grids and performs color quantization on each of the plurality of grids to unify them into a single color.

[0046] The image preprocessing unit (121) can store one or more selected frames and gridded frames in memory and database based on the metadata of the original image data.

[0047] The image determination unit (120) can determine whether one or more gridded frames match by comparing each of them with a pre-stored pattern layer, classify the matching frames as event occurrence frames corresponding to the pattern layer, and extract a thumbnail image based on the frames.

[0048] The pattern layer may be a layer containing a pattern corresponding to at least one event included in the original video data. For example, if the original video data is a soccer video, the pattern layer may include a pattern layer corresponding to a goal, a pattern layer corresponding to a shot on target, a pattern layer corresponding to a corner kick, etc.

[0049] In one embodiment, the image discrimination unit (120) compares each of the pattern layer and one or more gridded frames, and if there is no frame that matches the pattern layer, it can select at least one of the one or more frames as a new pattern layer and store it in memory.

[0050] In one embodiment, the image discrimination unit (120) compares each of the pattern layer and one or more gridded frames, and if there is no frame that matches the pattern layer, it can select one or more frames again based on a different period.

[0051] In one embodiment, the image determination unit (120) can determine whether the pattern layer and one or more gridded frames match by converting each of the one or more frames into a frame vector having the color value of the grid as a component and converting the pattern layer pattern value as a component, calculating the similarity between each of the pattern vector and one or more frame vectors, and determining that the pattern layer and one or more gridded frames match if the similarity is greater than or equal to a predetermined threshold.

[0052] In one embodiment, the image discrimination unit (120) may set a region of interest for each of a plurality of frames based on statistical data. Here, 'region of interest' may refer to an area located at a specific coordinate with a specific size within a frame. The coordinates of the region of interest may be defined based on the coordinates of the center pixel of the region of interest.

[0053] In one embodiment, the region of interest (L) may be set based on reference coordinates within the frame. The reference coordinates may be, for example, the center coordinates of the frame or the edge coordinates of the frame. In another embodiment, the region of interest may be set based on a specific event. That is, the region of interest may be set according to a specific event (e.g., scoring, passing, dribbling, etc.) detected in the previous frame or the current frame. In yet another embodiment, the region of interest may be set based on a detected object (target).

[0054] The image discrimination unit (120) can detect the location and movement of an object using an object tracking algorithm or an object tracking filter, and set the area around the object as a region of interest. In another embodiment, the region of interest may be set based on the rate of change of pixel values. That is, the region of interest may be set based on an area where the change in pixel values ​​is large within a specific frame (e.g., an area with fast movement or change).

[0055] The image discrimination unit (120) can pre-supervised train a multilayer neural network (122) using multiple training data. A multilayer neural network is a prediction model implemented in software or hardware that mimics the computational power of a biological system using a large number of artificial neurons (or nodes).

[0056] The learning engine (123) can supervise the multilayer neural network (122) using learning data in which image data and correct answer data are input values ​​and challenge achievement result data is output value, so that the multilayer neural network (122) can accurately determine whether a challenge is achieved based on image data.

[0057] In this context, supervised learning refers to a learning process that uses data containing input and corresponding output values ​​as training data to find the output value corresponding to a given input; it signifies learning that takes place while the correct answer is known. In supervised learning, the set of input and output values ​​provided is called training data.

[0058] Referring to FIG. 3, the multilayer neural network (122) may include an input layer, one or more hidden layers, and an output layer.

[0059] In one embodiment, the multilayer neural network (122) may include an input layer that receives image data and has nodes corresponding to a plurality of grids of one or more frames, a first hidden layer that outputs by multiplying each output value of the input layer by a weight and adding a bias, a second hidden layer that outputs by multiplying each output value of the first hidden layer by a weight and adding a bias, and an output layer that outputs by multiplying each output value of the second hidden layer by a weight and outputting the result using an activation function. Although only two hidden layers are shown in FIG. 4, one or more hidden layers may include a greater number of hidden layers in addition to the first hidden layer and the second hidden layer.

[0060] For example, the activation function may be a Softmax function, but embodiments of the present invention are not limited thereto, and the activation function may be various other functions such as a LeRU function. Weights and biases may be continuously updated by supervised learning. Specifically, the output vector may be input to a loss function layer connected to the output layer. The loss function layer may output a loss value using a loss function that compares the output vector with the correct answer vector for each training data. The parameters of the multilayer neural network (122) may be supervised learning in a direction that reduces the loss value.

[0061] A learning model may include an input layer, one or more hidden layers, and an output layer. Training data is input into the input layer, passes through one or more hidden layers and the output layer to produce an output vector, and the output vector may be input into a loss function layer connected to the output layer. The loss function layer may output a loss value using a loss function that compares the output vector with the correct answer vector for each training data. The parameters of the learning model may be trained in a direction that reduces the loss value.

[0062] To this end, the image discrimination unit (120) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (100) itself where the artificial intelligence model is executed, or through a separate server.

[0063] Learning algorithms may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above. An artificial intelligence model may include multiple artificial neural network layers. An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0064] The challenge management unit (130) can create a challenge and set and manage the rules and conditions of the challenge. The challenge management unit (130) can also create a challenge in response to a user's request. Alternatively, the challenge management unit (130) can receive data regarding the creation of a challenge from an external server (not shown) and create a challenge based on this.

[0065] The challenge management unit (130) can set detailed conditions such as the goal, time limit, and number of participants for a challenge registered on the platform, or receive and store detailed conditions such as the goal, time limit, and number of participants for a challenge registered on the platform from an external source. For a challenge registered on the platform, unique participation conditions, challenge success or execution conditions, and success or execution rewards may be set.

[0066] For example, the challenge management department (130) can set the points required for a user to participate in a challenge, and when a user applies to participate in a challenge, the pre-specified points for the user can be deducted.

[0067] In one embodiment, the challenges created and managed by the challenge management unit (130) can be classified into participation-type challenges and challenge-type challenges. Participation-type challenges and challenge-type challenges differ in the user's method of participation and the criteria for determining success.

[0068] A participatory challenge refers to a challenge in which a user applies for a specific challenge, uploads a verification video to prove that they have met the conditions for performing the challenge, and the video discrimination unit (120) evaluates whether the user has participated according to the challenge criteria. A participatory challenge is recognized as having participated in the challenge when the user has met all the pre-set conditions for performing the challenge. For example, in the case of a running challenge, the user runs to meet the distance and time pre-specified for the running challenge, and then submits a video to the electronic device (100) certifying that the running was performed according to the conditions for performing the challenge, so the video discrimination unit (120) can determine whether the conditions for performing the participatory challenge have been met. Since a participatory challenge only examines whether the user has fulfilled the required conditions for performing the challenge, the user is considered to have participated in the challenge if only the conditions for performing the challenge are met.

[0069] On the other hand, a challenge-type challenge is a method in which a user applies for a specific challenge, performs a challenging mission, and proves that they have succeeded. A challenge-type challenge strictly evaluates whether the user has achieved the mission's goal, and the user transmits a video that can certify the mission's success to an electronic device (100), and the success of the challenge mission is automatically checked by a video discrimination unit (120). That is, unlike a participation-type challenge, a challenge-type challenge goes beyond simply satisfying conditions; it is recognized as successful only if the user uploads a video of successfully performing the mission and that video passes the pre-set challenge success conditions. For example, if a mission to make 10 consecutive 3-point shots is given in a basketball challenge, the user performs the mission and uploads a video of the success, and the success is determined through a review process.

[0070] Participatory challenges that can be created by the challenge management department (130) include running challenges, going to the gym challenges, and playing badminton challenges. Since participatory challenges only determine whether the challenge is performed, participation in the challenge can be verified through a relatively simple process.

[0071] Challenge-type challenges that can be generated by the challenge management department (130) include a basketball shooting challenge in which one must make 10 three-point shots during a given time, a plank challenge in which one must maintain a plank position for a certain period of time, and a machine exercise repetition challenge.

[0072] In one embodiment, the challenge type provides the user with an option to select a difficulty level, and may provide a differential reward upon successful completion of the challenge according to the selected difficulty level. For example, in the case of a basketball shooting challenge where one must make 10 three-point shots within a given time, the difficulty levels can be classified as easy, normal, and hard, and conditions can be set as follows: easy: 15 points reward, 5 minutes time limit, 10 successes; normal: 30 points reward, 3 minutes time limit, 10 successes; hard: 50 points reward, 2 minutes time limit, 10 successes. As another example, in the case of an exercise (bench press) repetition challenge, the difficulty can be classified into easy, medium, and hard, and conditions can be set as follows: Easy: 15 reward points, 10 repetitions with 50kg, Medium: 30 reward points, 10 repetitions with 70kg, Hard: 50 reward points, 10 repetitions with 100kg.

[0073] In one embodiment, the number of times a challenge can be attempted may be limited. For example, the number of attempts may be limited to once a day or three times a week.

[0074] The reward management department (140) can establish and manage a reward system that provides points and rewards to users who successfully complete a challenge.

[0075] For example, the reward management department (140) can award a pre-specified point to the user when the user succeeds in a challenge.

[0076] The database (150) can comprehensively store and manage learning data, standard data, and analysis data necessary for various sports multi-content services provided by the electronic device (100). The database (150) can be linked with the user DB (112) to establish a data foundation for providing customized services tailored to the characteristics of individual users.

[0077] In one embodiment, the database (150) can be constructed by storing a dataset including an abnormal behavior detection dataset, a specialized dataset by sport, an emotion analysis standard dataset, an exercise tendency classification dataset, a growth analysis dataset, a game prediction model dataset, and a voice command metadata set.

[0078] Here, the abnormal behavior detection dataset may include training data for identifying abnormal user behavior patterns such as falling, leaving one's seat, slumping, and abrupt cessation of movement.

[0079] Specialized datasets for each sport may include data regarding the unique rules, major events, game progression, and highlight generation criteria of each sport, such as soccer, basketball, tennis, and baseball.

[0080] The sentiment analysis reference dataset may include reference data that maps biosignals, such as facial expressions, brainwaves, and heart rate, to the user's emotional state.

[0081] The exercise tendency classification dataset may include classification criteria for recommending customized challenges by analyzing individual exercise patterns, preferred sports, difficulty levels, and participation history.

[0082] The growth analysis dataset may include age- and gender-specific data on physical development indicators such as muscular endurance, flexibility, power, and coordination for children, as well as reference data for analyzing growth trends.

[0083] In one embodiment, the database (150) may be implemented using a relational database management system (RDBMS) such as Oracle, Informix, Sybase, DB2, or an object-oriented database management system (OODBMS) such as Gemston, Orion, O2.

[0084] The sensing processing unit (160) can collect and process various biosignals and behavioral data of the user in real time to provide advanced services such as emotion analysis, motion correction, and abnormal behavior detection. The sensing processing unit (160) can acquire and analyze additional sensing data in addition to the image data collected from the shooting system (300) and the user terminal (200).

[0085] To this end, the sensing processing unit (160) may include a biosignal sensing module, an eye tracking sensing module, a motion recognition sensing module, a voice recognition module, and an emergency detection module.

[0086] The biosignal sensing module may include at least one of a facial recognition sensor for analyzing a user's facial expressions, an EEG sensor for measuring brain waves, and a heart rate sensor for measuring heart rate. The biosignal sensing module can determine the user's emotional state in real time based on sensing data from the sensors.

[0087] The eye-tracking sensing module detects the user's gaze direction and level of concentration to identify the point of interest and gaze of the user in multi-angle video.

[0088] The motion recognition sensing module can analyze the user's accurate posture and movement using 3D motion capture technology, detect incorrect posture during atypical movements, and provide real-time corrective feedback.

[0089] The voice recognition module can recognize the user's voice commands and process them using natural language to provide functions for searching specific video segments and controlling playback.

[0090] The emergency detection module can automatically detect emergency situations based on the results of abnormal behavior detection and, if necessary, send automatic notifications to medical personnel or emergency services.

[0091] In one embodiment, the sensing processing unit (160) can fuse data collected from a plurality of sensing modules in real time to perform more accurate user status determination.

[0092] The personalization unit (170) can provide customized content and services by analyzing the individual characteristics, exercise history, preferences, and tendencies of the user. To this end, the personalization unit (170) may include an emotion analysis engine, a game prediction AI, a motion correction engine, a growth analysis engine, a natural language processing engine, a customized recommendation system, and an adaptive learning module.

[0093] The emotion analysis engine can analyze biometric data collected from the sensing processing unit (160) in real time to identify the user's current emotional state and automatically recommend customized highlight videos or challenges accordingly.

[0094] Match prediction AI can predict match results by comprehensively analyzing player stats, team strength, and recent performance, and generate a summary video comparing the predicted result with actual match footage.

[0095] The motion correction engine analyzes the user's exercise movements in real time to detect abnormal posture during atypical exercises, such as stretching and flexibility exercises, and can recommend methods to correct the posture.

[0096] The growth analysis engine can periodically analyze exercise performance data of youth users to visualize physical development trends and generate personalized coaching content.

[0097] In one embodiment, the personalization unit (170) can automatically adjust the difficulty of the challenge according to the user's exercise ability level to maintain an appropriate sense of challenge.

[0098] The multimedia management unit (180) can generate an image of at least one of VR, AR, and XR based on original image data collected from the shooting system (300) and provide it to the user terminal (200).

[0099] The multimedia management unit (180) may include a virtual image generation module, a multi-angle synchronization engine, a real-time highlight editor, a prediction-actual comparison image generator, and a highlight generator for each item.

[0100] The virtual video generation module can render key scenes of a sports game in a 360-degree stereoscopic view. For example, the virtual video generation module can generate VR, AR, and XR videos that vividly recreate key moments, such as goal scenes and blocking scenes, from multiple angles.

[0101] The multi-angle synchronization engine performs time synchronization of images collected from the first camera (310), the second camera (320), and the third camera (330), and can generate personalized highlights in real time at the point in time and focus desired by the user based on the eye tracking data of the sensing processing unit (160).

[0102] The real-time highlight editor can automatically extract and edit key moments of each sport, such as soccer goals, basketball dunks, and tennis aces, by utilizing an automatic editing algorithm that reflects the characteristics of each sport.

[0103] The prediction-actual comparison video generator can compare and analyze the game prediction AI results of the personalization unit (170) with the actual game video to generate a summary video centered on key scenes different from the prediction.

[0104] In one embodiment, the multimedia management unit (180) can immediately search for and play a video segment corresponding to a specific player, a specific time period, or a specific event according to the user's voice command.

[0105] In one embodiment, the multimedia management unit (180) can accurately interpret natural language commands such as "the most impressive goal scene in today's match" or "second half comeback situation" by linking the metadata of the video with the natural language processing results, and provide the corresponding video.

[0106] The electronic device (100) may have the same configuration as a conventional web server or WAP server in terms of hardware. However, in terms of software, it may include program modules that perform various functions and are implemented through any language such as C, C++, Java, Visual Basic, Visual C, etc. Furthermore, the electronic device (100) generally refers to a computer system that is connected to an unspecified number of clients and / or other servers through an open computer network such as the Internet, receives requests for task execution from clients or other servers, and derives and provides the results of the task, and computer software (server program) installed for that purpose. In addition, the electronic device (100) should be understood as a broad concept that includes, in addition to the aforementioned server program, a series of application programs operating on the electronic device (100) and, in some cases, various databases (DB: Database, hereinafter referred to as "DB") built internally or externally.

[0108] FIG. 4 is a block diagram for explaining the structure of the user terminal (200) of FIG. 1.

[0109] Referring to FIG. 4, the user terminal (200) may include a processor (210), memory (220), communication module (230), antenna module (240), touch display (250), and camera module (260). In one embodiment, at least one of these components may be omitted from the user terminal (200), or one or more other components may be added. In another embodiment, some of these components may be integrated into a single component.

[0110] The processor (210) can, for example, execute software to control at least one other component of a user terminal (200) connected to the processor (210) and perform various data processing or operations. In one embodiment, as at least part of the data processing or operations, the processor (210) can store commands or data received from other components in volatile memory (221), process the commands or data stored in volatile memory (221), and store result data in non-volatile memory (222). In one embodiment, the processor (210) may include a main processor (211) (e.g., a central processing unit or an application processor) or an auxiliary processor (212) that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), a sensor hub processor). For example, if the user terminal (200) includes a main processor (211) and an auxiliary processor (212), the auxiliary processor (212) may be configured to use less power than the main processor (211) or to be specialized for a designated function. The auxiliary processor (212) may be implemented separately from the main processor (211) or as part thereof.

[0111] The auxiliary processor (212) can control at least some of the functions or states associated with at least one of the components of the user terminal (200), for example, on behalf of the main processor (211) while the main processor (211) is in an inactive state, or together with the main processor (211) while the main processor (211) is in an active state. In one embodiment, the auxiliary processor (212) may be implemented as part of another functionally related component.

[0112] In one embodiment, if the auxiliary processor (212) is a neural network processing device, the auxiliary processor (212) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the user terminal (200) itself where the artificial intelligence model is executed, or through a separate server (e.g., electronic device (100)).

[0113] Learning algorithms may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above. An artificial intelligence model may include multiple artificial neural network layers. An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0114] The memory (220) can store various data used by at least one component of the user terminal (200). The data may include, for example, input data or output data for software and related commands. The memory (220) may include volatile memory (221) or non-volatile memory (222).

[0115] The communication module (230) can support the establishment of a wired or wireless communication channel between a user terminal (200) and an external electronic device (e.g., electronic device (100)), and the performance of communication through the established communication channel. The communication module (230) operates independently of the processor (210) and may include one or more communication processors that support wired or wireless communication. In one embodiment, the communication module (230) may include a wireless communication module or a wired communication module. The corresponding communication module among these communication modules can communicate with the electronic device (100) through a network. These various types of communication modules may be integrated into a single component or implemented as multiple separate components.

[0116] The wireless communication module can support 5G networks following 4G networks and next-generation communication technologies, such as new radio access technology (NR access technology). NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module can support high-frequency bands, for example, to achieve high data transmission rates. The wireless communication module can support various technologies to secure performance in high-frequency bands, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beamforming, or large-scale antenna. The wireless communication module can support various requirements specified in the user terminal (200), external electronic device, or network system. In one embodiment, the wireless communication module can support a Peak data rate for eMBB realization, lossy coverage for mMTC realization, or U-plane latency for URLLC realization.

[0117] The antenna module (240) can transmit a signal or power to an external electronic device or receive it from the outside. In one embodiment, the antenna module (240) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate. In one embodiment, the antenna module (240) may include a plurality of antennas. In this case, at least one antenna suitable for a communication method used in a communication network, such as a network, may be selected from the plurality of antennas, for example, by a communication module (230). The signal or power may be transmitted or received between the communication module (230) and the external electronic device through the selected at least one antenna.

[0118] In one embodiment, commands or data may be transmitted or received between electronic devices (100) connected to a network. In one embodiment, all or part of the operations executed at a user terminal (200) may be executed at an electronic device (100).

[0119] For example, when a user terminal (200) needs to perform a function or service automatically or in response to a request from a user or another device, the user terminal (200) may request one or more external electronic devices to perform at least a part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the request may perform at least a part of the requested function or service, or additional functions or services related to the request, and transmit the result of the execution to the user terminal (200). The user terminal (200) may provide the result as is or additionally processed and as at least part of the response to the request. To this end, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The user terminal (200) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing.

[0120] Here, the network refers to a connection structure capable of exchanging information between each node, such as terminals and servers, or a network connecting a user terminal (200) and an electronic device (100). The network includes, but is not limited to, the Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), 3G, 4G, LTE, 5G, Wi-Fi, etc. The network may be a closed network such as a LAN or WAN, but it is preferable that it be an open network such as the Internet. The Internet refers to a global open computer network structure that provides the TCP / IP protocol and various services existing in the upper layer, namely HTTP (HyperText Transfer Protocol), Telnet, FTP (File Transfer Protocol), DNS (Domain Name System), SMTP (Simple Mail Transfer Protocol), SNMP (Simple Network Management Protocol), NFS (Network File Service), and NIS (Network Information Service).

[0121] The touch display (250) may be an LCD (liquid crystal display) panel, an OLED (organic light emitting diode) panel, etc. The touch display may also include a touch screen panel (TSP) that includes a touch electrode for detecting a touch on the display panel. In one embodiment, the touch display may be an in-cell type in which the display panel and the touch panel are combined integrally. However, this is exemplary and the embodiments of the present invention are not limited thereto. The touch display may electrically recognize contact such as a finger or pen, generate an electrical signal, and transmit it to a processor.

[0122] The camera module (260) can capture still images and video. In one embodiment, the camera module (260) may include one or more cameras. Each of the one or more cameras may include one or more lenses, image sensors, image signal processors, or flashes. Depending on the needs, the camera module (260) may include one camera or two or more different types of cameras. The camera module (260) can generate original image data by capturing an authentication video for challenge participation authentication.

[0124] FIG. 5 is a block diagram for explaining the structure of the sensor device (400) of FIG. 1.

[0125] Referring to FIG. 5, the sensor device (400) may be composed of wearable devices that collect a user's biometric information and may include a heart rate sensor (410), an accelerometer (420), a gyroscope sensor (430), a GPS module (440), a bioelectric sensor (450), a temperature sensor (460), a pressure sensor (470), and an oxygen saturation sensor (480). The sensor device (400) according to one embodiment of the present invention may include all of the above components or selectively configure only some of the sensors as needed, depending on the characteristics of the exercise or the purpose of monitoring.

[0126] The heart rate sensor (410) is a sensor that monitors the user's heart rate in real time and may use at least one of the photoplethysmography (PPG) method or the electrocardiogram (ECG) method. The heart rate sensor (410) detects changes in blood flow using green light (425 nm) and infrared (940 nm) LEDs and can measure a heart rate range of 30 to 220 beats per minute with an accuracy of within ±2 bpm.

[0127] Specifically, the heart rate sensor (410) has a built-in 16-bit ADC (Analog-to-Digital Converter) capable of acquiring high-resolution signals of more than 1,000 samples per second, and can remove motion artifacts in real time through a DSP (Digital Signal Processing) chip. The heart rate sensor (410) can evaluate autonomic nervous system activity through Heart Rate Variability (HRV) analysis and calculate HRV indicators such as RMSSD, pNN50, and SDNN to quantify the user's stress state and fatigue level.

[0128] In one embodiment, the heart rate sensor (410) can detect arrhythmias such as atrial fibrillation by measuring a single-lead electrocardiogram for 30 seconds in a manner similar to the Digital Crown-based ECG sensor of the Apple Watch. In another embodiment, the heart rate sensor (410) is implemented in the form of a chest strap such as Polar H10 to achieve a heart rate measurement accuracy of 99% or higher. In yet another embodiment, the heart rate sensor (410) can simultaneously measure blood hemoglobin concentration and oxygen saturation using a multi-wavelength LED array.

[0129] The acceleration sensor (420) is a MEMS (Micro-Electro-Mechanical Systems) based sensor that measures the user's movement and posture changes in three axes (X, Y, Z axes) and can support a measurement range from ±2g to ±16g. The acceleration sensor (420) supports a sampling frequency of up to 8kHz to precisely detect high-speed motion or impact, and can measure minute changes in acceleration of less than 0.001g through 16-bit resolution.

[0130] Specifically, the acceleration sensor (420) can be implemented with a high-performance IMU chip such as the Bosch BMI270 or STMicroelectronics LSM6DSO, and can provide temperature compensation and vibration immunity, enabling stable measurements in various environments. The acceleration sensor (420) can automatically classify the user's posture (standing, sitting, lying down, walking, running) through gravity vector analysis and identify the type of exercise through an activity pattern recognition algorithm.

[0131] The acceleration sensor (420) has a fall detection algorithm built in to detect a sudden change in acceleration (more than 3g) and a zero-gravity state (less than 0.5g) continuously, and can determine that it is a fall. In addition, the acceleration sensor (420) can provide additional functions such as counting steps, estimating calorie consumption, and analyzing sleep stages.

[0132] In one embodiment, the acceleration sensor (420) is equipped with a machine learning-based activity recognition model and can classify more than 20 types of exercise movements with an accuracy of more than 95%. In another embodiment, the acceleration sensor (420) can detect muscle tremors or spasms through vibration pattern analysis to detect neurological abnormalities early. In yet another embodiment, the acceleration sensor (420) is designed with a waterproof structure that operates even during underwater exercise and can be applied to water sports such as swimming and diving.

[0133] The gyroscope sensor (430) is a sensor that measures the user's rotational motion and angular velocity in three axes and can support a measurement range from ±250 dps to ±2000 dps (degrees per second). The gyroscope sensor (430) provides 16-bit resolution and a sampling frequency of up to 8 kHz, so it can accurately track fast rotational motions, and through a temperature drift compensation function, it can enable stable measurements for a long time.

[0134] Specifically, the gyroscope sensor (430) can be implemented in a form integrated into a 6-axis or 9-axis IMU such as InvenSense ICM-20948 or Bosch BMI260, and can improve attitude estimation accuracy through sensor fusion with the accelerometer sensor (420). The gyroscope sensor (430) can prevent gimbal lock and provide stable 3D orientation information by applying a quaternion-based attitude estimation algorithm and a Kalman filter.

[0135] The gyroscope sensor (430) can analyze the user's rotational movement pattern to precisely measure the trajectory and speed of complex movement actions such as golf swings, tennis strokes, and baseball batting. Additionally, the gyroscope sensor (430) can calculate static and dynamic balance indices for evaluating the sense of balance and detect dizziness or balance disorders early.

[0136] In one embodiment, the gyroscope sensor (430) can provide relative direction information without a magnetic sensor by applying the Attitude and Heading Reference System (AHRS) algorithm. In another embodiment, the gyroscope sensor (430) can provide diagnostic assistance information to medical staff by quantifying hand tremors, which are an early symptom of Parkinson's disease, through vibration analysis. In yet another embodiment, the gyroscope sensor (430) can be utilized as a head tracking sensor for VR / AR applications to provide an immersive exercise experience.

[0137] The GPS module (440) is a module that tracks the user's exact location and movement path through a satellite navigation system and can support multiple GNSS (Global Navigation Satellite Systems) such as GPS, GLONASS, Galileo, BeiDou, and QZSS. The GPS module (440) provides positional accuracy within a minimum of 1 meter, and can achieve centimeter-level precision when differential GPS (DGPS) or RTK (Real-Time Kinematic) technology is applied.

[0138] Specifically, the GPS module (440) can be implemented as a multi-band GNSS receiver such as the u-blox M8, M9 series or Broadcom BCM47755, and can correct ionospheric errors by simultaneously receiving L1, L2, and L5 frequency bands. The GPS module (440) supports an update frequency of 10 Hz or more per second, enabling accurate position tracking even during high-speed movement, and can provide fast satellite acquisition times within 5 seconds for a Cold Start and within 1 second for a Hot Start.

[0139] The GPS module (440) can calculate and store the user's exercise path, distance traveled, average speed, maximum speed, altitude change, etc. in real time. In addition, the GPS module (440) can automatically generate an alert when the user moves out of a pre-set safe zone through a geo-fencing function, and can transmit accurate location information to a rescue team in case of an emergency.

[0140] In one embodiment, the GPS module (440) can reduce satellite acquisition time even indoors or in urban areas by utilizing auxiliary data from a cellular network through an Assisted GPS (A-GPS) function. In another embodiment, the GPS module (440) can enable continuous location tracking even in tunnels or underground sections where GPS signals are blocked through fusion with an Inertial Navigation System (INS). In yet another embodiment, the GPS module (440) can provide performance analysis by section by linking with detailed map information such as golf courses, running tracks, and cycling routes as a sports-specific function.

[0141] The bioelectric sensor (450) is a sensor that measures the user's electromyography (EMG), electroencephalography (EEG), and electrodermal activity (EDA), and can quantify muscle activity and mental stress state. The bioelectric sensor (450) has a 24-bit ADC and a noise level of 1 Hz or less, so it can accurately detect even minute biosignals, and can have a built-in 50Hz / 60Hz power noise removal filter.

[0142] The temperature sensor (460) is a dual sensor that simultaneously measures the user's body temperature and the ambient temperature, and can support a measurement accuracy of ±0.1°C and a measurement range of -40°C to 125°C. The temperature sensor (460) can detect danger signals such as fever, dehydration, and heatstroke at an early stage by analyzing the body temperature change pattern, and can calculate a heat stress index based on exercise intensity.

[0143] The pressure sensor (470) is a sensor that detects contact pressure applied by a user along with altitude estimation through atmospheric pressure measurement, and can measure altitude with an accuracy of ±1m with a resolution of 1 hPa or less. The pressure sensor (470) can monitor the risk of altitude sickness in sports with large altitude changes such as climbing, skiing, and paragliding, and can evaluate walking patterns and balance through plantar pressure analysis.

[0144] The oxygen saturation sensor (480) is a sensor that non-invasively measures blood oxygen saturation using pulse oximetry and can provide an accuracy of ±2% in the range of 70% to 100%. The oxygen saturation sensor (480) calculates the ratio of oxidized hemoglobin to reduced hemoglobin using the difference in absorption rates between red light (660 nm) and infrared light (940 nm), and can detect hypoxia or respiratory disease early.

[0145] All sensors of the sensor device (400) are connected to the electronic device (100) via wireless communication of BLE (Bluetooth Low Energy) 5.0 or higher, and can provide adaptive sampling frequency adjustment and sleep mode switching functions to optimize battery life. The collected multi-sensor data undergoes time synchronization and calibration processes in the electronic device (100) and is fused with the video analysis results to enable more accurate and reliable detection of abnormal behavior.

[0147] FIG. 6 is a flowchart illustrating the process of preprocessing original image data according to an embodiment of the present invention. FIG. 7 is a diagram schematically illustrating the process of dividing a plurality of original image data into frames.

[0148] First, in step S610, the image preprocessing unit (121) of the electronic device (100) can divide the input original image data (v_raw) into frames.

[0149] In step S620, the image preprocessing unit (121) of the electronic device (100) can generate preprocessed data by normalizing each frame (f). The electronic device (100) can normalize each frame of the original image data by adjusting it to the same pixel size. For example, if the original image data is a 90-minute video captured at 30 frames per second (fps), a total of 162,000 frames can be generated.

[0150] In step S630, the image preprocessing unit (121) of the electronic device (100) can divide the preprocessed data into a plurality of consecutive frames to generate a continuous frame set (cf).

[0151] In one embodiment, a continuous frame set (cf) may be defined as a collection of frames captured at consecutive time intervals. For example, frames constituting a video over one second may be defined as a single "continuous frame set."

[0152] Referring to FIG. 7, it is exemplarily illustrated that a first continuous frame set (cf1) including f1, f2, and f3 among a plurality of frames (f) generated by preprocessing the first original image data (v_raw1) and a second continuous frame set (cf2) including f1, f2, and f3 among a plurality of frames (f) generated by preprocessing the second original image data (v_raw2) are shown. In FIG. 7, L1 and L2 represent a first region of interest and a second region of interest, respectively.

[0153] In another embodiment, the electronic device (100) may generate a continuous frame set based on the time of occurrence of a specific event. The specific event may be, for example, a scoring scene in soccer, a shooting motion in basketball, abnormal behavior, sudden behavior, emergency behavior, etc. In this case, the image preprocessing unit (121) may detect the occurrence of a specific event in conjunction with other components (e.g., a sensing processing unit (160)), and generate a continuous frame set including a frame preceding the detected frame and a frame following the detected frame based on the frame (e.g., f2) in which the occurrence of the specific event is detected.

[0154] In another embodiment, the electronic device (100) can generate a continuous frame set based on a section in which a specific object (e.g., ball, player, goalpost, racket, etc.) is recognized or tracked based on an object recognition and object tracking algorithm. Similarly, the image preprocessing unit (121) can recognize a specific object in conjunction with another component (e.g., sensing processing unit (160)) and generate a continuous frame set including a frame preceding the recognized frame and a frame following the recognized frame, based on the frame in which the specific object is recognized (e.g., f2).

[0155] The electronic device (100) can extract a video clip corresponding to preset operation information or event information from a generated continuous frame set. The electronic device (100) can divide a video segment into video clips of a specific time unit.

[0157] FIG. 8 is a flowchart illustrating a method for generating a highlight image based on statistical data according to an embodiment of the present invention.

[0158] Referring to FIG. 8, first, in step S810, the electronic device (100) receives original image data from a shooting system (200) including a plurality of cameras.

[0159] A plurality of cameras may include a first camera (210) installed to recognize a user's face area and photograph the face area, a second camera (220) installed to recognize a user's full body area at a different angle from the first camera (210) and photograph the full body area, and a third camera (230) installed to photograph the user (P) in a different direction from the first camera (210) and the second camera (220).

[0160] The original image data may include first original image data obtained from a first camera, second original image data obtained from a second camera, and third original image data obtained from a third camera.

[0161] In step S820, the electronic device (100) can divide the input first to third original image data into frames and normalize each frame (f) to generate first preprocessed data including a plurality of first continuous frame sets, second preprocessed data including a plurality of second continuous frame sets, and third preprocessed data including a plurality of third continuous frame sets.

[0162] In one embodiment, a continuous frameset may be defined as a collection of frames captured at consecutive time intervals. For example, frames constituting a video over one second may be defined as a single "continuous frameset." In another embodiment, the electronic device (100) may generate a continuous frameset based on the occurrence of a specific event. The specific event may be, for example, a scoring scene in soccer, a shooting motion in basketball, etc. In yet another embodiment, the electronic device (100) may generate a continuous frameset based on a section in which a specific object (e.g., ball, player, goalpost, racket, etc.) is recognized or tracked based on an object recognition and object tracking algorithm.

[0163] The electronic device (100) can extract a video clip corresponding to preset operation information or event information from a generated continuous frame set. The electronic device (100) can divide a video segment into video clips of a specific time unit.

[0164] A continuous frame set may include a first continuous frame set generated by dividing and normalizing the first original image data into frame units, and a second continuous frame set generated by dividing and normalizing the second original image data into frame units.

[0165] In step S830, the electronic device (100) extracts visual feature information based on a CNN (Convolutional Neural Network) model for the first to third consecutive frame sets.

[0166] The first to third consecutive frame sets can be input into a CNN model. The CNN model can learn and extract visual features (e.g., motion, color patterns, shape of objects, etc.) for each frame of the consecutive frame set. In this process, the electronic device (100) can vectorize the visual features into a virtual vector space to generate multiple feature vectors for each consecutive frame set.

[0167] In one embodiment, the electronic device (100) may specify a first frame among a plurality of frames of preprocessed data and specify a second frame that follows the first frame in order to generate a plurality of feature vectors. Here, 'first' and 'second' should be understood as referring to a specific frame among the plurality of frames and should not be interpreted as being limited to the meaning of the first and second. However, the meaning that the second frame follows the first frame should be understood as the second frame being a frame that is generated later (after) in time series than the first frame. The second frame may be a frame that follows immediately after the first frame, or it may be a frame that follows with one or more frames in between.

[0168] The electronic device (100) can specify a region of interest (L) for each of a plurality of frames. Then, the electronic device (100) can generate a feature vector based on the difference in pixel brightness of the region of interest (L) between the first frame and the second frame.

[0169] Here, 'region of interest' may refer to an area located at a specific coordinate and of a specific size within a frame. Referring to FIG. 7, a 3x3 region of interest (L) may be exemplarily set at a specific location within the first frame (f1). The coordinates of the region of interest may be defined based on the coordinates of the center pixel of the region of interest.

[0170] In one embodiment, the region of interest may be set based on reference coordinates within the frame. The reference coordinates may be, for example, the center coordinates of the frame or the edge coordinates of the frame.

[0171] In another embodiment, the region of interest may be set based on a specific event. That is, the region of interest may be set according to a specific event (e.g., scoring, passing, dribbling, etc.) detected in the previous frame or the current frame.

[0172] In another embodiment, the region of interest may be set based on a detected object (target). The electronic device (100) may detect the location and movement of an object using an object tracking algorithm or an object tracking filter, and set the area around the object as the region of interest.

[0173] In another embodiment, the region of interest may be set based on the rate of change of pixel values. That is, the region of interest may be set based on an area within a specific frame where the change in pixel values ​​is large (e.g., an area with fast movement or change).

[0174] If the pixel brightness value of the first frame is I1(x,y,t) and the pixel brightness value of the second frame is I2(x+dx, y+dy, t+dt), then <Equation 1> can be satisfied according to the brightness conservation assumption.

[0175]

[0176] In <Equation 1>, I1(x,y,t) represents the pixel brightness value at coordinates (x,y) at time t, dx and dy represent the distance the pixel moved in the x and y directions, respectively, and dt represents the time difference between two frames.

[0177] Equation 1 can be expressed as a Taylor series and simplified as follows.

[0178]

[0179] In <Equation 2>, u=dx / dt and v=dy / dt, where u represents the x-direction component of the optical flow and v represents the y-direction component of the optical flow.

[0180] In one embodiment, the electronic device (100) can construct a linear equation system by applying <Equation 2> to all pixels within the region of interest and generate a feature vector based on the linear equation system.

[0181] The electronic device (100) can form a linear equation system as in <Equation 2>.

[0182]

[0183] In <Equation 3>, Ix, Iy, and It represent derivative values ​​with respect to the x-direction, y-direction, and time (t), respectively, and p1, p2, … , pn may represent pixel coordinates within the region of interest.

[0184]

[0185] In one embodiment, the electronic device (100) can calculate a feature vector using <Equation 4>. In <Equation 4>, A may represent a matrix containing brightness values ​​of all pixels within the region of interest, and b may represent a feature vector.

[0186] In step S840, the electronic device (100) inputs a plurality of feature vectors obtained from a CNN model into a Recurrent Neural Network (RNN) model to perform sequence analysis and generates a highlight video clip including a portion of a first continuous frame set and a portion of a second continuous frame set based on the plurality of feature vectors.

[0188] FIG. 9 is a flowchart illustrating a method for providing customized video for each sports player based on stat data according to an embodiment of the present invention.

[0189] Referring to FIG. 9, in S910, the electronic device (100) can receive original image data from the shooting system (200) and generate preprocessed data.

[0190] Here, the original video data may be a game video in which multiple users, including the first user and the second user, all appear. For example, in the case of a soccer game, it may be a video in which 22 players and a referee appear together, and in the case of a basketball game, it may be a video in which 10 players appear. The electronic device (100) may divide the original video data into frames and normalize each frame to generate preprocessed data containing multiple frames. The normalization process may include a process of unifying the resolution of the frames and adjusting brightness and color to generate video data of consistent quality.

[0191] In S920, the electronic device (100) receives user information for a first user from a user terminal (300) and can receive first statistical data for the first user from an external server (not shown). The user information for the first user and the first statistical data are as described above.

[0192] In S930, the electronic device (100) can identify a first user for each of multiple frames of original image data. To do this, the electronic device (100) can detect human objects in each frame by utilizing an object detection model such as YOLO (You Only Look Once) v5 or Detectron2. The electronic device (100) can identify the first user by performing uniform number recognition on the detected human objects. Uniform number recognition can utilize Optical Character Recognition (OCR) technology or a CNN model specialized in number recognition.

[0193] In one embodiment, the electronic device (100) can identify a player by utilizing uniform colors and numbers together. The uniform colors for each team can be predefined in the HSV (Hue, Saturation, Value) color space, and the main color can be extracted from the upper body area of ​​the detected object to distinguish the team. Subsequently, a specific player can be identified by recognizing the number within the corresponding team.

[0194] In another embodiment, the electronic device (100) may utilize face recognition technology as an auxiliary tool. By using a face recognition model such as FaceNet or ArcFace, the player's face features can be extracted from a close-up frame and compared with the face features of a first user registered in advance to increase identification accuracy.

[0195] In S940, the electronic device (100) can set a region of interest for each of the multiple frames included in the preprocessed data based on the first statistical data. Here, if the first statistical data includes the first user's past game records, main activity zone, scoring location, pass success rate, etc., the electronic device (100) can analyze these elements to predict the zone where the first user is mainly active and the area where important plays are likely to occur.

[0196]

[0197] In <Equation 5>, ROI_size may represent the size of the region of interest, Base_size may represent the base region of interest size, w1 may represent a statistical data-based adjustment factor, and Act_rate may represent the activity frequency rate in the area. w1 may be between 0.1 and 0.5.

[0198] For example, if the first user is a soccer forward and the goal success rate inside the penalty box is 70%, when the first user enters the penalty box, if Activity_rate is set to 0.7 and w1 is set to 0.3, the ROI_size can be expanded to 1.21 times the Base_size.

[0199] In S950, the electronic device (100) can generate a target motion vector by analyzing pixel brightness differences within a set region of interest (L).

[0200] The electronic device (100) specifies a first frame among a plurality of frames of preprocessed data and specifies a second frame that follows the first frame. Here, 'first' and 'second' should be understood as referring to a specific frame among the plurality of frames and should not be interpreted as being limited to the meaning of the first and second. However, the meaning that the second frame follows the first frame should be understood as the second frame being a frame that is generated later (after) in time series than the first frame. The second frame may be a frame that follows immediately after the first frame, or it may be a frame that follows with one or more frames in between.

[0201] The electronic device (100) can specify the coordinates where the target is located for each of the multiple frames. For example, in the example above, in the first frame, the soccer ball is at coordinates (100, 150), and in the second frame, the soccer ball is at coordinates (105, 155).

[0202] The electronic device (100) can generate a target motion vector (u,v) based on the difference in pixel brightness of the region of interest (L) between the first frame and the second frame. The electronic device (100) can compare the brightness of pixels included in the region of interest (L) of the first frame (f1) and the second frame (f2), respectively. In this case, the size and location of the region of interest (L) may be the same for the first frame (f1) and the second frame (f2), respectively.

[0203] If the pixel brightness value of the first frame is I1(x,y,t) and the pixel brightness value of the second frame is I2(x+dx, y+dy, t+dt), then <Equation 1> can be satisfied according to the brightness conservation assumption.

[0204]

[0205] In <Equation 6>, I1(x,y,t) represents the pixel brightness value at coordinates (x,y) at time t, dx and dy represent the distance the pixel moved in the x and y directions, respectively, and dt represents the time difference between two frames.

[0206] Equation 6 can be expressed as a Taylor series and simplified as follows.

[0207]

[0208] In <Equation 4>, u=dx / dt and v=dy / dt, where u represents the x-direction component of the optical flow and v represents the y-direction component of the optical flow.

[0209] In one embodiment, the electronic device (100) can construct a linear equation system by applying <Equation 7> to all pixels within the region of interest and generate a target motion vector (u,v) based on the linear equation system.

[0210] The electronic device (100) can form a linear equation system as in <Equation 7>.

[0211]

[0212] In <Equation 8>, Ix, Iy, and It represent derivative values ​​with respect to the x-direction, y-direction, and time (t), respectively, and p1, p2, … , pn may represent pixel coordinates within the region of interest.

[0213]

[0214] In one embodiment, the electronic device (100) can calculate a target motion vector using <Equation 9>. In <Equation 9>, A may represent a matrix containing brightness values ​​of all pixels within the region of interest, and b may represent a target motion vector.

[0215] In one embodiment, the electronic device (100) may set a region of interest using a prediction filter that sets the expected position of a target within a plurality of frames. The prediction filter may be, for example, a Kalman filter.

[0216] In one embodiment, the electronic device (100) can generate a target motion vector based on the difference in brightness between the first frame and the third frame following the second frame when the difference in pixel brightness in the region of interest (L) between the first frame (f1) and the second frame (f2) is less than or equal to a threshold value.

[0217] In one embodiment, the electronic device (100) can detect a change in brightness again by moving the region of interest when the difference in pixel brightness in the region of interest (L) between the first frame (f1) and the second frame (f2) is less than or equal to a threshold value.

[0218] In one embodiment, the size of the region of interest may be proportional to the logarithm of the sum of the width and height of the frame. By setting the size of the region of interest to be proportional to the logarithm of W+H as described above, a change in the size of the region of interest that is mitigated with respect to changes in the frame size can be induced.

[0219] In one embodiment, the electronic device (100) can dynamically adjust the area of ​​interest by reflecting the positional characteristics of the first user. In the case of soccer, the forward can set an expanded area of ​​interest, the defender can set a left-right expanded area of ​​interest, and the midfielder can set an even area of ​​interest in all directions. In the case of baseball, the batter can set an expanded area in the direction of the batter's box and first base, and the pitcher can set a fixed area of ​​interest centered on the mound.

[0220] In another embodiment, the electronic device (100) can adjust the area of ​​interest by considering the context of the game situation. By comprehensively analyzing the position of the ball, the positions of other players, the game time, the score situation, etc., it can predict important moments and expand or contract the area of ​​interest.

[0221] The electronic device (100) can analyze the direction and speed of movement of the first user based on the target motion vector. The magnitude of the motion vector may indicate speed, and the direction may indicate the direction of movement. The electronic device (100) can remove noise and generate a smooth tracking path by applying a Kalman filter.

[0222] In S960, the electronic device (100) can select highlight frames based on the activity intensity of the first user. The activity intensity can be calculated by combining the speed of movement, acceleration, frequency of direction change, etc.

[0223] In one embodiment, the electronic device (100) can improve the accuracy of highlight selection by learning key play patterns appearing in the first user's stat data. It can learn movement patterns such as past goal scenes, assist scenes, and important defensive scenes using a Long Short-Term Memory (LSTM) network and assign high weights when similar patterns are detected.

[0224] In S970, the electronic device (100) can generate a customized highlight video for the first user by connecting selected highlight frames. Transitions between frames can be made natural by applying transition effects such as fade in / out and dissolve. Frames can be encoded using the FFmpeg library and compressed with an H.264 or H.265 codec to generate a final highlight video.

[0225] In another embodiment, the electronic device (100) can generate more dynamic highlight videos by utilizing video footage captured from multiple camera angles. It can automatically switch and edit the full game footage from the main camera and the detailed footage from the close-up camera according to the activities of the first user. Through this, broadcast-quality personalized highlight videos can be provided.

[0226] In one embodiment, the electronic device (100) can identify noise pixels using a noise filter. The noise filter may be a median filter or a Gaussian filter.

[0227] The electronic device (100) can identify noise pixels using a Gaussian filter composed of a kernel that uses a two-dimensional Gaussian distribution. Here, the kernel is a matrix used to remove noise from an image, and the Gaussian kernel is generated based on a Gaussian function and can be used to calculate a weighted average of the surrounding pixels of a given pixel.

[0228] The electronic device (100) applies a Gaussian filter to each of the multiple frames and selects noise pixels within the frames based on the filtered values.

[0229] The electronic device (100) can assign a new pixel value by calculating a weighted average of a specific pixel within a frame and the surrounding pixels of said specific pixel using a Gaussian filter. The electronic device (100) can compare the newly assigned pixel value for a specific pixel with surrounding pixels, and if the difference between the pixel value of the specific pixel and the average pixel value of the surrounding pixels is greater than or equal to a threshold value, the specific pixel can be identified as a noise pixel. Through this process, the electronic device (100) can identify a pixel with a pixel value of 255 in the region of interest (L) of the first frame (f1) shown in FIG. 4 as a noise pixel.

[0230] The filtering strength of a Gaussian filter can be determined by the matrix size (e.g., 3x3, 5x5, etc.) and standard deviation. In this case, the central element of the kernel has the largest weight, and the weight decreases exponentially as it moves away from the center.

[0231] The kernel size of a Gaussian kernel can be the same as the size of the frame's region of interest.

[0232] In one embodiment, the electronic device (100) can set the size of the region of interest based on the ratio of noise pixels to total pixels within the frame.

[0233]

[0234] In one embodiment, the electronic device (100) may set the size of the region of interest based on <Equation 10>. In <Equation 10>, N(D) may represent the size of the region of interest, D may represent the ratio of noise pixels to total pixels, f may represent the default value of the region of interest size, and c and e may represent correction factors. Here, e may have a real value greater than 0 and less than or equal to 0.01, and c may have a real value in the range of 1 to 10.

[0235] As the proportion of noise pixels within a frame increases, a larger region of interest is required for accurate object detection, so a function for the size of the region of interest according to noise density can be set as in <Equation 6>.

[0236] In one embodiment, the default value (F) of the region of interest size in <Equation 6> can be calculated by <Equation 7> below.

[0237]

[0238] In <Equation 11>, W represents the width of the frame, H represents the height of the frame, and f(W,H) represents an approximation of the size of the optimal region of interest.

[0239]

[0240] In one embodiment, the electronic device (100) can set the size of the region of interest based on <Equation 12>. In <Equation 12>, N represents the size of the region of interest, W represents the width of the frame, H represents the height of the frame, and D represents the ratio of noise pixels to total pixels. As in <Equation 12>, the diagonal length of the frame can be divided by the logarithm of the product of the noise density and the frame size to set the optimal size of the region of interest according to the noise density and the frame size.

[0241] As described above, the method for providing customized video for each sports player based on stat data according to one embodiment of the present invention enables video analysis optimized for the characteristics of the player by setting a region of interest using the player's stat data. Through this, customized highlight videos for each player can be automatically generated, thereby significantly reducing video editing time.

[0242] In addition, real-time processing is possible by adopting a motion vector generation method based on pixel brightness differences, and stable performance can be maintained in various shooting environments through a function that adjusts the region of interest size according to the ratio of noise pixels.

[0243] Furthermore, by matching and storing target motion vectors with statistical data, a player's play patterns can be databased, which can be utilized for future player performance analysis and strategy formulation. The function of automatically moving the region of interest to re-detect when a pixel brightness difference below a threshold is detected enables continuous tracking even when the player moves out of the screen or the camera angle changes.

[0244] The embodiments described above may be implemented as hardware components, software components, and / or combinations of hardware and software components. For example, the devices, methods, and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.

[0245] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0246] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0247] Although the embodiments have been described above with reference to the limited drawings, those skilled in the art can apply various technical modifications and variations based on the above. For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0248] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

Claim 1 A step of receiving original video data from a shooting system by means of an electronic device, dividing the original video data into frame units, and normalizing each frame to generate preprocessed data including multiple frames; - the original video data is a game video in which both a first user and a second user appear, and a step of receiving user information regarding the first user from a user terminal and receiving first statistical data regarding the first user from an external server by means of the electronic device, identifying the first user by utilizing an object detection model in each of the multiple frames, and setting a region of interest for each of the multiple frames based on the first statistical data; - The size of the region of interest is dynamically set by applying an adjustment coefficient reflecting the activity frequency rate based on the first statistical data to the basic region of interest size, and the user information includes at least one of the first user's name, contact information, date of birth, residence, and gender; the electronic device identifies a first frame among the plurality of frames, identifies a second frame following the first frame, and generates a target motion vector based on the pixel brightness difference of the region of interest between the first frame and the second frame; the electronic device analyzes the direction and speed of movement of the target based on the target motion vector, identifies the target to select a highlight frame, and connects the selected highlight frames to generate a highlight image; the electronic device further includes the step of matching and storing the target motion vector and the statistical data; and the electronic device adjusts the size of the region of interest based on the ratio of noise pixels to the total pixels within the frame.The method further includes the step of moving the region of interest to detect a brightness change again by means of the electronic device when the difference in pixel brightness in the region of interest between the first frame and the second frame is less than or equal to a threshold value, wherein the size of the region of interest is calculated based on the following mathematical formula [Mathematical Formula]; ROIsz represents the size of the region of interest, Basesz represents the size of the basic region of interest, w1 represents a statistical data-based adjustment coefficient, and Actrate represents the activity frequency rate in a specific zone; wherein, if the first user is a soccer forward and the goal success rate within the penalty box is 70%, when the first user enters the penalty box, the Actrate is set to 0.7 and w1 is set to 0.3, a method for providing customized video for each sports player based on statistical data. Claim 2 delete Claim 3 delete Claim 4 delete Claim 5 delete

Images