Method and device for detecting abnormal behavior-based sports video events

Abstract

The embodiments provide a method for detecting sports video events based on abnormal behavior. The method according to the embodiments may include: receiving original video data from a shooting system by an electronic device and detecting candidate frames of abnormal behavior by analyzing a user's movement pattern from the original video data; calculating an abnormal behavior score for the candidate frames of abnormal behavior based on at least one of a posture deviation value, a rate of change of speed, and a duration of an abnormal state by the electronic device, and determining a frame in which abnormal behavior occurs by comparing the abnormal behavior score with a preset threshold value; generating a continuous frame set including a preceding frame and a succeeding frame based on the frame in which abnormal behavior occurs by the electronic device; analyzing the type of abnormal behavior based on the continuous frame set by the electronic device, classifying it into a highlight video or a warning video, and generating a final video; and transmitting the final video to a user terminal via a network by the electronic device.

Description

Technology Field

[0001] Embodiments of the present invention relate to a method and apparatus for detecting sports video events based on abnormal behavior, and more specifically, to a method and apparatus for detecting sports video events based on abnormal behavior such as falling, leaving one's seat, or squatting based on a user's normal exercise pattern, and classifying the corresponding frame as a warning event. Background Technology

[0002] With the advancement of digital media and sports broadcasting environments, various technologies are being researched to automatically analyze game footage and extract highlight scenes or major events. Conventional technologies tend to focus on specific events defined by game rules, such as goals, shots, and fouls, and have limitations in that they cannot detect sudden abnormal behaviors of players or users.

[0003] For example, in fitness training videos or amateur competition videos, cases where a user suddenly falls or moves away from their seat need to be recognized as safety warning events, but conventional technology has not been able to automatically identify them. In addition, existing object recognition-based event detection methods rely only on predefined motion patterns, which has the problem of failing to reflect the unique exercise characteristics of individual users.

[0004] Therefore, a method and device are required to detect abnormal behavior based on an artificial intelligence model that has learned user patterns and classify it as a highlight or warning event. The problem to be solved

[0005] Embodiments of the present invention aim to provide a method and apparatus for detecting abnormal behavior-based sports video events to solve problems such as the limitations of the prior art. Specifically, the invention provides a method and apparatus that can simultaneously enhance the safety and immersion of sports video by detecting abnormal behavior by comparing it with a user's normal exercise pattern and classifying the detected frames as events.

[0006] The technical problems to be solved in the embodiments are not limited to those mentioned above, and other unmentioned technical problems may be considered by those skilled in the art from the various embodiments described below. means of solving the problem

[0007] A method for detecting abnormal behavior-based sports video events according to an embodiment of the present invention may include: receiving original video data from a shooting system by an electronic device and detecting abnormal behavior candidate frames by analyzing a user's movement pattern from the original video data; calculating an abnormal behavior score for the abnormal behavior candidate frames based on at least one of a posture deviation value, a rate of change of speed, and a duration of an abnormal state by the electronic device, and determining an abnormal behavior occurrence frame by comparing the abnormal behavior score with a preset threshold value; generating a continuous frame set including a preceding frame and a succeeding frame based on the abnormal behavior occurrence frame by the electronic device; analyzing the type of abnormal behavior based on the continuous frame set by the electronic device, classifying it into a highlight video or a warning video, and generating a final video; and transmitting the final video to a user terminal via a network by the electronic device.

[0008] The above-described shooting system includes a first camera, a second camera, and a third camera installed to photograph the user from different directions, and the step of detecting the abnormal behavior candidate frame includes: a step of detecting a first abnormal behavior candidate frame from first original image data generated by the first camera; a step of detecting a second abnormal behavior candidate frame from second original image data generated by the second camera; and a step of detecting a third abnormal behavior candidate frame from third original image data generated by the third camera; and the first to third abnormal behavior candidate frames may be frames captured at the same time.

[0009] The step of determining the abnormal behavior occurrence frame may include: a step of calculating the abnormal behavior score for each of the first to third abnormal behavior candidate frames; and a step of determining the entire first to third abnormal behavior candidate frames as abnormal behavior occurrence frames when two or more of the first to third abnormal behavior candidate frames have an abnormal behavior score greater than or equal to the threshold value.

[0010] <Mathematical Formula>

[0011]

[0012] The step of calculating the abnormal behavior score for each of the first to third abnormal behavior candidate frames includes the step of calculating the abnormal behavior score using the <mathematical formula>, wherein in the <mathematical formula>, A_score represents the abnormal behavior score, P_deviation represents the attitude deviation value, V_change represents the rate of change in velocity, T_duration represents the duration of the abnormal state, and w_1, w_2, and w_3 represent their respective weights.

[0013] The step of detecting the above abnormal behavior candidate frame may include: setting a region of interest centered on at least one of the user's body center, major joint area, and contact point with exercise equipment within the frame of the original image data; and calculating at least one of the posture deviation value and the rate of change of velocity within the region of interest.

[0014] The above abnormal behavior includes at least one of falling, leaving one's seat, sitting down, and abrupt cessation of movement, and the step of calculating the above abnormal behavior score may include the step of applying different weights according to the type of above abnormal behavior.

[0015] The step of transmitting the final video includes: a step of generating metadata regarding the type, time of occurrence, and severity of the abnormal behavior; and a step of transmitting the metadata together with the final video to the user terminal; and if the abnormal behavior is determined to be a negative event, the electronic device may additionally transmit a notification to emergency contacts stored in a database. Effects of the invention

[0016] According to embodiments of the present invention, a multi-camera-based sports video recording system can monitor a user's exercise situation in real time from multiple angles, and an AI-based abnormal behavior detection algorithm can be used to automatically detect abnormal situations such as falling, leaving one's seat, or sitting down. In particular, the objectivity and accuracy of abnormal behavior determination can be significantly improved through a quantitative evaluation formula that combines posture deviation values, speed change rates, and duration.

[0017] Furthermore, the multi-camera cross-verification method prevents false recognition by a single camera and enhances the reliability of abnormal behavior detection. By automatically classifying detected abnormal behavior into positive events (ceremonies) and negative events (injury risks), and generating appropriate highlight or warning videos for each, it is possible to provide customized content to users. This allows for a departure from conventional simple video recording methods, enabling the simultaneous provision of intelligent sports safety management and entertainment services, thereby ensuring exercise safety and enhancing user satisfaction. Brief explanation of the drawing

[0018] The accompanying drawings, included as part of the detailed description to aid in understanding the embodiments, provide various embodiments and explain the technical features of the various embodiments together with the detailed description. FIG. 1 is a schematic diagram illustrating a sports game video recording system (10) according to one embodiment of the present invention. FIG. 2 is a block diagram for explaining the structure of the electronic device (100) of FIG. 1. Figure 3 is a diagram illustrating the multilayer neural network (122) of Figure 2. FIG. 4 is a block diagram for explaining the structure of the user terminal (200) of FIG. 1. FIG. 5 is a block diagram for explaining the structure of the sensor device (400) of FIG. 1. FIG. 6 is a flowchart illustrating the original image data preprocessing process according to one embodiment of the present invention. Figure 7 is a diagram schematically illustrating the process of dividing multiple original image data into frames. FIG. 8 is a flowchart illustrating the process of generating a recording video based on abnormal behavior according to one embodiment of the present invention. Specific details for implementing the invention

[0019] The following embodiments are combinations of the components and features of the embodiments in a specific form. Each component or feature may be considered optional unless otherwise explicitly stated. Each component or feature may be implemented in a form not combined with other components or features. Additionally, various embodiments may be constructed by combining some components and / or features. The order of operations described in various embodiments may be changed. Some components or features of one embodiment may be included in another embodiment, or may be replaced with corresponding components or features of another embodiment.

[0020] In the description of the drawings, procedures or steps that could obscure the essence of the various embodiments were not described, nor were procedures or steps that could be understood by a person of ordinary knowledge in the relevant technical field described.

[0021] Throughout the specification, when a part is described as "comprising" or "including" a component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Furthermore, terms such as "...part," "...unit," and "module" as used in the specification refer to a unit that performs at least one function or operation, and this may be implemented in hardware, software, or a combination of hardware and software. Additionally, "one (a or an)," "one," "the," and similar related terms may be used in the context describing various embodiments (particularly in the context of the following claims) in both singular and plural forms, unless otherwise indicated in the specification or clearly contradicted by the context.

[0022] Hereinafter, embodiments according to various examples will be described in detail with reference to the accompanying drawings. The detailed description disclosed below, together with the accompanying drawings, is intended to describe exemplary embodiments of various examples and is not intended to represent the only embodiment.

[0023] In addition, specific terms used in various embodiments are provided to aid in understanding the various embodiments, and the use of such specific terms may be modified in other forms within the scope of not departing from the technical concept of the various embodiments.

[0024] FIG. 1 is a schematic diagram illustrating a sports game video recording system (10) according to one embodiment of the present invention.

[0025] Referring to FIG. 1, the electronic device (100) can receive original image data in real time from the shooting system (300).

[0026] The shooting system (300) may include a plurality of cameras. The plurality of cameras may include a first camera (310), a second camera (320), and a third camera (330). As illustrated in FIG. 1, the first camera (310), the second camera (320), and the third camera (330) may each be installed to photograph the user (P) from different directions. Each of the plurality of cameras may capture still images and video of the user (P). In one embodiment, each of the plurality of cameras may include one or more lenses, image sensors, image signal processors, or flashes. Although FIG. 1 illustrates the shooting system (300) as including three cameras, embodiments of the present invention are not limited thereto. If necessary, the shooting system (300) may include four or more cameras, or two or fewer cameras. However, in order to acquire image data of the user (P) from different angles, it is preferable for the shooting system (300) to include three or more cameras.

[0027] Each of the plurality of cameras included in the shooting system (300) can capture the stationary state or operating state of the user (P) to generate original image data (v_raw). The original image data (v_raw) may refer to data in which the stationary state or moving state of the user (P) is optically recorded.

[0028] In one embodiment, the shooting system (300) may include a first camera (310), a second camera (320) installed to photograph a user (P) from a different direction from the first camera (310), and a third camera (330) installed to photograph a user (P) from a different direction from the first camera (310) and the second camera (320).

[0029] The first camera (310) can capture the user (P) to generate the first original image data (v_raw1). The second camera (320) can capture the user (P) to generate the second original image data (v_raw2). The third camera (330) can capture the user (P) to generate the third original image data (v_raw3). Each image data may include metadata including time information corresponding to the time of image capture, a shooting area, and camera coordinate information. The metadata may be stored in formats such as IPTC (International Press Telecommunications Council), XMP (Extensible Metadata Platform from Adobe), and EXIF ​​(Exchangeable Image File Format).

[0030] In one embodiment, the electronic device (100) provides a membership registration interface to a user terminal (200) and receives user information corresponding to the membership registration interface from the user terminal (200), thereby registering the user to a sports multi-content system implemented by the electronic device (100). The sports multi-content system implemented by the electronic device (100) may include at least one of a multi-angle-based highlight video generation service, a social network-based challenge participation competition service, an abnormal behavior-based sports video event detection service, an emotion feedback-based AI sports video recommendation service, an individual sports discipline-specific highlight automatic generation service, a customized challenge video recommendation service based on individual exercise tendencies, an AI-based game prediction result-linked video summary generation service, a user voice command-based real-time highlight search service, a VR environment immersive sports highlight provision service, a real-time motion correction service based on unstructured exercise data, and a youth exercise data-based growth analysis and coaching video provision service.

[0031] In one embodiment, the electronic device (100) can implement a social network-based challenge participation competition system. The electronic device (100) can induce continuous participation by setting clear rules for the challenges in which users participate, establishing an authentication system capable of fairly judging them, and providing substantial rewards to users. The challenges provided on the challenge participation competition platform implemented by the electronic device (100) can be classified into participation-type challenges and challenge-type challenges. Each challenge may have unique participation conditions, challenge success or execution conditions, and success or execution rewards set.

[0032] In one embodiment, the electronic device (100) can determine whether a user participates in a challenge based on video data captured by a user terminal (200) and determine whether to pay a reward for participating in the challenge.

[0033] In one embodiment, the electronic device (100) may collect original image data from a plurality of user terminals (200). In this case, the original image data (v_raw) may refer to data optically recorded by capturing the stationary or operational state of a user (e.g., athlete, training coach, amateur, etc.) or an object by the camera module (260) of the user terminal (200). The original image data may include metadata including time information corresponding to the time of image capture, a shooting area, and camera coordinate information. The metadata may be stored in formats such as IPTC (International Press Telecommunications Council), XMP (Extensible Metadata Platform from Adobe), and EXIF ​​(Exchangeable Image File Format).

[0034] In one embodiment, the electronic device (100) may collect statistical data from an external server (not shown). Statistic data may refer to factual information recording the performance of a user. For example, when a user plays soccer, there are goals, assists, passes, sprints, aerial duels, etc., and in the case of a batter in baseball, there are hits, home runs, ground balls, flies, bunts, singles, doubles, triples, fouls, etc. The statistical data collected by the electronic device (100) from the external server may be collected in a database format determined according to the sport and the player. Here, the external server may be an official website of a sports league such as baseball, basketball, or soccer, or a data provider (API). In this specification, statistical data may refer to data classified by a user (e.g., player, amateur, training coach, etc.) and a sport (e.g., soccer, basketball, baseball, tennis, golf, etc.), and may include one or more factual pieces of information according to the classification.

[0036] FIG. 2 is a block diagram for explaining the structure of the electronic device (100) of FIG. 1. FIG. 3 is a diagram for explaining the multilayer neural network (122) of FIG. 2.

[0037] Referring to FIG. 2, the electronic device (100) may include a user management unit (110), an image discrimination unit (120), a challenge management unit (130), and a reward management unit (140).

[0038] The user management unit (110) can register users to the sports multi-content system through a membership registration process and collect and manage user information. The user management unit (110) can perform user registration and user information input processes. The user management unit (110) can provide a membership registration interface to the user terminal (200) and receive user information from the user terminal (200). Here, the user information may include at least one of the user's name, contact information, date of birth, place of residence, and gender. The user management unit (110) can provide services such as photo uploading, photo editing, payment, and guidance video output to users registered as members.

[0039] The user management unit (110) may further include a ranking calculation module (111) that aggregates and manages rankings based on the performance of users and teams, and a user DB (112) which is a database built by collecting user information received and information regarding the status of users' challenge participation, performance, ranking, etc.

[0040] The ranking calculation module (111) can calculate a ranking based on the reward points obtained from a challenge in which a user or team participated, and can update and aggregate the ranking according to a predetermined period and provide it to the user.

[0041] The user DB (112) may have a general data structure implemented in the storage space (hard disk or memory) of a computer system using a database management program (DBMS). The database may have a data storage form that allows for the free retrieval (extraction), deletion, editing, and addition of data. The database may be implemented in accordance with the purpose of one embodiment of the present disclosure using a relational database management system (RDBMS) such as Oracle, Informix, Sybase, and DB2, an object-oriented database management system (OODBMS) such as Gemston, Orion, and O2, and an XML native database such as Excelon, Tamino, and Sekaiju, and may have appropriate fields or elements to achieve its function.

[0042] The user management unit (110) can classify input user information and build a database. The user management unit (110) can classify input user information and store and manage it in the user DB (112). The user DB (112) can be implemented using server programs provided in various ways depending on operating systems such as DOS, Windows, Linux, UNIX, and Macintosh on general server hardware. Representative examples include Website and IIS (Internet Information Server) used in Windows environments, and CERN, NCSA, and APPACH used in UNIX environments. Additionally, the user DB (112) may be linked with an authentication system and a payment system for user authentication of services provided on a platform according to an embodiment of the present invention or for purchase payment related to services.

[0043] The image discrimination unit (120) collects image data submitted by the user to prove the achievement of the challenge, and can determine whether the user has achieved the challenge by using object detection technology based on the image data.

[0044] The image determination unit (120) may require the user to provide more than a predetermined number of image data to determine whether the user has performed or succeeded in the challenge.

[0045] The image determination unit (120) can analyze the image using an object detection algorithm and determine whether the image submitted by the user satisfies the actual challenge conditions. For example, if the user participates in a basketball challenge, the image determination unit (120) can collect image data from the user and determine whether the user's shot passes accurately through the rim in the image data to determine whether the challenge is successful.

[0046] To this end, the image discrimination unit (120) may further include an image preprocessing unit (121), a multilayer neural network (122), and a learning engine (123).

[0047] The image preprocessing unit (121) can receive original image data and preprocess it to generate preprocessed data. Here, preprocessing may refer to the operation of dividing the original image data into frames and normalizing each frame to generate preprocessed data. The image preprocessing unit (121) can normalize each frame of the original image data by adjusting it to the same pixel size. The original image data can be processed into preprocessed data containing a plurality of normalized frames (f1, f2, f3, …, fn). For example, if the original image data is a 90-minute video captured at 30 frames per second (fps), a total of 162,000 frames can be generated.

[0048] In one embodiment, the image preprocessing unit (121) can preprocess original image data to generate preprocessed data including a plurality of frames. The image preprocessing unit (121) can select one or more frames among the plurality of frames according to a preset period.

[0049] Here, the period may be a predetermined period based on the classification of the original image data. The classification of the original image data may be determined based on the metadata of the original image data. Alternatively, the classification of the original image data may be determined based on statistical data.

[0050] The preset period can vary. For example, the period can be 0.1 seconds, 1 second, 3 seconds, 5 seconds, etc. Or, the period can be in frames. For example, the period can be 10 frames, 20 frames, 30 frames, 120 frames, etc.

[0051] In one embodiment, the image preprocessing unit (121) may apply a color filter to each of one or more selected frames to form a grid. In one embodiment, the color filter may be a filter that divides each of the plurality of frames into a plurality of grids and performs color quantization on each of the plurality of grids to unify them into a single color.

[0052] The image preprocessing unit (121) can store one or more selected frames and gridded frames in memory and database based on the metadata of the original image data.

[0053] The image determination unit (120) can determine whether one or more gridded frames match by comparing each of them with a pre-stored pattern layer, classify the matching frames as event occurrence frames corresponding to the pattern layer, and extract a thumbnail image based on the frames.

[0054] The pattern layer may be a layer containing a pattern corresponding to at least one event included in the original video data. For example, if the original video data is a soccer video, the pattern layer may include a pattern layer corresponding to a goal, a pattern layer corresponding to a shot on target, a pattern layer corresponding to a corner kick, etc.

[0055] In one embodiment, the image discrimination unit (120) compares each of the pattern layer and one or more gridded frames, and if there is no frame that matches the pattern layer, it can select at least one of the one or more frames as a new pattern layer and store it in memory.

[0056] In one embodiment, the image discrimination unit (120) compares each of the pattern layer and one or more gridded frames, and if there is no frame that matches the pattern layer, it can select one or more frames again based on a different period.

[0057] In one embodiment, the image determination unit (120) can determine whether the pattern layer and one or more gridded frames match by converting each of the one or more frames into a frame vector having the color value of the grid as a component and converting the pattern layer pattern value as a component, calculating the similarity between each of the pattern vector and one or more frame vectors, and determining that the pattern layer and one or more gridded frames match if the similarity is greater than or equal to a predetermined threshold.

[0058] In one embodiment, the image discrimination unit (120) may set a region of interest for each of a plurality of frames based on statistical data. Here, 'region of interest' may refer to an area located at a specific coordinate within a frame with a specific size. The coordinates of the region of interest may be defined based on the coordinates of the center pixel of the region of interest.

[0059] In one embodiment, the region of interest (L) may be set based on reference coordinates within the frame. The reference coordinates may be, for example, the center coordinates of the frame or the edge coordinates of the frame. In another embodiment, the region of interest may be set based on a specific event. That is, the region of interest may be set according to a specific event (e.g., scoring, passing, dribbling, etc.) detected in the previous frame or the current frame. In yet another embodiment, the region of interest may be set based on a detected object (target).

[0060] The image discrimination unit (120) can detect the location and movement of an object using an object tracking algorithm or an object tracking filter, and set the area around the object as a region of interest. In another embodiment, the region of interest may be set based on the rate of change of pixel values. That is, the region of interest may be set based on an area where the change in pixel values ​​is large within a specific frame (e.g., an area with fast movement or change).

[0061] The image discrimination unit (120) can pre-supervised train a multilayer neural network (122) using multiple training data. A multilayer neural network is a prediction model implemented in software or hardware that mimics the computational power of a biological system using a large number of artificial neurons (or nodes).

[0062] The learning engine (123) can supervise the multilayer neural network (122) using learning data in which image data and correct answer data are input values ​​and challenge achievement result data is output value, so that the multilayer neural network (122) can accurately determine whether a challenge is achieved based on image data.

[0063] In this context, supervised learning refers to a learning process that uses data containing input and corresponding output values ​​as training data to find the output value corresponding to a given input; it signifies learning that takes place while the correct answer is known. In supervised learning, the set of input and output values ​​provided is called training data.

[0064] Referring to FIG. 3, the multilayer neural network (122) may include an input layer, one or more hidden layers, and an output layer.

[0065] In one embodiment, the multilayer neural network (122) may include an input layer that receives image data and has nodes corresponding to a plurality of grids of one or more frames, a first hidden layer that outputs by multiplying each output value of the input layer by a weight and adding a bias, a second hidden layer that outputs by multiplying each output value of the first hidden layer by a weight and adding a bias, and an output layer that outputs by multiplying each output value of the second hidden layer by a weight and outputting the result using an activation function. Although only two hidden layers are shown in FIG. 4, one or more hidden layers may include a greater number of hidden layers in addition to the first hidden layer and the second hidden layer.

[0066] For example, the activation function may be a Softmax function, but embodiments of the present invention are not limited thereto, and the activation function may be various other functions such as a LeRU function. Weights and biases may be continuously updated by supervised learning. Specifically, the output vector may be input to a loss function layer connected to the output layer. The loss function layer may output a loss value using a loss function that compares the output vector with the correct answer vector for each training data. The parameters of the multilayer neural network (122) may be supervised learning in a direction that reduces the loss value.

[0067] A learning model may include an input layer, one or more hidden layers, and an output layer. Training data is input into the input layer, passes through one or more hidden layers and the output layer to produce an output vector, and the output vector may be input into a loss function layer connected to the output layer. The loss function layer may output a loss value using a loss function that compares the output vector with the correct answer vector for each training data. The parameters of the learning model may be trained in a direction that reduces the loss value.

[0068] To this end, the image discrimination unit (120) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (100) itself where the artificial intelligence model is executed, or through a separate server.

[0069] Learning algorithms may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above. An artificial intelligence model may include multiple artificial neural network layers. An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0070] The challenge management unit (130) can create a challenge and set and manage the rules and conditions of the challenge. The challenge management unit (130) can also create a challenge in response to a user's request. Alternatively, the challenge management unit (130) can receive data regarding the creation of a challenge from an external server (not shown) and create a challenge based on this.

[0071] The challenge management unit (130) can set detailed conditions such as the goal, time limit, and number of participants for a challenge registered on the platform, or receive and store detailed conditions such as the goal, time limit, and number of participants for a challenge registered on the platform from an external source. For a challenge registered on the platform, unique participation conditions, challenge success or execution conditions, and success or execution rewards may be set.

[0072] For example, the challenge management department (130) can set the points required for a user to participate in a challenge, and when a user applies to participate in a challenge, the pre-specified points for the user can be deducted.

[0073] In one embodiment, the challenges created and managed by the challenge management unit (130) can be classified into participation-type challenges and challenge-type challenges. Participation-type challenges and challenge-type challenges differ in the user's method of participation and the criteria for determining success.

[0074] A participatory challenge refers to a challenge in which a user applies for a specific challenge, uploads a verification video to prove that they have met the conditions for performing the challenge, and the video discrimination unit (120) evaluates whether the user has participated according to the challenge criteria. A participatory challenge is recognized as having participated in the challenge when the user has met all the pre-set conditions for performing the challenge. For example, in the case of a running challenge, the user runs to meet the distance and time pre-specified for the running challenge, and then submits a video to the electronic device (100) certifying that the running was performed according to the conditions for performing the challenge, so the video discrimination unit (120) can determine whether the conditions for performing the participatory challenge have been met. Since a participatory challenge only examines whether the user has fulfilled the required conditions for performing the challenge, the user is considered to have participated in the challenge if only the conditions for performing the challenge are met.

[0075] On the other hand, a challenge-type challenge is a method in which a user applies for a specific challenge, performs a challenging mission, and proves that they have succeeded. A challenge-type challenge strictly evaluates whether the user has achieved the mission's goal, and the user transmits a video that can certify the mission's success to an electronic device (100), and the success of the challenge mission is automatically checked by a video discrimination unit (120). That is, unlike a participation-type challenge, a challenge-type challenge goes beyond simply satisfying conditions; it is recognized as successful only if the user uploads a video of successfully performing the mission and that video passes the pre-set challenge success conditions. For example, if a mission to make 10 consecutive 3-point shots is given in a basketball challenge, the user performs the mission and uploads a video of the success, and the success is determined through a review process.

[0076] Participatory challenges that can be created by the challenge management department (130) include running challenges, going to the gym challenges, and playing badminton challenges. Since participatory challenges only determine whether the challenge is performed, participation in the challenge can be verified through a relatively simple process.

[0077] Challenge-type challenges that can be generated by the challenge management department (130) include a basketball shooting challenge in which one must make 10 three-point shots during a given time, a plank challenge in which one must maintain a plank position for a certain period of time, and a machine exercise repetition challenge.

[0078] In one embodiment, the challenge type provides the user with an option to select a difficulty level, and may provide a differential reward upon successful completion of the challenge according to the selected difficulty level. For example, in the case of a basketball shooting challenge where one must make 10 three-point shots within a given time, the difficulty levels can be classified as easy, normal, and hard, and conditions can be set as follows: easy: 15 points reward, 5 minutes time limit, 10 successes; normal: 30 points reward, 3 minutes time limit, 10 successes; hard: 50 points reward, 2 minutes time limit, 10 successes. As another example, in the case of an exercise (bench press) repetition challenge, the difficulty can be classified into easy, medium, and hard, and conditions can be set as follows: Easy: 15 reward points, 10 repetitions with 50kg, Medium: 30 reward points, 10 repetitions with 70kg, Hard: 50 reward points, 10 repetitions with 100kg.

[0079] In one embodiment, the number of times a challenge can be attempted may be limited. For example, the number of attempts may be limited to once a day or three times a week.

[0080] The reward management department (140) can establish and manage a reward system that provides points and rewards to users who successfully complete a challenge.

[0081] For example, the reward management department (140) can award a predetermined point to the user when the user succeeds in the challenge.

[0082] The database (150) can comprehensively store and manage learning data, standard data, and analysis data necessary for various sports multi-content services provided by the electronic device (100). The database (150) can be linked with the user DB (112) to establish a data foundation for providing customized services tailored to the characteristics of individual users.

[0083] In one embodiment, the database (150) can be constructed by storing a dataset including an abnormal behavior detection dataset, a specialized dataset by sport, an emotion analysis standard dataset, an exercise tendency classification dataset, a growth analysis dataset, a game prediction model dataset, and a voice command metadata set.

[0084] Here, the abnormal behavior detection dataset may include training data for identifying abnormal user behavior patterns such as falling, leaving one's seat, slumping, and abrupt cessation of movement.

[0085] Specialized datasets for each sport may include data regarding the unique rules, major events, game progression, and highlight generation criteria of each sport, such as soccer, basketball, tennis, and baseball.

[0086] The sentiment analysis reference dataset may include reference data that maps biosignals, such as facial expressions, brainwaves, and heart rate, to the user's emotional state.

[0087] The exercise tendency classification dataset may include classification criteria for recommending customized challenges by analyzing individual exercise patterns, preferred sports, difficulty levels, and participation history.

[0088] The growth analysis dataset may include age- and gender-specific data on physical development indicators such as muscular endurance, flexibility, power, and coordination for children, as well as reference data for analyzing growth trends.

[0089] In one embodiment, the database (150) may be implemented using a relational database management system (RDBMS) such as Oracle, Informix, Sybase, DB2, or an object-oriented database management system (OODBMS) such as Gemston, Orion, O2.

[0090] The sensing processing unit (160) can collect and process various biosignals and behavioral data of the user in real time to provide advanced services such as emotion analysis, motion correction, and abnormal behavior detection. The sensing processing unit (160) can acquire and analyze additional sensing data in addition to the image data collected from the shooting system (300) and the user terminal (200).

[0091] To this end, the sensing processing unit (160) may include a biosignal sensing module, an eye tracking sensing module, a motion recognition sensing module, a voice recognition module, and an emergency detection module.

[0092] The biosignal sensing module may include at least one of a facial recognition sensor for analyzing a user's facial expressions, an EEG sensor for measuring brain waves, and a heart rate sensor for measuring heart rate. The biosignal sensing module can determine the user's emotional state in real time based on sensing data from the sensors.

[0093] The eye-tracking sensing module detects the user's gaze direction and level of concentration to identify the point of interest and gaze of the user in multi-angle video.

[0094] The motion recognition sensing module can analyze the user's accurate posture and movement using 3D motion capture technology, detect incorrect posture during atypical movements, and provide real-time corrective feedback.

[0095] The voice recognition module can recognize the user's voice commands and process them using natural language to provide functions for searching specific video segments and controlling playback.

[0096] The emergency detection module can automatically detect emergency situations based on the results of abnormal behavior detection and, if necessary, send automatic notifications to medical personnel or emergency services.

[0097] In one embodiment, the sensing processing unit (160) can fuse data collected from a plurality of sensing modules in real time to perform more accurate user status determination.

[0098] The personalization unit (170) can provide customized content and services by analyzing the individual characteristics, exercise history, preferences, and tendencies of the user. To this end, the personalization unit (170) may include an emotion analysis engine, a game prediction AI, a motion correction engine, a growth analysis engine, a natural language processing engine, a customized recommendation system, and an adaptive learning module.

[0099] The emotion analysis engine can analyze biometric data collected from the sensing processing unit (160) in real time to identify the user's current emotional state and automatically recommend customized highlight videos or challenges accordingly.

[0100] Match prediction AI can predict match results by comprehensively analyzing player stats, team strength, and recent performance, and generate a summary video comparing the predicted result with actual match footage.

[0101] The motion correction engine analyzes the user's exercise movements in real time to detect abnormal posture during atypical exercises, such as stretching and flexibility exercises, and can recommend methods to correct the posture.

[0102] The growth analysis engine can periodically analyze exercise performance data of youth users to visualize physical development trends and generate personalized coaching content.

[0103] In one embodiment, the personalization unit (170) can automatically adjust the difficulty of the challenge according to the user's exercise ability level to maintain an appropriate sense of challenge.

[0104] The multimedia management unit (180) can generate an image of at least one of VR, AR, and XR based on original image data collected from the shooting system (300) and provide it to the user terminal (200).

[0105] The multimedia management unit (180) may include a virtual image generation module, a multi-angle synchronization engine, a real-time highlight editor, a prediction-actual comparison image generator, and a highlight generator for each item.

[0106] The virtual video generation module can render key scenes of a sports game in a 360-degree stereoscopic view. For example, the virtual video generation module can generate VR, AR, and XR videos that vividly recreate key moments, such as goal scenes and blocking scenes, from multiple angles.

[0107] The multi-angle synchronization engine performs time synchronization of images collected from the first camera (310), the second camera (320), and the third camera (330), and can generate personalized highlights in real time at the point in time and focus desired by the user based on the eye tracking data of the sensing processing unit (160).

[0108] The real-time highlight editor can automatically extract and edit key moments of each sport, such as soccer goals, basketball dunks, and tennis aces, by utilizing an automatic editing algorithm that reflects the characteristics of each sport.

[0109] The prediction-actual comparison video generator can compare and analyze the game prediction AI results of the personalization unit (170) with the actual game video to generate a summary video centered on key scenes different from the prediction.

[0110] In one embodiment, the multimedia management unit (180) can immediately search for and play a video segment corresponding to a specific player, a specific time period, or a specific event according to the user's voice command.

[0111] In one embodiment, the multimedia management unit (180) can accurately interpret natural language commands such as "the most impressive goal scene in today's match" or "second half comeback situation" by linking the metadata of the video with the natural language processing results, and provide the corresponding video.

[0112] The electronic device (100) may have the same configuration as a conventional web server or WAP server in terms of hardware. However, in terms of software, it may include program modules that perform various functions and are implemented through any language such as C, C++, Java, Visual Basic, Visual C, etc. Furthermore, the electronic device (100) generally refers to a computer system that is connected to an unspecified number of clients and / or other servers through an open computer network such as the Internet, receives requests for task execution from clients or other servers, and derives and provides the results of the task, and computer software (server program) installed for that purpose. In addition, the electronic device (100) should be understood as a broad concept that includes, in addition to the aforementioned server program, a series of application programs operating on the electronic device (100) and, in some cases, various databases (DB: Database, hereinafter referred to as "DB") built internally or externally.

[0114] FIG. 4 is a block diagram for explaining the structure of the user terminal (200) of FIG. 1.

[0115] Referring to FIG. 4, the user terminal (200) may include a processor (210), memory (220), communication module (230), antenna module (240), touch display (250), and camera module (260). In one embodiment, at least one of these components may be omitted from the user terminal (200), or one or more other components may be added. In another embodiment, some of these components may be integrated into a single component.

[0116] The processor (210) can, for example, execute software to control at least one other component of a user terminal (200) connected to the processor (210) and perform various data processing or operations. In one embodiment, as at least part of the data processing or operations, the processor (210) can store commands or data received from other components in volatile memory (221), process the commands or data stored in volatile memory (221), and store result data in non-volatile memory (222). In one embodiment, the processor (210) may include a main processor (211) (e.g., a central processing unit or an application processor) or an auxiliary processor (212) that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), a sensor hub processor). For example, if the user terminal (200) includes a main processor (211) and an auxiliary processor (212), the auxiliary processor (212) may be configured to use less power than the main processor (211) or to be specialized for a designated function. The auxiliary processor (212) may be implemented separately from the main processor (211) or as part thereof.

[0117] The auxiliary processor (212) can control at least some of the functions or states associated with at least one of the components of the user terminal (200), for example, on behalf of the main processor (211) while the main processor (211) is in an inactive state, or together with the main processor (211) while the main processor (211) is in an active state. In one embodiment, the auxiliary processor (212) may be implemented as part of another functionally related component.

[0118] In one embodiment, if the auxiliary processor (212) is a neural network processing device, the auxiliary processor (212) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the user terminal (200) itself where the artificial intelligence model is executed, or through a separate server (e.g., electronic device (100)).

[0119] Learning algorithms may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above. An artificial intelligence model may include multiple artificial neural network layers. An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0120] The memory (220) can store various data used by at least one component of the user terminal (200). The data may include, for example, input data or output data for software and related commands. The memory (220) may include volatile memory (221) or non-volatile memory (222).

[0121] The communication module (230) can support the establishment of a wired or wireless communication channel between a user terminal (200) and an external electronic device (e.g., electronic device (100)), and the performance of communication through the established communication channel. The communication module (230) operates independently of the processor (210) and may include one or more communication processors that support wired or wireless communication. In one embodiment, the communication module (230) may include a wireless communication module or a wired communication module. The corresponding communication module among these communication modules can communicate with the electronic device (100) through a network. These various types of communication modules may be integrated into a single component or implemented as multiple separate components.

[0122] The wireless communication module can support 5G networks following 4G networks and next-generation communication technologies, such as new radio access technology (NR access technology). NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module can support high-frequency bands, for example, to achieve high data transmission rates. The wireless communication module can support various technologies to secure performance in high-frequency bands, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beamforming, or large-scale antenna. The wireless communication module can support various requirements specified in the user terminal (200), external electronic device, or network system. In one embodiment, the wireless communication module can support a peak data rate for eMBB realization, lossy coverage for mMTC realization, or U-plane latency for URLLC realization.

[0123] The antenna module (240) can transmit a signal or power to an external electronic device or receive it from the outside. In one embodiment, the antenna module (240) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate. In one embodiment, the antenna module (240) may include a plurality of antennas. In this case, at least one antenna suitable for a communication method used in a communication network, such as a network, may be selected from the plurality of antennas, for example, by a communication module (230). The signal or power may be transmitted or received between the communication module (230) and the external electronic device through the selected at least one antenna.

[0124] In one embodiment, commands or data may be transmitted or received between electronic devices (100) connected to a network. In one embodiment, all or part of the operations executed at a user terminal (200) may be executed at an electronic device (100).

[0125] For example, when a user terminal (200) needs to perform a function or service automatically or in response to a request from a user or another device, the user terminal (200) may request one or more external electronic devices to perform at least a part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the request may perform at least a part of the requested function or service, or additional functions or services related to the request, and transmit the result of the execution to the user terminal (200). The user terminal (200) may provide the result as is or additionally processed and as at least part of the response to the request. To this end, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The user terminal (200) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing.

[0126] Here, the network refers to a connection structure capable of exchanging information between each node, such as terminals and servers, or a network connecting a user terminal (200) and an electronic device (100). The network includes, but is not limited to, the Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), 3G, 4G, LTE, 5G, Wi-Fi, etc. The network may be a closed network such as a LAN or WAN, but it is preferable that it be an open network such as the Internet. The Internet refers to a global open computer network structure that provides the TCP / IP protocol and various services existing in the upper layer, namely HTTP (HyperText Transfer Protocol), Telnet, FTP (File Transfer Protocol), DNS (Domain Name System), SMTP (Simple Mail Transfer Protocol), SNMP (Simple Network Management Protocol), NFS (Network File Service), and NIS (Network Information Service).

[0127] The touch display (250) may be an LCD (liquid crystal display) panel, an OLED (organic light emitting diode) panel, etc. The touch display may also include a touch screen panel (TSP) that includes a touch electrode for detecting a touch on the display panel. In one embodiment, the touch display may be an in-cell type in which the display panel and the touch panel are combined integrally. However, this is exemplary and the embodiments of the present invention are not limited thereto. The touch display may electrically recognize contact such as a finger or pen, generate an electrical signal, and transmit it to a processor.

[0128] The camera module (260) can capture still images and video. In one embodiment, the camera module (260) may include one or more cameras. Each of the one or more cameras may include one or more lenses, image sensors, image signal processors, or flashes. Depending on the needs, the camera module (260) may include one camera or two or more different types of cameras. The camera module (260) can generate original image data by capturing an authentication video for challenge participation authentication.

[0130] FIG. 5 is a block diagram for explaining the structure of the sensor device (400) of FIG. 1.

[0131] Referring to FIG. 5, the sensor device (400) may be composed of wearable devices that collect a user's biometric information and may include a heart rate sensor (410), an accelerometer (420), a gyroscope sensor (430), a GPS module (440), a bioelectric sensor (450), a temperature sensor (460), a pressure sensor (470), and an oxygen saturation sensor (480). The sensor device (400) according to one embodiment of the present invention may include all of the above components or selectively configure only some of the sensors as needed, depending on the characteristics of the exercise or the purpose of monitoring.

[0132] The heart rate sensor (410) is a sensor that monitors the user's heart rate in real time and may use at least one of the photoplethysmography (PPG) method or the electrocardiogram (ECG) method. The heart rate sensor (410) detects changes in blood flow using green light (425 nm) and infrared (940 nm) LEDs and can measure a heart rate range of 30 to 220 beats per minute with an accuracy of within ±2 bpm.

[0133] Specifically, the heart rate sensor (410) has a built-in 16-bit ADC (Analog-to-Digital Converter) capable of acquiring high-resolution signals of more than 1,000 samples per second, and can remove motion artifacts in real time through a DSP (Digital Signal Processing) chip. The heart rate sensor (410) can evaluate autonomic nervous system activity through Heart Rate Variability (HRV) analysis and calculate HRV indicators such as RMSSD, pNN50, and SDNN to quantify the user's stress state and fatigue level.

[0134] In one embodiment, the heart rate sensor (410) can detect arrhythmias such as atrial fibrillation by measuring a single-lead electrocardiogram for 30 seconds in a manner similar to the Digital Crown-based ECG sensor of the Apple Watch. In another embodiment, the heart rate sensor (410) is implemented in the form of a chest strap such as Polar H10 to achieve a heart rate measurement accuracy of 99% or higher. In yet another embodiment, the heart rate sensor (410) can simultaneously measure blood hemoglobin concentration and oxygen saturation using a multi-wavelength LED array.

[0135] The acceleration sensor (420) is a MEMS (Micro-Electro-Mechanical Systems) based sensor that measures the user's movement and posture changes in three axes (X, Y, Z axes) and can support a measurement range from ±2g to ±16g. The acceleration sensor (420) supports a sampling frequency of up to 8kHz to precisely detect high-speed motion or impact, and can measure minute changes in acceleration of less than 0.001g through 16-bit resolution.

[0136] Specifically, the acceleration sensor (420) can be implemented with a high-performance IMU chip such as the Bosch BMI270 or STMicroelectronics LSM6DSO, and can provide temperature compensation and vibration immunity, enabling stable measurements in various environments. The acceleration sensor (420) can automatically classify the user's posture (standing, sitting, lying down, walking, running) through gravity vector analysis and identify the type of exercise through an activity pattern recognition algorithm.

[0137] The acceleration sensor (420) has a fall detection algorithm built in to detect a sudden change in acceleration (more than 3g) and a zero-gravity state (less than 0.5g) continuously, and can determine that it is a fall. In addition, the acceleration sensor (420) can provide additional functions such as counting steps, estimating calorie consumption, and analyzing sleep stages.

[0138] In one embodiment, the acceleration sensor (420) is equipped with a machine learning-based activity recognition model and can classify more than 20 types of exercise movements with an accuracy of more than 95%. In another embodiment, the acceleration sensor (420) can detect muscle tremors or spasms through vibration pattern analysis to detect neurological abnormalities early. In yet another embodiment, the acceleration sensor (420) is designed with a waterproof structure that operates even during underwater exercise and can be applied to water sports such as swimming and diving.

[0139] The gyroscope sensor (430) is a sensor that measures the user's rotational motion and angular velocity in three axes and can support a measurement range from ±250 dps to ±2000 dps (degrees per second). The gyroscope sensor (430) provides 16-bit resolution and a sampling frequency of up to 8 kHz, so it can accurately track fast rotational motions, and through a temperature drift compensation function, it can enable stable measurements for a long time.

[0140] Specifically, the gyroscope sensor (430) can be implemented in a form integrated into a 6-axis or 9-axis IMU such as InvenSense ICM-20948 or Bosch BMI260, and can improve attitude estimation accuracy through sensor fusion with the accelerometer sensor (420). The gyroscope sensor (430) can prevent gimbal lock and provide stable 3D orientation information by applying a quaternion-based attitude estimation algorithm and a Kalman filter.

[0141] The gyroscope sensor (430) can analyze the user's rotational movement pattern to precisely measure the trajectory and speed of complex movement actions such as golf swings, tennis strokes, and baseball batting. Additionally, the gyroscope sensor (430) can calculate static and dynamic balance indices for evaluating the sense of balance and detect dizziness or balance disorders early.

[0142] In one embodiment, the gyroscope sensor (430) can provide relative direction information without a magnetic sensor by applying the Attitude and Heading Reference System (AHRS) algorithm. In another embodiment, the gyroscope sensor (430) can provide diagnostic assistance information to medical staff by quantifying hand tremors, which are an early symptom of Parkinson's disease, through vibration analysis. In yet another embodiment, the gyroscope sensor (430) can be utilized as a head tracking sensor for VR / AR applications to provide an immersive exercise experience.

[0143] The GPS module (440) is a module that tracks the user's exact location and movement path through a satellite navigation system and can support multiple GNSS (Global Navigation Satellite Systems) such as GPS, GLONASS, Galileo, BeiDou, and QZSS. The GPS module (440) provides positional accuracy within a minimum of 1 meter, and can achieve centimeter-level precision when differential GPS (DGPS) or RTK (Real-Time Kinematic) technology is applied.

[0144] Specifically, the GPS module (440) can be implemented as a multi-band GNSS receiver such as the u-blox M8, M9 series or Broadcom BCM47755, and can correct ionospheric errors by simultaneously receiving L1, L2, and L5 frequency bands. The GPS module (440) supports an update frequency of 10 Hz or more per second, enabling accurate position tracking even during high-speed movement, and can provide fast satellite acquisition times within 5 seconds for a Cold Start and within 1 second for a Hot Start.

[0145] The GPS module (440) can calculate and store the user's exercise path, distance traveled, average speed, maximum speed, altitude change, etc. in real time. In addition, the GPS module (440) can automatically generate an alert when the user moves out of a pre-set safe zone through a geo-fencing function, and can transmit accurate location information to a rescue team in case of an emergency.

[0146] In one embodiment, the GPS module (440) can reduce satellite acquisition time even indoors or in urban areas by utilizing auxiliary data from a cellular network through an Assisted GPS (A-GPS) function. In another embodiment, the GPS module (440) can enable continuous location tracking even in tunnels or underground sections where GPS signals are blocked through fusion with an Inertial Navigation System (INS). In yet another embodiment, the GPS module (440) can provide performance analysis by section by linking with detailed map information such as golf courses, running tracks, and cycling routes as a sports-specific function.

[0147] The bioelectric sensor (450) is a sensor that measures the user's electromyography (EMG), electroencephalography (EEG), and electrodermal activity (EDA), and can quantify muscle activity and mental stress state. The bioelectric sensor (450) has a 24-bit ADC and a noise level of 1 Hz or less, so it can accurately detect even minute biosignals, and can have a built-in 50Hz / 60Hz power noise removal filter.

[0148] The temperature sensor (460) is a dual sensor that simultaneously measures the user's body temperature and the ambient temperature, and can support a measurement accuracy of ±0.1°C and a measurement range of -40°C to 125°C. The temperature sensor (460) can detect danger signals such as fever, dehydration, and heatstroke at an early stage by analyzing the body temperature change pattern, and can calculate a heat stress index based on exercise intensity.

[0149] The pressure sensor (470) is a sensor that detects contact pressure applied by a user along with altitude estimation through atmospheric pressure measurement, and can measure altitude with an accuracy of ±1m with a resolution of 1 hPa or less. The pressure sensor (470) can monitor the risk of altitude sickness in sports with large altitude changes such as climbing, skiing, and paragliding, and can evaluate walking patterns and balance through plantar pressure analysis.

[0150] The oxygen saturation sensor (480) is a sensor that non-invasively measures blood oxygen saturation using pulse oximetry and can provide an accuracy of ±2% in the range of 70% to 100%. The oxygen saturation sensor (480) calculates the ratio of oxidized hemoglobin to reduced hemoglobin using the difference in absorption rates between red light (660 nm) and infrared light (940 nm), and can detect hypoxia or respiratory disease early.

[0151] All sensors of the sensor device (400) are connected to the electronic device (100) via wireless communication of BLE (Bluetooth Low Energy) 5.0 or higher, and can provide adaptive sampling frequency adjustment and sleep mode switching functions to optimize battery life. The collected multi-sensor data undergoes time synchronization and calibration processes in the electronic device (100) and is fused with the video analysis results to enable more accurate and reliable detection of abnormal behavior.

[0153] FIG. 6 is a flowchart illustrating the process of preprocessing original image data according to an embodiment of the present invention. FIG. 7 is a diagram schematically illustrating the process of dividing a plurality of original image data into frames.

[0154] First, in step S610, the image preprocessing unit (121) of the electronic device (100) can divide the received original image data (v_raw) into frames.

[0155] In step S620, the image preprocessing unit (121) of the electronic device (100) can generate preprocessed data by normalizing each frame (f). The electronic device (100) can normalize each frame of the original image data by adjusting it to the same pixel size. For example, if the original image data is a 90-minute video captured at 30 frames per second (fps), a total of 162,000 frames can be generated.

[0156] In step S630, the image preprocessing unit (121) of the electronic device (100) can divide the preprocessed data into a plurality of consecutive frames to generate a continuous frame set (cf).

[0157] In one embodiment, a continuous frame set (cf) may be defined as a collection of frames captured at consecutive time intervals. For example, frames constituting a video over one second may be defined as a single "continuous frame set."

[0158] Referring to FIG. 7, an exemplary method is illustrated for generating a first continuous frame set (cf1) including f1, f2, and f3 among a plurality of frames (f) generated by preprocessing the first original image data (v_raw1), and generating a second continuous frame set (cf2) including f1, f2, and f3 among a plurality of frames (f) generated by preprocessing the second original image data (v_raw2). In FIG. 7, L1 and L2 represent a first region of interest and a second region of interest, respectively.

[0159] In another embodiment, the electronic device (100) may generate a continuous frame set based on the time of occurrence of a specific event. The specific event may be, for example, a scoring scene in soccer, a shooting motion in basketball, abnormal behavior, sudden behavior, emergency behavior, etc. In this case, the image preprocessing unit (121) may detect the occurrence of a specific event in conjunction with other components (e.g., a sensing processing unit (160)), and generate a continuous frame set including a frame preceding the detected frame and a frame following the detected frame based on the frame (e.g., f2) in which the occurrence of the specific event is detected.

[0160] In another embodiment, the electronic device (100) can generate a continuous frameset based on a section in which a specific object (e.g., ball, player, goalpost, racket, etc.) is recognized or tracked based on an object recognition and object tracking algorithm. Similarly, the image preprocessing unit (121) can recognize a specific object in conjunction with another component (e.g., sensing processing unit (160)) and generate a continuous frameset including a frame preceding the recognized frame and a frame following the recognized frame, based on the frame in which the specific object is recognized (e.g., f2).

[0161] The electronic device (100) can extract a video clip corresponding to preset operation information or event information from a generated continuous frame set. The electronic device (100) can divide a video segment into video clips of a specific time unit.

[0163] FIG. 8 is a flowchart illustrating the process of generating a recording video based on abnormal behavior according to one embodiment of the present invention.

[0164] Referring to FIG. 8, the electronic device (100) can analyze original image data received from the shooting system (300) to automatically detect abnormal behavior of the user and, based on this, generate a customized final image and provide it to the user terminal (200).

[0165] First, in step S810, the sensing processing unit (160) of the electronic device (100) can detect abnormal behavior candidate frames by analyzing the user's movement pattern from the original image data.

[0166] The sensing processing unit (160) can identify abnormal behavior candidates by comparing them with the user's normal movement pattern based on the abnormal behavior detection dataset stored in the database (150). To this end, the database (150) can store the abnormal behavior detection dataset by classifying the abnormal behavior into detailed types. The detailed types may include at least one of falling, departure, sitting, sudden stop, and ceremony.

[0167] Types of abnormal behavior Duration Event Classification priority Falling 3 seconds or more Warning Event height Displacement 10 seconds or more Warning Event middle collapsing 5 seconds or more Warning Event middle Sudden stop 2 seconds or more Warning Event lowness Ceremonial movements 3 seconds or more Highlight Event height

[0168] is a classification criterion for events by type of abnormal behavior according to one embodiment of the present invention. As shown in , behaviors that may be dangerous to safety (e.g., falling, leaving one's seat, etc.) can be classified as warning events, and positive emotional expressions (e.g., celebratory movements) can be classified as highlight events.

[0169] In one embodiment, the sensing processing unit (160) can analyze the user's 3D coordinate change amount, velocity change rate, and posture change angle in real time by utilizing a motion recognition sensing module. The sensing processing unit (160) can quantify the possibility of abnormal behavior by calculating the deviation from a pre-learned normal motion pattern.

[0170] In step S820, the sensing processing unit (160) of the electronic device (100) can calculate an abnormal behavior score for a detected abnormal behavior candidate frame and determine whether an actual abnormal behavior has occurred by comparing it with a preset threshold value. In one embodiment, the sensing processing unit (160) of the electronic device (100) can calculate an abnormal behavior score for an abnormal behavior candidate frame based on at least one of a posture deviation value, a rate of change of speed, and a duration of the abnormal state, and determine whether an abnormal behavior has occurred by comparing the abnormal behavior score with a preset threshold value.

[0171]

[0172] In <Equation 1>, A_score represents the abnormal behavior score, P_deviation represents the attitude deviation value, V_change represents the rate of change of velocity, T_duration represents the duration of the abnormal state, and w_1, w_2, and w_3 represent their respective weights.

[0173] The sensing processing unit (160) can classify a frame as an abnormal behavior occurrence frame if the A_score is greater than or equal to a preset threshold (e.g., 0.7).

[0174] Specifically, the sensing processing unit (160) can set a region of interest within an abnormal behavior candidate frame and calculate an abnormal behavior score by applying <Equation 1> within the set region of interest. Here, the region of interest can be set around at least one of the user's body center, the user's major joint area, or a point of contact with exercise equipment. The sensing processing unit (160) can calculate P_deviation by analyzing the amount of change in pixel coordinates within the region of interest and calculate V_change by analyzing the speed of movement of the region of interest between consecutive frames.

[0175] In one embodiment, the sensing processing unit (160) may apply different weights depending on the type of abnormal behavior. For example, in the case of falling, a high weight (w_1 = 0.5) may be applied to P_deviation, and in the case of leaving one's seat, a high weight (w_3 = 0.6) may be applied to T_duration. As another example, in the case of slumping, a high weight (w_2 = 0.4) may be applied to V_change and a medium weight (w_3 = 0.4) to T_duration to evaluate gradual speed changes and duration. In the case of sudden cessation of movement, a maximum weight (w_2 = 0.7) may be applied to V_change to use instantaneous speed changes as the main criterion for determination.

[0176] In another embodiment, the sensing processing unit (160) may apply abnormal behavior detection criteria specialized for each sport. For soccer, a high weight may be set for leg movement pattern analysis along with setting a region of interest centered on the ankle joint, and for basketball, emphasis may be placed on landing pattern analysis of jump motions along with setting a region of interest centered on the knee and shoulder joints.

[0177] In one embodiment, the sensing processing unit (160) can detect abnormal behavior candidate frames for each of the original image data captured by different cameras. As illustrated in FIG. 1, when the shooting system (300) includes three cameras, abnormal behavior candidate frames can be detected for each of the first original image data (v_raw1) generated by the first camera (310), the second original image data (v_raw2) generated by the second camera (320), and the third original image data (v_raw3) generated by the third camera (330). Accordingly, the sensing processing unit (160) can detect the first abnormal behavior candidate frame, the second abnormal behavior candidate frame, and the third abnormal behavior candidate frame. Here, the first to third abnormal behavior candidate frames must be frames generated (captured) by different cameras at the same time.

[0178] The sensing processing unit (160) can determine whether abnormal behavior occurs for each of the first to third abnormal behavior candidate frames and classify them as frames where abnormal behavior occurs. At this time, the sensing processing unit (160) can calculate an abnormal behavior score by applying a correction coefficient based on the shooting angle and distance of each camera. For example, when the first camera (310) shoots the front of the user, the reliability of P_deviation can be set high because it is advantageous for detecting posture deviation, and when the second camera (320) shoots the side, the reliability of V_change can be set high because it is advantageous for detecting speed change.

[0179] The sensing processing unit (160) can determine that if two or more of the three candidate frames of abnormal behavior are determined to have occurred, all three candidate frames of abnormal behavior captured at that time are frames in which abnormal behavior occurred. Through this multiple verification method, errors caused by misrecognition of a single camera can be prevented and the accuracy of abnormal behavior detection can be improved. In addition, the sensing processing unit (160) can classify the highest confidence level when abnormal behavior is detected in all three cameras, and classify the medium confidence level when abnormal behavior is detected in only two cameras, and process them differentially during the subsequent video generation process.

[0180] In step S830, the image preprocessing unit (121) of the electronic device (100) can generate a continuous frame set centered on the frame in which abnormal behavior is detected. The image preprocessing unit (121) can form a single continuous frame set including a preceding frame and a succeeding frame based on the frame in which abnormal behavior occurred.

[0181] Specifically, the video preprocessing unit (121) can generate a continuous frame set including the previous 30 frames (1 second) and the subsequent 60 frames (2 seconds) centered on the frame in which abnormal behavior is detected (e.g., f_anomaly). Through this, a video segment containing context information before and after the occurrence of abnormal behavior can be obtained.

[0182] In one embodiment, the image preprocessing unit (121) can adjust the length of the continuous frame set according to the type of abnormal behavior. For momentary abnormal behavior such as falling, a continuous frame set containing a relatively short section (total 3 seconds) can be generated, and for continuous abnormal behavior such as leaving one's seat, a continuous frame set containing a long section (total 10 seconds) can be generated. In another embodiment, the image preprocessing unit (121) can integrate multiple abnormal behaviors into one extended continuous frame set when they occur consecutively.

[0183] In step S840, the multimedia management unit (180) of the electronic device (100) can generate a final image based on the generated continuous frame set. The multimedia management unit (180) can generate a final image by analyzing the severity and type of abnormal behavior and classifying it into a highlight image or a warning image.

[0184] The multimedia management unit (180) can distinguish whether abnormal behavior is a positive event (e.g., ceremony motion, cheering motion) or a negative event (e.g., injury, falling) by linking with the emotion analysis engine of the personalization unit (170). If it is determined to be a positive event, a highlight video can be generated, and if it is determined to be a negative event, a warning video can be generated.

[0185] In one embodiment, the multimedia management unit (180) can automatically apply editing effects such as slow motion effects, replay effects, and music overlays when generating a highlight video. In another embodiment, the multimedia management unit (180) can add markers, notification text, warning sounds, etc., that visually highlight the point where abnormal behavior occurs when generating a warning video.

[0186] In another embodiment, the multimedia management unit (180) can generate a multi-angle final video by synchronizing scenes of abnormal behavior captured from multiple camera viewpoints using a multi-angle synchronization engine. In this case, the user can view the moment of abnormal behavior occurrence from multiple angles.

[0187] In step S850, the electronic device (100) can transmit the generated final image to the user terminal (200) via a network. When transmitting, the electronic device (100) can transmit metadata such as the type of abnormal behavior, time of occurrence, and severity together so that the user terminal (200) can select an appropriate notification method.

[0188] In one embodiment, the electronic device (100) may transmit the final video along with a social media sharing function in the case of positive abnormal behavior (highlight). In another embodiment, the electronic device (100) may transmit the final video along with an immediate notification in the case of negative abnormal behavior (warning), and may also transmit a notification to emergency contacts as needed.

[0189] In another embodiment, the electronic device (100) may provide the final video in a real-time streaming manner or a file download manner according to the user's settings. The final video may be transmitted with an optimal resolution and compression rate by considering the network status and the performance of the user terminal (200).

[0190] Through this process, the electronic device (100) can automatically detect various abnormal behaviors occurring during the user's exercise and generate and provide customized video content suitable for the situation. This enables effective recording and sharing of exercise performance while simultaneously managing the user's safety.

[0191] In step S840, the multimedia management unit (180) of the electronic device (100) can generate a final image based on the generated continuous frame set. The multimedia management unit (180) can generate a final image by analyzing the severity and type of abnormal behavior and classifying it into a highlight image or a warning image.

[0192] The multimedia management unit (180) can distinguish whether abnormal behavior is a positive event (e.g., ceremony motion, cheering motion) or a negative event (e.g., injury, falling) by linking with the emotion analysis engine of the personalization unit (170). If it is determined to be a positive event, a highlight video can be generated, and if it is determined to be a negative event, a warning video can be generated.

[0193] In one embodiment, the multimedia management unit (180) can automatically apply editing effects such as slow motion effects, replay effects, and music overlays when generating a highlight video. In another embodiment, the multimedia management unit (180) can add markers, notification text, warning sounds, etc., that visually highlight the point where abnormal behavior occurs when generating a warning video.

[0194] In another embodiment, the multimedia management unit (180) can generate a multi-angle final video by synchronizing scenes of abnormal behavior captured from multiple camera viewpoints using a multi-angle synchronization engine. In this case, the user can view the moment of abnormal behavior occurrence from multiple angles.

[0195] In step S850, the electronic device (100) can transmit the generated final image to the user terminal (200) via a network. When transmitting, the electronic device (100) can transmit metadata such as the type of abnormal behavior, time of occurrence, and severity together so that the user terminal (200) can select an appropriate notification method.

[0196] In one embodiment, the electronic device (100) may transmit the final video along with a social media sharing function in the case of positive abnormal behavior (highlight). In another embodiment, the electronic device (100) may transmit the final video along with an immediate notification in the case of negative abnormal behavior (warning), and may also transmit a notification to emergency contacts as needed.

[0197] In another embodiment, the electronic device (100) may provide the final video in a real-time streaming manner or a file download manner according to the user's settings. The final video may be transmitted with an optimal resolution and compression rate by considering the network status and the performance of the user terminal (200).

[0198] Through this process, the electronic device (100) can automatically detect various abnormal behaviors occurring during the user's exercise and generate and provide customized video content suitable for the situation. This enables effective recording and sharing of exercise performance while simultaneously managing the user's safety.

[0199] As described above, a method according to one embodiment of the present invention monitors a user's exercise situation in real time from multiple angles through a multi-camera-based sports video recording system and can automatically detect abnormal situations such as falling, leaving one's seat, or slumping by utilizing an AI-based abnormal behavior detection algorithm. In particular, the objectivity and accuracy of abnormal behavior determination can be significantly improved through a quantitative evaluation formula that combines posture deviation values, speed change rates, and duration.

[0200] Furthermore, the multi-camera cross-verification method prevents false recognition by a single camera and enhances the reliability of abnormal behavior detection. By automatically classifying detected abnormal behavior into positive events (ceremonies) and negative events (injury risks), and generating appropriate highlight or warning videos for each, it is possible to provide customized content to users. This allows for a departure from conventional simple video recording methods, enabling the simultaneous provision of intelligent sports safety management and entertainment services, thereby ensuring exercise safety and enhancing user satisfaction.

[0201] The embodiments described above may be implemented as hardware components, software components, and / or combinations of hardware and software components. For example, the devices, methods, and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, although the processing unit is described as being used as a single unit, those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.

[0202] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0203] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0204] Although the embodiments have been described above with reference to the limited drawings, those skilled in the art can apply various technical modifications and variations based on the above. For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0205] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

Claim 1 A step of receiving original video data from a shooting system by means of an electronic device and detecting candidate frames of abnormal behavior by analyzing the user's movement pattern from the original video data; a step of calculating an abnormal behavior score for the candidate frames of abnormal behavior by means of the electronic device based on at least one of a posture deviation value, a rate of change of velocity, and a duration of an abnormal state, and determining a frame in which abnormal behavior occurs by comparing the abnormal behavior score with a preset threshold value; a step of generating a continuous frame set including a preceding frame and a succeeding frame based on the frame in which abnormal behavior occurs by means of the electronic device; a step of analyzing the type of abnormal behavior based on the continuous frame set by means of the electronic device, classifying it into a highlight video or a warning video, and generating a final video. The method comprises the step of transmitting the final image to a user terminal via a network by means of the electronic device; wherein the shooting system comprises a first camera, a second camera, and a third camera installed to photograph the user from different directions, and the abnormal behavior candidate frame comprises a first abnormal behavior candidate frame detected by analyzing the user's movement pattern from first original image data generated by the first camera, a second abnormal behavior candidate frame detected by analyzing the user's movement pattern from second original image data generated by the second camera, and a third abnormal behavior candidate frame detected by analyzing the user's movement pattern from third original image data generated by the third camera, and the first to third abnormal behavior candidate frames are frames captured at the same time, and the step of detecting the abnormal behavior candidate frame comprises the step of setting a region of interest centered on at least one of the user's body center, major joint area, and contact point with exercise equipment within the frame of the original image data; and the step of calculating at least one of the posture deviation value and the rate of change of velocity within the region of interest.The method comprises: a step of calculating an abnormal behavior score for each of the first to third abnormal behavior candidate frames based on the attitude deviation value, the rate of change of speed, and the duration of the abnormal state; and a step of determining all of the first to third abnormal behavior candidate frames as abnormal behavior occurrence frames if two or more of the first to third abnormal behavior candidate frames have an abnormal behavior score greater than or equal to the threshold value, wherein the abnormal behavior score is calculated based on the following mathematical formula, <Mathematical Formula>; In the above <mathematical formula>, A_score represents the abnormal behavior score, P_deviation represents the attitude deviation value, V_change represents the rate of change of velocity, T_duration represents the duration of the abnormal state, and w_1, w_2, and w_3 represent their respective weights. Claim 2 delete Claim 3 delete Claim 4 A method according to claim 1, wherein the abnormal behavior includes at least one of falling, leaving one's seat, sitting down, and abrupt cessation of movement, and the step of calculating the abnormal behavior score includes the step of applying different weights according to the type of abnormal behavior. Claim 5 A method according to claim 1, wherein the step of transmitting the final image comprises: a step of generating metadata regarding the type, time of occurrence, and severity of the abnormal behavior; and a step of transmitting the metadata together with the final image to the user terminal; and, if the abnormal behavior is determined to be a negative event, the electronic device additionally transmits a notification to emergency contacts stored in a database.

Images