Device and method for generating highlight videos using biosignal-based emotion recognition

Abstract

The embodiments provide an apparatus and method for generating a highlight video through biosignal-based emotion recognition. The method according to the embodiments may include: a step of collecting at least one sensing data including time information from a sensor device and a user's biosignal at a time according to the time information by an electronic device, and recognizing the user's emotional state for each time by comparing the sensing data with a preset threshold value; a step of calculating an emotion weight for each of the recognized emotional states for each time by the electronic device; a step of dividing original video data into a plurality of time intervals by the electronic device and generating an emotion-reflecting continuous frame set by mapping the emotion weight corresponding to each time interval by the electronic device; a step of generating a highlight final video based on the emotion-reflecting continuous frame set by the electronic device; and a step of transmitting the highlight final video to a user terminal by the electronic device. The step of generating the emotion-reflecting continuous frame set by mapping the emotion weights may include the step of selecting frames in time intervals where the emotion weight is greater than or equal to a preset threshold as highlight candidate frames, and the step of sorting time intervals in order of highest emotion weight to select the interval of the upper preset ratio.

Description

Technology Field

[0001] Embodiments of the present invention relate to an apparatus and method for generating a highlight video through biosignal-based emotion recognition, and more specifically, to an apparatus and method for generating a highlight video based on a frame through the recognition of a user's biosignal-based emotion. Background Technology

[0002] With the rapid advancement of technologies for live sports broadcasting and highlight video production in recent years, user demand for personalized content is increasing. In particular, the technology to extract key moments from various sports such as soccer, basketball, and baseball to create short highlight videos has become an essential feature on OTT services, social media, and mobile platforms.

[0003] Existing highlight video generation technology primarily utilizes video analysis algorithms to automatically detect specific events within a match (goals, fouls, scoring, etc.) and extract highlight segments based on these events. For example, in soccer matches, technology that identifies important moments by analyzing sudden changes in movement within the goal area, players' concentration, and the volume of crowd cheers is widely used.

[0004] However, these existing technologies have the following limitations. Because they generate highlights based on the same criteria for all users, they fail to reflect individual interests or preferences. For example, some users may find intense contested matches between players more interesting than goal scenes, but existing systems cannot account for these individual differences. Furthermore, relying solely on objective video analysis, they fail to reflect the subjective emotional reactions of actual viewers. Even for the same goal scene, emotional responses can differ significantly depending on whether the goal belongs to the team the user supports or the opposing team, and the user's real-time level of immersion or excitement can also be a crucial factor in selecting highlight segments.

[0005] Due to the limitations of existing technologies, there is a growing need for personalized highlight video generation technology that continuously improves by learning from past history while reflecting individual users' emotional responses and preferences in real time. The problem to be solved

[0006] Embodiments of the present invention aim to provide an apparatus and method for generating highlight images through biosignal-based emotion recognition to solve problems such as the limitations of the prior art. The technical problems to be solved by the embodiments are not limited to those mentioned above, and other unmentioned technical problems may be considered by those skilled in the art from the various embodiments described below. means of solving the problem

[0007] A method for generating a highlight video through biosignal-based emotion recognition according to an embodiment of the present invention may include: a step of collecting at least one sensing data including time information and a user's biosignal at a time according to the time information from a sensor device by an electronic device, and recognizing the user's emotional state for each time by comparing the sensing data with a preset threshold value; a step of calculating an emotion weight for each of the recognized emotional states for each time by the electronic device; a step of dividing original video data into a plurality of time intervals by the electronic device and generating an emotion-reflecting continuous frame set by mapping the emotion weight corresponding to each time interval by the electronic device; a step of generating a highlight final video based on the emotion-reflecting continuous frame set by the electronic device; and a step of transmitting the highlight final video to a user terminal by the electronic device. The step of generating the emotion-reflecting continuous frame set by mapping the emotion weights may include the step of selecting frames in time intervals where the emotion weight is greater than or equal to a preset threshold as highlight candidate frames, and the step of sorting time intervals in order of highest emotion weight to select the interval of the upper preset ratio.

[0008] <Mathematical Formula>

[0009]

[0010] The method further includes the step of calculating the emotional weight based on the <mathematical formula> by the electronic device; wherein, in the <mathematical formula>, E_weight represents the emotional weight, w_1 represents a weighting factor for heart rate, w_2 represents a weighting factor for skin conductivity, w_3 represents a weighting factor for motion intensity, HR_ratio represents the ratio of the user's current heart rate to the normal heart rate, GSR_norm represents a normalized skin conductivity value, and Motion_intensity represents motion intensity calculated based on acceleration and gyroscope data.

[0011] The sensor device comprises at least one of a heart rate sensor, a skin conductivity sensor, an accelerometer, and a gyroscope, and the sensing data may comprise at least one of heart rate sensing data, skin conductivity sensing data, and motion sensing data. The step of recognizing the emotional state by the electronic device may include the step of determining the user's emotional state as an excited state when the heart rate sensing data detects a change of more than a preset ratio relative to the user's normal heart rate, and when the skin conductivity sensing data detects a change of more than a preset threshold.

[0012] The above-described shooting system includes a first camera, a second camera, and a third camera installed to photograph the user from different directions, and the step of recognizing the emotional state may include: a step of generating a first emotional state determination result for first original image data generated by the first camera, a second emotional state determination result for second original image data generated by the second camera, and a third emotional state determination result for third original image data generated by the third camera; and a step of finally determining the entire original image data at the corresponding time as the specific emotional state when two or more of the first emotional state determination results to the third emotional state determination results are determined as a specific emotional state.

[0013] The method may include the step of generating a personalized emotion analysis report by analyzing at least one of the past match-specific emotion weight history data, preferred highlight length data, and match-specific reaction pattern data stored in a user database by the electronic device; and the step of transmitting the personalized emotion analysis report together with the highlight final video to the user terminal by the electronic device.

[0014] The above personalized emotion analysis report includes at least one of the reaction time distribution by emotion state, average excitement index, emotion duration pattern, and preferred game situation type, and the personalization unit can optimize and adjust the value of the weighting coefficient used when calculating the emotion weight based on the analysis results for each user. Effects of the invention

[0015] According to embodiments of the present invention, personalized highlight videos can be automatically generated through the recognition of an emotional state based on a user's real-time biosignal, thereby significantly improving user satisfaction compared to conventional uniform highlight production methods.

[0016] In particular, the accuracy and reliability of emotional state recognition can be significantly improved through a dual verification method that combines biosignal data collected from multiple sensors, such as heart rate sensors and skin conductivity sensors, with multi-camera image analysis.

[0017] In addition, through the analysis of past viewing history via the personalization unit and machine learning-based personalized parameter optimization, a differentiated highlight video service reflecting individual user preferences can be provided, and the quality of the sports viewing experience can be enhanced by offering insightful feedback on users' game viewing patterns through the generation of real-time sentiment analysis reports. Brief explanation of the drawing

[0018] The accompanying drawings, included as part of the detailed description to aid in understanding the embodiments, provide various embodiments and explain the technical features of the various embodiments together with the detailed description. FIG. 1 is a schematic diagram illustrating a sports game video recording system (10) according to one embodiment of the present invention. FIG. 2 is a block diagram for explaining the structure of the electronic device (100) of FIG. 1. Figure 3 is a diagram illustrating the multilayer neural network (122) of Figure 2. FIG. 4 is a block diagram for explaining the structure of the user terminal (200) of FIG. 1. FIG. 5 is a block diagram for explaining the structure of the sensor device (400) of FIG. 1. FIG. 6 is a flowchart illustrating the original image data preprocessing process according to one embodiment of the present invention. Figure 7 is a diagram schematically illustrating the process of dividing multiple original image data into frames. FIG. 8 is a flowchart illustrating a method for generating a highlight image through biosignal-based emotion recognition according to an embodiment of the present invention. Specific details for implementing the invention

[0019] The following embodiments are combinations of the components and features of the embodiments in a specific form. Each component or feature may be considered optional unless otherwise explicitly stated. Each component or feature may be implemented in a form not combined with other components or features. Additionally, various embodiments may be constructed by combining some components and / or features. The order of operations described in various embodiments may be changed. Some components or features of one embodiment may be included in another embodiment, or may be replaced with corresponding components or features of another embodiment.

[0020] In the description of the drawings, procedures or steps that could obscure the essence of the various embodiments were not described, nor were procedures or steps that can be understood by a person of ordinary knowledge in the relevant technical field described.

[0021] Throughout the specification, when a part is described as "comprising" or "including" a component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Furthermore, terms such as "...part," "...unit," and "module" as used in the specification refer to a unit that performs at least one function or operation, and this may be implemented in hardware, software, or a combination of hardware and software. Additionally, "one (a or an)," "one," "the," and similar related terms may be used in the context describing various embodiments (particularly in the context of the following claims) in both singular and plural forms, unless otherwise indicated in the specification or clearly contradicted by the context.

[0022] Hereinafter, embodiments according to various examples will be described in detail with reference to the accompanying drawings. The detailed description disclosed below, together with the accompanying drawings, is intended to describe exemplary embodiments of various examples and is not intended to represent the only embodiment.

[0023] In addition, specific terms used in various embodiments are provided to aid in understanding the various embodiments, and the use of such specific terms may be modified in other forms within the scope of not departing from the technical concept of the various embodiments.

[0024] FIG. 1 is a schematic diagram illustrating a sports game video recording system (10) according to one embodiment of the present invention.

[0025] Referring to FIG. 1, the electronic device (100) can receive original image data in real time from the shooting system (300).

[0026] The shooting system (300) may include a plurality of cameras. The plurality of cameras may include a first camera (310), a second camera (320), and a third camera (330). As illustrated in FIG. 1, the first camera (310), the second camera (320), and the third camera (330) may each be installed to photograph the user (P) from different directions. Each of the plurality of cameras may capture still images and video of the user (P). In one embodiment, each of the plurality of cameras may include one or more lenses, image sensors, image signal processors, or flashes. Although FIG. 1 illustrates the shooting system (300) as including three cameras, embodiments of the present invention are not limited thereto. If necessary, the shooting system (300) may include four or more cameras, or two or fewer cameras. However, in order to acquire image data of the user (P) from different angles, it is preferable for the shooting system (300) to include three or more cameras.

[0027] Each of the plurality of cameras included in the shooting system (300) can capture the stationary state or operating state of the user (P) to generate original image data (v_raw). The original image data (v_raw) may refer to data in which the stationary state or moving state of the user (P) is optically recorded.

[0028] In one embodiment, the shooting system (300) may include a first camera (310), a second camera (320) installed to photograph a user (P) from a different direction from the first camera (310), and a third camera (330) installed to photograph a user (P) from a different direction from the first camera (310) and the second camera (320).

[0029] The first camera (310) can capture the user (P) to generate the first original image data (v_raw1). The second camera (320) can capture the user (P) to generate the second original image data (v_raw2). The third camera (330) can capture the user (P) to generate the third original image data (v_raw3). Each image data may include metadata including time information corresponding to the time of image capture, a shooting area, and camera coordinate information. The metadata may be stored in formats such as IPTC (International Press Telecommunications Council), XMP (Extensible Metadata Platform from Adobe), and EXIF ​​(Exchangeable Image File Format).

[0030] In one embodiment, the electronic device (100) provides a membership registration interface to a user terminal (200) and receives user information corresponding to the membership registration interface from the user terminal (200), thereby registering the user to a sports multi-content system implemented by the electronic device (100). The sports multi-content system implemented by the electronic device (100) may include at least one of a multi-angle-based highlight video generation service, a social network-based challenge participation competition service, an abnormal behavior-based sports video event detection service, an emotion feedback-based AI sports video recommendation service, an individual sports discipline-specific highlight automatic generation service, a customized challenge video recommendation service based on individual exercise tendencies, an AI-based game prediction result-linked video summary generation service, a user voice command-based real-time highlight search service, a VR environment immersive sports highlight provision service, a real-time motion correction service based on unstructured exercise data, and a youth exercise data-based growth analysis and coaching video provision service.

[0031] In one embodiment, the electronic device (100) can implement a social network-based challenge participation competition system. The electronic device (100) can induce continuous participation by setting clear rules for the challenges in which users participate, establishing an authentication system capable of fairly judging them, and providing substantial rewards to users. The challenges provided on the challenge participation competition platform implemented by the electronic device (100) can be classified into participation-type challenges and challenge-type challenges. Each challenge may have unique participation conditions, challenge success or execution conditions, and success or execution rewards set.

[0032] In one embodiment, the electronic device (100) can determine whether a user participates in a challenge based on video data captured by a user terminal (200) and determine whether to pay a reward for participating in the challenge.

[0033] In one embodiment, the electronic device (100) may collect original image data from a plurality of user terminals (200). In this case, the original image data (v_raw) may refer to data optically recorded by capturing the stationary or operational state of a user (e.g., athlete, training coach, amateur, etc.) or an object by the camera module (260) of the user terminal (200). The original image data may include metadata including time information corresponding to the time of image capture, a shooting area, and camera coordinate information. The metadata may be stored in formats such as IPTC (International Press Telecommunications Council), XMP (Extensible Metadata Platform from Adobe), and EXIF ​​(Exchangeable Image File Format).

[0034] In one embodiment, the electronic device (100) may collect statistical data from an external server (not shown). Statistic data may refer to factual information recording the performance of a user. For example, when a user plays soccer, there are goals, assists, passes, sprints, aerial duels, etc., and in the case of a batter in baseball, there are hits, home runs, ground balls, flies, bunts, singles, doubles, triples, fouls, etc. The statistical data collected by the electronic device (100) from the external server may be collected in a database format determined according to the sport and the player. Here, the external server may be an official website of a sports league such as baseball, basketball, or soccer, or a data provider (API). In this specification, statistical data may refer to data classified by a user (e.g., player, amateur, training coach, etc.) and a sport (e.g., soccer, basketball, baseball, tennis, golf, etc.), and may include one or more factual information according to the classification.

[0036] FIG. 2 is a block diagram for explaining the structure of the electronic device (100) of FIG. 1. FIG. 3 is a diagram for explaining the multilayer neural network (122) of FIG. 2.

[0037] Referring to FIG. 2, the electronic device (100) may include a user management unit (110), an image discrimination unit (120), a challenge management unit (130), and a reward management unit (140).

[0038] The user management unit (110) can register users to the sports multi-content system through a membership registration process and collect and manage user information. The user management unit (110) can perform user registration and user information input processes. The user management unit (110) can provide a membership registration interface to the user terminal (200) and receive user information from the user terminal (200). Here, the user information may include at least one of the user's name, contact information, date of birth, place of residence, and gender. The user management unit (110) can provide services such as photo uploading, photo editing, payment, and guidance video output to users registered as members.

[0039] The user management unit (110) may further include a ranking calculation module (111) that aggregates and manages rankings based on the performance of users and teams, and a user DB (112) which is a database built by collecting user information received and information regarding the status of users' challenge participation, performance, ranking, etc.

[0040] The ranking calculation module (111) can calculate a ranking based on the reward points obtained from a challenge in which a user or team participated, and can update and aggregate the ranking according to a predetermined period and provide it to the user.

[0041] The user DB (112) may have a general data structure implemented in the storage space (hard disk or memory) of a computer system using a database management program (DBMS). The database may have a data storage form that allows for the free retrieval (extraction), deletion, editing, and addition of data. The database may be implemented in accordance with the purpose of one embodiment of the present disclosure using a relational database management system (RDBMS) such as Oracle, Informix, Sybase, and DB2, an object-oriented database management system (OODBMS) such as Gemston, Orion, and O2, and an XML native database such as Excelon, Tamino, and Sekaiju, and may have appropriate fields or elements to achieve its function.

[0042] The user management unit (110) can classify input user information and build a database. The user management unit (110) can classify input user information and store and manage it in the user DB (112). The user DB (112) can be implemented using server programs provided in various ways depending on operating systems such as DOS, Windows, Linux, UNIX, and Macintosh on general server hardware. Representative examples include Website and IIS (Internet Information Server) used in Windows environments, and CERN, NCSA, and APPACH used in UNIX environments. Additionally, the user DB (112) may be linked with an authentication system and a payment system for user authentication of services provided on a platform according to an embodiment of the present invention or for purchase payment related to services.

[0043] The image discrimination unit (120) collects image data submitted by the user to prove the achievement of the challenge, and can determine whether the user has achieved the challenge by using object detection technology based on the image data.

[0044] The image determination unit (120) may require the user to provide more than a predetermined number of image data to determine whether the user has performed or succeeded in the challenge.

[0045] The image determination unit (120) can analyze the image using an object detection algorithm and determine whether the image submitted by the user satisfies the actual challenge conditions. For example, if the user participates in a basketball challenge, the image determination unit (120) can collect image data from the user and determine whether the user's shot passes accurately through the rim in the image data to determine whether the challenge is successful.

[0046] To this end, the image discrimination unit (120) may further include an image preprocessing unit (121), a multilayer neural network (122), and a learning engine (123).

[0047] The image preprocessing unit (121) can receive original image data and preprocess it to generate preprocessed data. Here, preprocessing may refer to the operation of dividing the original image data into frames and normalizing each frame to generate preprocessed data. The image preprocessing unit (121) can normalize each frame of the original image data by adjusting it to the same pixel size. The original image data can be processed into preprocessed data containing a plurality of normalized frames (f1, f2, f3, …, fn). For example, if the original image data is a 90-minute video captured at 30 frames per second (fps), a total of 162,000 frames can be generated.

[0048] In one embodiment, the image preprocessing unit (121) can preprocess original image data to generate preprocessed data including a plurality of frames. The image preprocessing unit (121) can select one or more frames among the plurality of frames according to a preset period.

[0049] Here, the period may be a predetermined period based on the classification of the original image data. The classification of the original image data may be determined based on the metadata of the original image data. Alternatively, the classification of the original image data may be determined based on statistical data.

[0050] The preset period can vary. For example, the period can be 0.1 seconds, 1 second, 3 seconds, 5 seconds, etc. Or, the period can be in frames. For example, the period can be 10 frames, 20 frames, 30 frames, 120 frames, etc.

[0051] In one embodiment, the image preprocessing unit (121) may apply a color filter to each of one or more selected frames to form a grid. In one embodiment, the color filter may be a filter that divides each of the plurality of frames into a plurality of grids and performs color quantization on each of the plurality of grids to unify them into a single color.

[0052] The image preprocessing unit (121) can store one or more selected frames and gridded frames in memory and database based on the metadata of the original image data.

[0053] The image determination unit (120) can determine whether one or more gridded frames match by comparing each of them with a pre-stored pattern layer, classify the matching frames as event occurrence frames corresponding to the pattern layer, and extract a thumbnail image based on the frames.

[0054] The pattern layer may be a layer containing a pattern corresponding to at least one event included in the original video data. For example, if the original video data is a soccer video, the pattern layer may include a pattern layer corresponding to a goal, a pattern layer corresponding to a shot on target, a pattern layer corresponding to a corner kick, etc.

[0055] In one embodiment, the image discrimination unit (120) compares each of the pattern layer and one or more gridded frames, and if there is no frame that matches the pattern layer, it can select at least one of the one or more frames as a new pattern layer and store it in memory.

[0056] In one embodiment, the image discrimination unit (120) compares each of the pattern layer and one or more gridded frames, and if there is no frame that matches the pattern layer, it can select one or more frames again based on a different period.

[0057] In one embodiment, the image determination unit (120) can determine whether the pattern layer and one or more gridded frames match by converting each of the one or more frames into a frame vector having the color value of the grid as a component and converting the pattern layer pattern value as a component, calculating the similarity between each of the pattern vector and one or more frame vectors, and determining that the pattern layer and one or more gridded frames match if the similarity is greater than or equal to a predetermined threshold.

[0058] In one embodiment, the image discrimination unit (120) may set a region of interest for each of a plurality of frames based on statistical data. Here, 'region of interest' may refer to an area located at a specific coordinate within a frame with a specific size. The coordinates of the region of interest may be defined based on the coordinates of the center pixel of the region of interest.

[0059] In one embodiment, the region of interest (L) may be set based on reference coordinates within the frame. The reference coordinates may be, for example, the center coordinates of the frame or the edge coordinates of the frame. In another embodiment, the region of interest may be set based on a specific event. That is, the region of interest may be set according to a specific event (e.g., scoring, passing, dribbling, etc.) detected in the previous frame or the current frame. In yet another embodiment, the region of interest may be set based on a detected object (target).

[0060] The image discrimination unit (120) can detect the location and movement of an object using an object tracking algorithm or an object tracking filter, and set the area around the object as a region of interest. In another embodiment, the region of interest may be set based on the rate of change of pixel values. That is, the region of interest may be set based on an area where the change in pixel values ​​is large within a specific frame (e.g., an area with fast movement or change).

[0061] The image discrimination unit (120) can pre-supervised train a multilayer neural network (122) using multiple training data. A multilayer neural network is a prediction model implemented in software or hardware that mimics the computational power of a biological system using a large number of artificial neurons (or nodes).

[0062] The learning engine (123) can supervise the multilayer neural network (122) using learning data in which image data and correct answer data are input values ​​and challenge achievement result data is output value, so that the multilayer neural network (122) can accurately determine whether a challenge is achieved based on image data.

[0063] In this context, supervised learning refers to a learning process that uses data containing input and corresponding output values ​​as training data to find the output value corresponding to a given input; it signifies learning that takes place while the correct answer is known. In supervised learning, the set of input and output values ​​provided is called training data.

[0064] Referring to FIG. 3, the multilayer neural network (122) may include an input layer, one or more hidden layers, and an output layer.

[0065] In one embodiment, the multilayer neural network (122) may include an input layer that receives image data and has nodes corresponding to a plurality of grids of one or more frames, a first hidden layer that outputs by multiplying each output value of the input layer by a weight and adding a bias, a second hidden layer that outputs by multiplying each output value of the first hidden layer by a weight and adding a bias, and an output layer that outputs by multiplying each output value of the second hidden layer by a weight and outputting the result using an activation function. Although only two hidden layers are shown in FIG. 4, one or more hidden layers may include a greater number of hidden layers in addition to the first hidden layer and the second hidden layer.

[0066] For example, the activation function may be a Softmax function, but embodiments of the present invention are not limited thereto, and the activation function may be various other functions such as a LeRU function. Weights and biases may be continuously updated by supervised learning. Specifically, the output vector may be input to a loss function layer connected to the output layer. The loss function layer may output a loss value using a loss function that compares the output vector with the correct answer vector for each training data. The parameters of the multilayer neural network (122) may be supervised learning in a direction that reduces the loss value.

[0067] A learning model may include an input layer, one or more hidden layers, and an output layer. Training data is input into the input layer, passes through one or more hidden layers and the output layer to produce an output vector, and the output vector may be input into a loss function layer connected to the output layer. The loss function layer may output a loss value using a loss function that compares the output vector with the correct answer vector for each training data. The parameters of the learning model may be trained in a direction that reduces the loss value.

[0068] To this end, the image discrimination unit (120) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (100) itself where the artificial intelligence model is executed, or through a separate server.

[0069] Learning algorithms may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above. An artificial intelligence model may include multiple artificial neural network layers. An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0070] The challenge management unit (130) can create a challenge and set and manage the rules and conditions of the challenge. The challenge management unit (130) can also create a challenge in response to a user's request. Alternatively, the challenge management unit (130) can receive data regarding the creation of a challenge from an external server (not shown) and create a challenge based on this.

[0071] The challenge management unit (130) can set detailed conditions such as the goal, time limit, and number of participants for a challenge registered on the platform, or receive and store detailed conditions such as the goal, time limit, and number of participants for a challenge registered on the platform from an external source. For a challenge registered on the platform, unique participation conditions, challenge success or execution conditions, and success or execution rewards may be set.

[0072] For example, the challenge management department (130) can set the points required for a user to participate in a challenge, and when a user applies to participate in a challenge, the pre-specified points for the user can be deducted.

[0073] In one embodiment, the challenges created and managed by the challenge management unit (130) can be classified into participation-type challenges and challenge-type challenges. Participation-type challenges and challenge-type challenges differ in the user's method of participation and the criteria for determining success.

[0074] A participatory challenge refers to a challenge in which a user applies for a specific challenge, uploads a verification video to prove that they have met the conditions for performing the challenge, and the video discrimination unit (120) evaluates whether the user has participated according to the challenge criteria. A participatory challenge is recognized as having participated in the challenge when the user has met all the pre-set conditions for performing the challenge. For example, in the case of a running challenge, the user runs to meet the distance and time pre-specified for the running challenge, and then submits a video to the electronic device (100) certifying that the running was performed according to the conditions for performing the challenge, so the video discrimination unit (120) can determine whether the conditions for performing the participatory challenge have been met. Since a participatory challenge only examines whether the user has fulfilled the required conditions for performing the challenge, the user is considered to have participated in the challenge if only the conditions for performing the challenge are met.

[0075] On the other hand, a challenge-type challenge is a method in which a user applies for a specific challenge, performs a challenging mission, and proves that they have succeeded. A challenge-type challenge strictly evaluates whether the user has achieved the mission's goal, and the user transmits a video that can certify the mission's success to an electronic device (100), and the success of the challenge mission is automatically checked by a video discrimination unit (120). That is, unlike a participation-type challenge, a challenge-type challenge goes beyond simply satisfying conditions; it is recognized as successful only if the user uploads a video of successfully performing the mission and that video passes the pre-set challenge success conditions. For example, if a mission to make 10 consecutive 3-point shots is given in a basketball challenge, the user performs the mission and uploads a video of the success, and the success is determined through a review process.

[0076] Participatory challenges that can be created by the challenge management department (130) include running challenges, going to the gym challenges, and playing badminton challenges. Since participatory challenges only determine whether the challenge is performed, participation in the challenge can be verified through a relatively simple process.

[0077] Challenge-type challenges that can be generated by the challenge management department (130) include a basketball shooting challenge in which one must make 10 three-point shots during a given time, a plank challenge in which one must maintain a plank position for a certain period of time, and a machine exercise repetition challenge.

[0078] In one embodiment, the challenge type provides the user with an option to select a difficulty level, and may provide a differential reward upon successful completion of the challenge according to the selected difficulty level. For example, in the case of a basketball shooting challenge where one must make 10 three-point shots within a given time, the difficulty levels can be classified as easy, normal, and hard, and conditions can be set as follows: easy: 15 points reward, 5 minutes time limit, 10 successes; normal: 30 points reward, 3 minutes time limit, 10 successes; hard: 50 points reward, 2 minutes time limit, 10 successes. As another example, in the case of an exercise (bench press) repetition challenge, the difficulty can be classified into easy, medium, and hard, and conditions can be set as follows: Easy: 15 reward points, 10 repetitions with 50kg, Medium: 30 reward points, 10 repetitions with 70kg, Hard: 50 reward points, 10 repetitions with 100kg.

[0079] In one embodiment, the number of times a challenge can be attempted may be limited. For example, the number of attempts may be limited to once a day or three times a week.

[0080] The reward management department (140) can establish and manage a reward system that provides points and rewards to users who successfully complete a challenge.

[0081] For example, the reward management department (140) can award a pre-specified point to the user when the user succeeds in a challenge.

[0082] The database (150) can comprehensively store and manage learning data, standard data, and analysis data necessary for various sports multi-content services provided by the electronic device (100). The database (150) can be linked with the user DB (112) to establish a data foundation for providing customized services tailored to the characteristics of individual users.

[0083] In one embodiment, the database (150) can be constructed by storing a dataset including an abnormal behavior detection dataset, a specialized dataset by sport, an emotion analysis standard dataset, an exercise tendency classification dataset, a growth analysis dataset, a game prediction model dataset, and a voice command metadata set.

[0084] Here, the abnormal behavior detection dataset may include training data for identifying abnormal user behavior patterns such as falling, leaving one's seat, slumping, and abrupt cessation of movement.

[0085] Specialized datasets for each sport may include data regarding the unique rules, major events, game progression, and highlight generation criteria of each sport, such as soccer, basketball, tennis, and baseball.

[0086] The sentiment analysis reference dataset may include reference data that maps biosignals, such as facial expressions, brainwaves, and heart rate, to the user's emotional state.

[0087] The exercise tendency classification dataset may include classification criteria for recommending customized challenges by analyzing individual exercise patterns, preferred sports, difficulty levels, and participation history.

[0088] The growth analysis dataset may include age- and gender-specific data on physical development indicators such as muscular endurance, flexibility, power, and coordination for children, as well as reference data for analyzing growth trends.

[0089] In one embodiment, the database (150) may be implemented using a relational database management system (RDBMS) such as Oracle, Informix, Sybase, DB2, or an object-oriented database management system (OODBMS) such as Gemston, Orion, O2.

[0090] The sensing processing unit (160) can collect and process various biosignals and behavioral data of the user in real time to provide advanced services such as emotion analysis, motion correction, and abnormal behavior detection. The sensing processing unit (160) can acquire and analyze additional sensing data in addition to the image data collected from the shooting system (300) and the user terminal (200).

[0091] To this end, the sensing processing unit (160) may include a biosignal sensing module, an eye tracking sensing module, a motion recognition sensing module, a voice recognition module, and an emergency detection module.

[0092] The biosignal sensing module may include at least one of a facial recognition sensor for analyzing a user's facial expressions, an EEG sensor for measuring brain waves, and a heart rate sensor for measuring heart rate. The biosignal sensing module can determine the user's emotional state in real time based on sensing data from the sensors.

[0093] The eye-tracking sensing module detects the user's gaze direction and level of concentration to identify the point of interest and gaze of the user in multi-angle video.

[0094] The motion recognition sensing module can analyze the user's accurate posture and movement using 3D motion capture technology, detect incorrect posture during atypical movements, and provide real-time corrective feedback.

[0095] The voice recognition module can recognize the user's voice commands and process them using natural language to provide functions for searching specific video segments and controlling playback.

[0096] The emergency detection module can automatically detect emergency situations based on the results of abnormal behavior detection and, if necessary, send automatic notifications to medical personnel or emergency services.

[0097] In one embodiment, the sensing processing unit (160) can fuse data collected from a plurality of sensing modules in real time to perform more accurate user status determination.

[0098] The personalization unit (170) can provide customized content and services by analyzing the individual characteristics, exercise history, preferences, and tendencies of the user. To this end, the personalization unit (170) may include an emotion analysis engine, a game prediction AI, a motion correction engine, a growth analysis engine, a natural language processing engine, a customized recommendation system, and an adaptive learning module.

[0099] The emotion analysis engine can analyze biometric data collected from the sensing processing unit (160) in real time to identify the user's current emotional state and automatically recommend customized highlight videos or challenges accordingly.

[0100] Match prediction AI can predict match results by comprehensively analyzing player stats, team strength, and recent performance, and generate a summary video comparing the predicted result with actual match footage.

[0101] The motion correction engine analyzes the user's exercise movements in real time to detect abnormal posture during atypical exercises, such as stretching and flexibility exercises, and can recommend methods to correct the posture.

[0102] The growth analysis engine can periodically analyze exercise performance data of youth users to visualize physical development trends and generate personalized coaching content.

[0103] In one embodiment, the personalization unit (170) can automatically adjust the difficulty of the challenge according to the user's exercise ability level to maintain an appropriate sense of challenge.

[0104] The multimedia management unit (180) can generate an image of at least one of VR, AR, and XR based on original image data collected from the shooting system (300) and provide it to the user terminal (200).

[0105] The multimedia management unit (180) may include a virtual image generation module, a multi-angle synchronization engine, a real-time highlight editor, a prediction-actual comparison image generator, and a highlight generator for each item.

[0106] The virtual video generation module can render key scenes of a sports game in a 360-degree stereoscopic view. For example, the virtual video generation module can generate VR, AR, and XR videos that vividly recreate key moments, such as goal scenes and blocking scenes, from multiple angles.

[0107] The multi-angle synchronization engine performs time synchronization of images collected from the first camera (310), the second camera (320), and the third camera (330), and can generate personalized highlights in real time at the point in time and focus desired by the user based on the eye tracking data of the sensing processing unit (160).

[0108] The real-time highlight editor can automatically extract and edit key moments of each sport, such as soccer goals, basketball dunks, and tennis aces, by utilizing an automatic editing algorithm that reflects the characteristics of each sport.

[0109] The prediction-actual comparison video generator can compare and analyze the game prediction AI results of the personalization unit (170) with the actual game video to generate a summary video centered on key scenes different from the prediction.

[0110] In one embodiment, the multimedia management unit (180) can immediately search for and play a video segment corresponding to a specific player, a specific time period, or a specific event according to the user's voice command.

[0111] In one embodiment, the multimedia management unit (180) can accurately interpret natural language commands such as "the most impressive goal scene in today's match" or "second half comeback situation" by linking the metadata of the video with the natural language processing results, and provide the corresponding video.

[0112] The electronic device (100) may have the same configuration as a conventional web server or WAP server in terms of hardware. However, in terms of software, it may include program modules that perform various functions and are implemented through any language such as C, C++, Java, Visual Basic, Visual C, etc. Furthermore, the electronic device (100) generally refers to a computer system that is connected to an unspecified number of clients and / or other servers through an open computer network such as the Internet, receives requests for task execution from clients or other servers, and derives and provides the results of the task, and computer software (server program) installed for that purpose. In addition, the electronic device (100) should be understood as a broad concept that includes, in addition to the aforementioned server program, a series of application programs operating on the electronic device (100) and, in some cases, various databases (DB: Database, hereinafter referred to as "DB") built internally or externally.

[0114] FIG. 4 is a block diagram for explaining the structure of the user terminal (200) of FIG. 1.

[0115] Referring to FIG. 4, the user terminal (200) may include a processor (210), memory (220), communication module (230), antenna module (240), touch display (250), and camera module (260). In one embodiment, at least one of these components may be omitted from the user terminal (200), or one or more other components may be added. In another embodiment, some of these components may be integrated into a single component.

[0116] The processor (210) can, for example, execute software to control at least one other component of a user terminal (200) connected to the processor (210) and perform various data processing or operations. In one embodiment, as at least part of the data processing or operations, the processor (210) can store commands or data received from other components in volatile memory (221), process the commands or data stored in volatile memory (221), and store result data in non-volatile memory (222). In one embodiment, the processor (210) may include a main processor (211) (e.g., a central processing unit or an application processor) or an auxiliary processor (212) that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), a sensor hub processor). For example, if the user terminal (200) includes a main processor (211) and an auxiliary processor (212), the auxiliary processor (212) may be configured to use less power than the main processor (211) or to be specialized for a designated function. The auxiliary processor (212) may be implemented separately from the main processor (211) or as part thereof.

[0117] The auxiliary processor (212) can control at least some of the functions or states associated with at least one of the components of the user terminal (200), for example, on behalf of the main processor (211) while the main processor (211) is in an inactive state, or together with the main processor (211) while the main processor (211) is in an active state. In one embodiment, the auxiliary processor (212) may be implemented as part of another functionally related component.

[0118] In one embodiment, if the auxiliary processor (212) is a neural network processing device, the auxiliary processor (212) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the user terminal (200) itself where the artificial intelligence model is executed, or through a separate server (e.g., electronic device (100)).

[0119] Learning algorithms may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above. An artificial intelligence model may include multiple artificial neural network layers. An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0120] The memory (220) can store various data used by at least one component of the user terminal (200). The data may include, for example, input data or output data for software and related commands. The memory (220) may include volatile memory (221) or non-volatile memory (222).

[0121] The communication module (230) can support the establishment of a wired or wireless communication channel between a user terminal (200) and an external electronic device (e.g., electronic device (100)), and the performance of communication through the established communication channel. The communication module (230) operates independently of the processor (210) and may include one or more communication processors that support wired or wireless communication. In one embodiment, the communication module (230) may include a wireless communication module or a wired communication module. The corresponding communication module among these communication modules can communicate with the electronic device (100) through a network. These various types of communication modules may be integrated into a single component or implemented as multiple separate components.

[0122] The wireless communication module can support 5G networks following 4G networks and next-generation communication technologies, such as new radio access technology (NR access technology). NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module can support high-frequency bands, for example, to achieve high data transmission rates. The wireless communication module can support various technologies to secure performance in high-frequency bands, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beamforming, or large-scale antenna. The wireless communication module can support various requirements specified in the user terminal (200), external electronic device, or network system. In one embodiment, the wireless communication module can support a peak data rate for eMBB realization, lossy coverage for mMTC realization, or U-plane latency for URLLC realization.

[0123] The antenna module (240) can transmit a signal or power to an external electronic device or receive it from the outside. In one embodiment, the antenna module (240) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate. In one embodiment, the antenna module (240) may include a plurality of antennas. In this case, at least one antenna suitable for a communication method used in a communication network, such as a network, may be selected from the plurality of antennas, for example, by a communication module (230). The signal or power may be transmitted or received between the communication module (230) and the external electronic device through the selected at least one antenna.

[0124] In one embodiment, commands or data may be transmitted or received between electronic devices (100) connected to a network. In one embodiment, all or part of the operations executed at a user terminal (200) may be executed at an electronic device (100).

[0125] For example, when a user terminal (200) needs to perform a function or service automatically or in response to a request from a user or another device, the user terminal (200) may request one or more external electronic devices to perform at least a part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the request may perform at least a part of the requested function or service, or additional functions or services related to the request, and transmit the result of the execution to the user terminal (200). The user terminal (200) may provide the result as is or additionally processed and as at least part of the response to the request. To this end, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The user terminal (200) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing.

[0126] Here, the network refers to a connection structure capable of exchanging information between each node, such as terminals and servers, or a network connecting a user terminal (200) and an electronic device (100). The network includes, but is not limited to, the Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), 3G, 4G, LTE, 5G, Wi-Fi, etc. The network may be a closed network such as a LAN or WAN, but it is preferable that it be an open network such as the Internet. The Internet refers to a global open computer network structure that provides the TCP / IP protocol and various services existing in the upper layer, namely HTTP (HyperText Transfer Protocol), Telnet, FTP (File Transfer Protocol), DNS (Domain Name System), SMTP (Simple Mail Transfer Protocol), SNMP (Simple Network Management Protocol), NFS (Network File Service), and NIS (Network Information Service).

[0127] The touch display (250) may be an LCD (liquid crystal display) panel, an OLED (organic light emitting diode) panel, etc. The touch display may also include a touch screen panel (TSP) that includes a touch electrode for detecting a touch on the display panel. In one embodiment, the touch display may be an in-cell type in which the display panel and the touch panel are combined integrally. However, this is exemplary and the embodiments of the present invention are not limited thereto. The touch display may electrically recognize contact such as a finger or pen, generate an electrical signal, and transmit it to a processor.

[0128] The camera module (260) can capture still images and video. In one embodiment, the camera module (260) may include one or more cameras. Each of the one or more cameras may include one or more lenses, image sensors, image signal processors, or flashes. Depending on the needs, the camera module (260) may include one camera or two or more different types of cameras. The camera module (260) can generate original image data by capturing an authentication video for challenge participation authentication.

[0130] FIG. 5 is a block diagram for explaining the structure of the sensor device (400) of FIG. 1.

[0131] Referring to FIG. 5, the sensor device (400) may be composed of wearable devices that collect a user's biometric information and may include a heart rate sensor (410), an accelerometer (420), a gyroscope sensor (430), a GPS module (440), a bioelectric sensor (450), a temperature sensor (460), a pressure sensor (470), and an oxygen saturation sensor (480). The sensor device (400) according to one embodiment of the present invention may include all of the above components or selectively configure only some of the sensors as needed, depending on the characteristics of the exercise or the purpose of monitoring.

[0132] The heart rate sensor (410) is a sensor that monitors the user's heart rate in real time and may use at least one of the photoplethysmography (PPG) method or the electrocardiogram (ECG) method. The heart rate sensor (410) detects changes in blood flow using green light (425 nm) and infrared (940 nm) LEDs and can measure a heart rate range of 30 to 220 beats per minute with an accuracy of within ±2 bpm.

[0133] Specifically, the heart rate sensor (410) has a built-in 16-bit ADC (Analog-to-Digital Converter) capable of acquiring high-resolution signals of more than 1,000 samples per second, and can remove motion artifacts in real time through a DSP (Digital Signal Processing) chip. The heart rate sensor (410) can evaluate autonomic nervous system activity through Heart Rate Variability (HRV) analysis and calculate HRV indicators such as RMSSD, pNN50, and SDNN to quantify the user's stress state and fatigue level.

[0134] In one embodiment, the heart rate sensor (410) can detect arrhythmias such as atrial fibrillation by measuring a single-lead electrocardiogram for 30 seconds in a manner similar to the Digital Crown-based ECG sensor of the Apple Watch. In another embodiment, the heart rate sensor (410) is implemented in the form of a chest strap such as Polar H10 to achieve a heart rate measurement accuracy of 99% or higher. In yet another embodiment, the heart rate sensor (410) can simultaneously measure blood hemoglobin concentration and oxygen saturation using a multi-wavelength LED array.

[0135] The acceleration sensor (420) is a MEMS (Micro-Electro-Mechanical Systems) based sensor that measures the user's movement and posture changes in three axes (X, Y, Z axes) and can support a measurement range from ±2g to ±16g. The acceleration sensor (420) supports a sampling frequency of up to 8kHz to precisely detect high-speed motion or impact, and can measure minute changes in acceleration of less than 0.001g through 16-bit resolution.

[0136] Specifically, the acceleration sensor (420) can be implemented with a high-performance IMU chip such as the Bosch BMI270 or STMicroelectronics LSM6DSO, and can provide temperature compensation and vibration immunity, enabling stable measurements in various environments. The acceleration sensor (420) can automatically classify the user's posture (standing, sitting, lying down, walking, running) through gravity vector analysis and identify the type of exercise through an activity pattern recognition algorithm.

[0137] The acceleration sensor (420) has a fall detection algorithm built in to detect a sudden change in acceleration (more than 3g) and a zero-gravity state (less than 0.5g) continuously, and can determine that it is a fall. In addition, the acceleration sensor (420) can provide additional functions such as counting steps, estimating calorie consumption, and analyzing sleep stages.

[0138] In one embodiment, the acceleration sensor (420) is equipped with a machine learning-based activity recognition model and can classify more than 20 types of exercise movements with an accuracy of more than 95%. In another embodiment, the acceleration sensor (420) can detect muscle tremors or spasms through vibration pattern analysis to detect neurological abnormalities early. In yet another embodiment, the acceleration sensor (420) is designed with a waterproof structure that operates even during underwater exercise and can be applied to water sports such as swimming and diving.

[0139] The gyroscope sensor (430) is a sensor that measures the user's rotational motion and angular velocity in three axes and can support a measurement range from ±250 dps to ±2000 dps (degrees per second). The gyroscope sensor (430) provides 16-bit resolution and a sampling frequency of up to 8 kHz, so it can accurately track fast rotational motions, and through a temperature drift compensation function, it can enable stable measurements for a long time.

[0140] Specifically, the gyroscope sensor (430) can be implemented in a form integrated into a 6-axis or 9-axis IMU such as InvenSense ICM-20948 or Bosch BMI260, and can improve attitude estimation accuracy through sensor fusion with the accelerometer sensor (420). The gyroscope sensor (430) can prevent gimbal lock and provide stable 3D orientation information by applying a quaternion-based attitude estimation algorithm and a Kalman filter.

[0141] The gyroscope sensor (430) can analyze the user's rotational movement pattern to precisely measure the trajectory and speed of complex movement actions such as golf swings, tennis strokes, and baseball batting. Additionally, the gyroscope sensor (430) can calculate static and dynamic balance indices for evaluating the sense of balance and detect dizziness or balance disorders early.

[0142] In one embodiment, the gyroscope sensor (430) can provide relative direction information without a magnetic sensor by applying the Attitude and Heading Reference System (AHRS) algorithm. In another embodiment, the gyroscope sensor (430) can provide diagnostic assistance information to medical staff by quantifying hand tremors, which are an early symptom of Parkinson's disease, through vibration analysis. In yet another embodiment, the gyroscope sensor (430) can be utilized as a head tracking sensor for VR / AR applications to provide an immersive exercise experience.

[0143] The GPS module (440) is a module that tracks the user's exact location and movement path through a satellite navigation system and can support multiple GNSS (Global Navigation Satellite Systems) such as GPS, GLONASS, Galileo, BeiDou, and QZSS. The GPS module (440) provides positional accuracy within a minimum of 1 meter, and can achieve centimeter-level precision when differential GPS (DGPS) or RTK (Real-Time Kinematic) technology is applied.

[0144] Specifically, the GPS module (440) can be implemented as a multi-band GNSS receiver such as the u-blox M8, M9 series or Broadcom BCM47755, and can correct ionospheric errors by simultaneously receiving L1, L2, and L5 frequency bands. The GPS module (440) supports an update frequency of 10 Hz or more per second, enabling accurate position tracking even during high-speed movement, and can provide fast satellite acquisition times within 5 seconds for a Cold Start and within 1 second for a Hot Start.

[0145] The GPS module (440) can calculate and store the user's exercise path, distance traveled, average speed, maximum speed, altitude change, etc. in real time. In addition, the GPS module (440) can automatically generate an alert when the user moves out of a pre-set safe zone through a geo-fencing function, and can transmit accurate location information to a rescue team in case of an emergency.

[0146] In one embodiment, the GPS module (440) can reduce satellite acquisition time even indoors or in urban areas by utilizing auxiliary data from a cellular network through an Assisted GPS (A-GPS) function. In another embodiment, the GPS module (440) can enable continuous location tracking even in tunnels or underground sections where GPS signals are blocked through fusion with an Inertial Navigation System (INS). In yet another embodiment, the GPS module (440) can provide performance analysis by section by linking with detailed map information such as golf courses, running tracks, and cycling routes as a sports-specific function.

[0147] The bioelectric sensor (450) is a sensor that measures the user's electromyography (EMG), electroencephalography (EEG), and electrodermal activity (EDA), and can quantify muscle activity and mental stress state. The bioelectric sensor (450) has a 24-bit ADC and a noise level of 1 Ω or less, so it can accurately detect even minute biosignals, and can have a built-in 50Hz / 60Hz power noise removal filter.

[0148] The temperature sensor (460) is a dual sensor that simultaneously measures the user's body temperature and the ambient temperature, and can support a measurement accuracy of ±0.1°C and a measurement range of -40°C to 125°C. The temperature sensor (460) can detect danger signals such as fever, dehydration, and heatstroke at an early stage by analyzing the body temperature change pattern, and can calculate a heat stress index based on exercise intensity.

[0149] The pressure sensor (470) is a sensor that detects contact pressure applied by a user along with altitude estimation through atmospheric pressure measurement, and can measure altitude with an accuracy of ±1m with a resolution of 1 hPa or less. The pressure sensor (470) can monitor the risk of altitude sickness in sports with large altitude changes such as climbing, skiing, and paragliding, and can evaluate walking patterns and balance through plantar pressure analysis.

[0150] The oxygen saturation sensor (480) is a sensor that non-invasively measures blood oxygen saturation using pulse oximetry and can provide an accuracy of ±2% in the range of 70% to 100%. The oxygen saturation sensor (480) calculates the ratio of oxidized hemoglobin to reduced hemoglobin using the difference in absorption rates between red light (660 nm) and infrared light (940 nm), and can detect hypoxia or respiratory disease early.

[0151] All sensors of the sensor device (400) are connected to the electronic device (100) via wireless communication of BLE (Bluetooth Low Energy) 5.0 or higher, and can provide adaptive sampling frequency adjustment and sleep mode switching functions to optimize battery life. The collected multi-sensor data undergoes time synchronization and calibration processes in the electronic device (100) and is fused with the video analysis results to enable more accurate and reliable detection of abnormal behavior.

[0153] FIG. 6 is a flowchart illustrating the process of preprocessing original image data according to an embodiment of the present invention. FIG. 7 is a diagram schematically illustrating the process of dividing a plurality of original image data into frames.

[0154] First, in step S610, the image preprocessing unit (121) of the electronic device (100) can divide the received original image data (v_raw) into frames.

[0155] In step S620, the image preprocessing unit (121) of the electronic device (100) can generate preprocessed data by normalizing each frame (f). The electronic device (100) can normalize each frame of the original image data by adjusting it to the same pixel size. For example, if the original image data is a 90-minute video captured at 30 frames per second (fps), a total of 162,000 frames can be generated.

[0156] In step S630, the image preprocessing unit (121) of the electronic device (100) can divide the preprocessed data into a plurality of consecutive frames to generate a continuous frame set (cf).

[0157] In one embodiment, a continuous frame set (cf) may be defined as a collection of frames captured at consecutive time intervals. For example, frames constituting a video over one second may be defined as a single "continuous frame set."

[0158] Referring to FIG. 7, an exemplary method is illustrated for generating a first continuous frame set (cf1) including f1, f2, and f3 among a plurality of frames (f) generated by preprocessing the first original image data (v_raw1), and generating a second continuous frame set (cf2) including f1, f2, and f3 among a plurality of frames (f) generated by preprocessing the second original image data (v_raw2). In FIG. 7, L1 and L2 represent a first region of interest and a second region of interest, respectively.

[0159] In another embodiment, the electronic device (100) may generate a continuous frame set based on the time of occurrence of a specific event. The specific event may be, for example, a scoring scene in soccer, a shooting motion in basketball, abnormal behavior, sudden behavior, emergency behavior, etc. In this case, the image preprocessing unit (121) may detect the occurrence of a specific event in conjunction with other components (e.g., a sensing processing unit (160)), and generate a continuous frame set including a frame preceding the detected frame and a frame following the detected frame based on the frame (e.g., f2) in which the occurrence of the specific event is detected.

[0160] In another embodiment, the electronic device (100) can generate a continuous frameset based on a section in which a specific object (e.g., ball, player, goalpost, racket, etc.) is recognized or tracked based on an object recognition and object tracking algorithm. Similarly, the image preprocessing unit (121) can recognize a specific object in conjunction with another component (e.g., sensing processing unit (160)) and generate a continuous frameset including a frame preceding the recognized frame and a frame following the recognized frame, based on the frame in which the specific object is recognized (e.g., f2).

[0161] The electronic device (100) can extract a video clip corresponding to preset operation information or event information from a generated continuous frame set. The electronic device (100) can divide a video segment into video clips of a specific time unit.

[0163] FIG. 8 is a flowchart illustrating a method for generating a highlight image through biosignal-based emotion recognition according to an embodiment of the present invention.

[0164] In S810, the image preprocessing unit (121) can recognize the user's emotional state among the original image data based on the sensing data sensed by the sensor device (400). The sensor device (400) can be provided inside the user terminal (200) or configured in the form of an external wearable device, and can measure the user's biosignal in real time.

[0165] Specifically, the sensor device (400) may include at least one of a heart rate sensor (410), a skin conductivity sensor (420), an accelerometer (430), and a gyroscope (440).

[0166] The heart rate sensor (410) can generate heart rate sensing data by measuring the user's heart rate (BPM, Beats Per Minute) and heart rate variability (HRV, Heart Rate Variability).

[0167] The skin conductivity sensor (420) can generate skin conductivity sensing data by measuring the electrical conductivity of the user's skin surface, and the skin conductivity sensing data can indicate the user's level of arousal and stress state.

[0168] The image preprocessing unit (121) can classify the user's emotional state by comparing the sensing data with a preset threshold value. In one embodiment, if the heart rate sensing data increases by more than 20% compared to the normal heart rate and the skin conductivity sensing data is 2.5 microsiemens or higher, the user's emotional state can be determined to be 'excited'. Here, in microsiemens (↓S), Siemens (S) means the reciprocal of skin resistance (1 / Ù).

[0169] In another embodiment, if the heart rate sensing data changes by less than 10% compared to the normal heart rate and the skin conductivity sensing data is less than 1.0 microsiemen, the user's emotional state can be determined as 'calm'.

[0170] In another embodiment, the accuracy of emotional state recognition can be improved by additionally considering motion sensing data obtained from the accelerometer (430) and the gyroscope (440). For example, if a user exhibits sudden body movements while watching a game, this may be highly likely to be associated with a goal situation or an important game moment; therefore, the image preprocessing unit (121) can improve the accuracy of emotional state recognition by additionally considering motion sensing data obtained from the accelerometer (430) and the gyroscope (440).

[0171] In S820, the image preprocessing unit (121) can calculate an emotional weight based on the recognized emotional state of the user. Here, the emotional weight is an indicator that quantifies the degree of change in the user's biosignal, and may represent the intensity of the user's emotional response as a real value in the range of 0.0 to 5.0.

[0172] Specifically, if the emotional weight is less than 1.0, it can be classified as a 'calm' state; if it is 1.0 or more but less than 2.0, it can be classified as an 'interested' state; if it is 2.0 or more but less than 3.0, it can be classified as an 'excited' state; and if it is 3.0 or more, it can be classified as a 'highly excited' state. The emotional weight can be used as an indicator to determine the importance of highlights in each consecutive frame set, and the video preprocessing unit (121) can process the frames in a way that increases the priority of the video of the corresponding time interval to be included in the highlight final video as the emotional weight increases.

[0173] Specifically, the image preprocessing unit (121) can calculate the emotional weight using the following <mathematical formula>.

[0174] [Mathematical Formula]

[0175]

[0176] In the <mathematical formula>, E_weight represents the emotional weight, w_1, w_2, and w_3 represent weighting factors for heart rate, skin conductivity, and motion intensity, respectively, HR_ratio represents the ratio of the current heart rate to the normal heart rate, GSR_norm represents the normalized skin conductivity value, and Motion_intensity represents the motion intensity calculated based on acceleration and gyroscope data.

[0177] In one embodiment, w_1 may have a value of 0.6, w_2 may have a value of 0.3, and w_3 may have a value of 0.1.

[0178] In another embodiment, the weighting factors can be set differently depending on the user's personal characteristics or the sport. For example, in the case of a soccer match, the w_2 value can be increased to 0.4 to amplify the effect of skin conductivity.

[0179] For example, the process of calculating emotional weights based on data measured during goal scenes in a soccer match is explained as follows.

[0180] If the user's normal resting heart rate is 70 BPM and the current heart rate in the bone scene is measured as 91 BPM, the HR_ratio can be 91 / 70 = 1.3. Additionally, if the value measured by the skin conductivity sensor (420) is 3.2 microsiemens, the GSR_norm can be 3.2 / 0.5 = 6.4, which is the value normalized by dividing the measured value by the reference value of 0.5 microsiemens. If the Motion_intensity is calculated as 0.85, indicating that the user's body movement is active as measured by the accelerometer (430) and gyroscope (440), applying w_1=0.6, w_2=0.3, and w_3=0.1 to the above <mathematical formula> can be calculated as follows.

[0181] E_weight = 0.6 × (1.3 - 1) + 0.3 × 6.4 + 0.1 × 0.85 = 0.6 × 0.3 + 0.3 × 6.4 + 0.1 × 0.85 = 0.18 + 1.92 + 0.085 = 2.185

[0182] Since the calculated emotional weight of 2.185 falls within the range of 2.0 or more and less than 3.0, the user's emotional state at that time can be classified as 'excited'.

[0183] Conversely, assuming that during a calm phase of the game, the user's current heart rate is 73 BPM (HR_ratio = 73 / 70 = 1.043), skin conductivity is 0.8 microSiemens (GSR_norm = 0.8 / 0.5 = 1.6), and Motion_intensity is 0.2, E_weight = 0.6 × (1.043 - 1) + 0.3 × 1.6 + 0.1 × 0.2 = 0.026 + 0.48 + 0.02 = 0.526, which can be classified as a 'calm' state of less than 1.0.

[0184] In S830, the image preprocessing unit (121) can generate an emotion-reflecting continuous frame set by reconstructing the existing continuous frame set based on the calculated emotion weight. The emotion-reflecting continuous frame set may preferentially include frames in time intervals where the user's emotional state appears high.

[0185] Specifically, the image preprocessing unit (121) can accurately map the time-based emotional weight to the image frame at the corresponding point in time by synchronizing the timestamp information included in the sensing data collected from the sensor device (400) with the frame-by-frame time information of the original image data.

[0186] For example, the emotional weight calculated based on the sensing data measured by the sensor device (400) at the time when 2 minutes and 13 seconds have elapsed from the time when shooting started by the shooting system (300) is mapped to the frames corresponding to the 2 minutes and 13 seconds interval of the original video data, and can be used as a basis for selecting the frames as highlight candidates.

[0187] Specifically, the image preprocessing unit (121) can divide the entire original image data into multiple time intervals in chronological order and map an emotion weight corresponding to each time interval. Frames in time intervals where the emotion weight is greater than or equal to a preset threshold (e.g., 1.5) can be selected as highlight candidate frames.

[0188] In one embodiment, the image preprocessing unit (121) can sort time intervals in order of highest emotion weight and select frames from the top 20% of intervals first to generate an emotion-reflecting continuous frame set.

[0189] In another embodiment, time intervals with an emotional weight of 2.0 or higher can be frame-selected with an extended length (e.g., an additional 5 seconds before and after) to provide richer highlight scenes.

[0190] In another embodiment, if the interval between consecutive high-emotion segments is less than 10 seconds, they can be integrated into a single continuous frame set to create a natural highlight flow. In one embodiment, the image preprocessing unit (121) can perform emotional state recognition for each of the original image data captured by different cameras. As illustrated in FIG. 1, when the shooting system (300) includes three cameras, the user's emotional state can be recognized for each of the first original image data (v_raw1) generated by the first camera (310), the second original image data (v_raw2) generated by the second camera (320), and the third original image data (v_raw3) generated by the third camera (330).

[0191] Accordingly, the image preprocessing unit (121) can generate a first emotional state determination result, a second emotional state determination result, and a third emotional state determination result. Here, the first to third emotional state determination results must be results derived from frames captured by different cameras at the same time. The image preprocessing unit (121) can calculate an emotional weight for each of the first to third emotional state determination results.

[0192] In one embodiment, the image preprocessing unit (121) can calculate an emotion weight by applying a correction coefficient based on the shooting angle and distance of each camera included in the shooting system (300). For example, when the first camera (310) shoots the front of the user, it is advantageous for detecting changes in facial expression, so the reliability of facial expression analysis can be set high, and when the second camera (320) shoots the side, it is advantageous for detecting changes in body gesture, so the reliability of motion intensity analysis can be set high.

[0193] In one embodiment, if two or more of the three emotional state determination results are determined to be the same emotional state (e.g., 'excitement'), the image preprocessing unit (121) can finally determine all three original image data taken at that time as the emotional state.

[0194] In another embodiment, the image preprocessing unit (121) can classify the same emotional state as highest reliability when detected by all three cameras and as medium reliability when detected by only two cameras, and process it differentially during the subsequent continuous frame set generation process.

[0195] In S840, the video preprocessing unit (121) can generate a highlight final video based on the generated emotion-reflecting continuous frame set. The highlight final video may be a video file edited in chronological order of game scenes in which the user's emotional response was strong. Specifically, the video preprocessing unit (121) may compress and encode the selected frames using video codecs such as MPEG-4, H.264, and H.265. Transition effects such as fade-in / fade-out and crossfade may be applied to ensure a natural transition between frames during the editing process. In one embodiment, the total length of the highlight final video may be determined within the range of 5-15% of the original video data and may be adjustable according to the user's settings.

[0196] In another embodiment, the playback speed of each highlight section can be applied differently according to the emotional weight. For example, sections with an emotional weight of 3.0 or higher can be set to play in slow motion (0.5x speed), sections between 1.5 and 3.0 at normal speed, and sections less than 1.5 at fast speed (1.5x speed).

[0197] In another embodiment, user emotional state information is included as metadata in the final highlight video so that it can be used as training data when generating personalized highlights in the future.

[0198] In S850, the highlight scene generating device (100) can transmit the generated highlight final image to the user terminal (200).

[0199] The transmitted highlight final video can be provided to the user through a touch display (250). Specifically, the highlight scene generating device (100) can transmit the highlight final video using wireless communication protocols such as Wi-Fi, 4G / 5G, etc. Before transmission, the resolution and bitrate of the video can be adjusted by taking into account network bandwidth.

[0200] In one embodiment, the personalization unit (170) of the highlight scene generation device (100) may generate and transmit a personalized emotion analysis report along with the final highlight video. The personalized emotion analysis report may include data that comprehensively analyzes the user's game viewing patterns, emotional response characteristics, and preferred highlight types.

[0201] Specifically, the personalization unit (170) can refer to the history data of past emotional weights for each match that is stored in advance in the user DB (112). The history data of past emotional weights for each match is a set of emotional weight values ​​measured by time period in each match previously watched by the user, and can be stored in the form of "match ID, time (seconds), emotional weight value". The personalization unit (170) can analyze this data to calculate statistical values ​​of the user's average emotional weight, frequency of achieving the highest emotional weight, and duration of emotion, and include these in a report in the form of "average excitement level: 2.34, high emotional interval ratio: 18.7%, average duration: 8.2 seconds".

[0202] Additionally, the personalization unit (170) may refer to preferred highlight length data stored in the user DB (112). The preferred highlight length data may consist of video length information based on which action the user took regarding previously generated highlight videos: "watch completed," "stopped midway," or "re-watched." The personalization unit (170) may calculate the average length of highlight videos with a viewing completion rate of 80% or more and reflect it in a report in the form of "Preferred highlight length: 4 minutes 12 seconds."

[0203] The personalization unit (170) may additionally refer to reaction pattern data by game type stored in the user DB (112). The reaction pattern data by game type may be data in which the average value of the user's emotional weight is recorded for detailed game situations such as "soccer-goal situation," "basketball-3-point shot," "baseball-home run." Based on this data, the personalization unit (170) may identify the top 3 game situations in which the user shows the highest emotional response and include them in a report in the form of "Situations of highest interest: 1) soccer goal scene (average 3.2), 2) comeback situation (average 2.9), 3) penalty shootout (average 2.7)."

[0204] In another embodiment, the personalization unit (170) can automatically adjust personalized parameters to be applied when generating highlights in the future based on the analysis results. The personalization unit (170) can optimize the values ​​of weighting coefficients w_1, w_2, and w_3 used when calculating emotional weights for each individual according to the user's emotional response pattern. For example, for a user sensitive to changes in heart rate, the value of w_1 can be increased to 0.7, and for a user with active motion response, the value of w_3 can be increased to 0.2, thereby enabling the calculation of emotional weights that reflect individual characteristics. The personalization unit (170) can include the adjusted parameter information in a report in the form of "Personal Optimization Coefficients: w_1=0.7, w_2=0.2, w_3=0.1".

[0205] As described above, the method according to one embodiment of the present invention can automatically generate personalized highlight videos by recognizing an emotional state based on a user's real-time biosignal, thereby significantly improving user satisfaction compared to the existing uniform highlight production method.

[0206] In particular, the accuracy and reliability of emotional state recognition can be significantly improved through a dual verification method that combines biosignal data collected from multiple sensors, such as heart rate sensors and skin conductivity sensors, with multi-camera image analysis.

[0207] In addition, through the analysis of past viewing history via the personalization unit and machine learning-based personalized parameter optimization, a differentiated highlight video service reflecting individual user preferences can be provided, and the quality of the sports viewing experience can be enhanced by offering insightful feedback on users' game viewing patterns through the generation of real-time sentiment analysis reports.

[0208] The embodiments described above may be implemented as hardware components, software components, and / or combinations of hardware and software components. For example, the devices, methods, and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, although the processing unit is described as being used as a single unit, those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.

[0209] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0210] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0211] Although the embodiments have been described above with reference to the limited drawings, those skilled in the art can apply various technical modifications and variations based on the above. For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0212] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

Claim 1 A step of collecting at least one sensing data including time information from a sensor device and a user’s biosignal at a time according to the time information by an electronic device, and recognizing the user’s emotional state for each time by comparing the sensing data with a preset threshold; a step of calculating an emotional weight for each of the recognized emotional states for each time by the electronic device; a step of dividing original video data into a plurality of time intervals by the electronic device and generating an emotionally reflective continuous frame set by mapping the emotional weight corresponding to each time interval by the electronic device; a step of generating a highlight final video based on the emotionally reflective continuous frame set by the electronic device; and a step of transmitting the highlight final video to a user terminal by the electronic device. A method comprising: a step of generating an emotion-reflecting continuous frame set by mapping the emotion weights, wherein the step of selecting frames in time intervals where the emotion weight is greater than or equal to a preset threshold as highlight candidate frames; and a step of sorting time intervals in order of highest emotion weight and selecting time intervals with a preset upper ratio among the sorted time intervals as highlight frames to generate the emotion-reflecting continuous frame set. Claim 2 In claim 1, the sensor device comprises at least one of a heart rate sensor, a skin conductivity sensor, an accelerometer, and a gyroscope, and the sensing data comprises at least one of heart rate sensing data, skin conductivity sensing data, and motion sensing data; the step of recognizing the emotional state by the electronic device further comprises the step of detecting by the electronic device that the heart rate sensing data has changed by more than a preset ratio relative to the user's normal heart rate, and when the electronic device detects that the skin conductivity sensing data is greater than a preset threshold, determining the user's emotional state as an excited state by the electronic device. Claim 3 A method according to claim 2, further comprising the step of receiving original image data from a shooting system including a plurality of cameras by means of the electronic device; wherein the shooting system includes a first camera, a second camera, and a third camera installed to photograph the user from different directions, and the step of recognizing the emotional state further comprises: the step of generating a first emotional state determination result for the first original image data generated by the first camera, a second emotional state determination result for the second original image data generated by the second camera, and a third emotional state determination result for the third original image data generated by the third camera; and the step of finally determining the entire original image data at the corresponding time as the specific emotional state when two or more of the first emotional state determination results to the third emotional state determination results are determined as a specific emotional state. Claim 4 A method according to claim 1, further comprising: a step of generating a personalized emotion analysis report by analyzing at least one of the past emotion weight history data, preferred highlight length data, and reaction pattern data by game type stored in a user database by the electronic device; and a step of transmitting the personalized emotion analysis report together with the final highlight video to the user terminal by the electronic device. Claim 5 delete

Images