Method of operating a server providing a platform service based on sports videos

Abstract

An operation method of a server providing a sports video-based platform service includes the steps of tracking a ball using pre-processing leaving a dynamic pixel in a sports video of a ball game, determining a non-identified player related to a score-related scene of the sports video using a tracking result of the ball, tracking the non-identified player until an adjacent frame in which the non-identified player can be identified, thereby identifying the non-identified player, and generating time intervals of the sports video and identification information of the non-identified player corresponding to the score-related scene.

Description

TECHNICAL FIELD

[0001] The following embodiments relate to an operation method of a server providing a platform service based on a sports video. BACKGROUND

[0002] In order to analyze a sports video, human resources of a considerable level of professionals are required. Although several schemes of analyzing a sports video have been proposed, these schemes still require a dedicated shooting device or a specialized team. Accordingly, a technology of efficiently analyzing a sports video with resources is required. SUMMARY

[0003] TECHNICAL PROBLEM

[0004] The embodiments provide a technology of significantly reducing human resources required for analyzing a sports video by using an artificial intelligence technology. Also, the embodiments provide a technology of finely searching a sports video by databaseing an analysis result of the sports video in linkage with a corresponding scene.

[0005] TECHNICAL SOLUTION

[0006] An operation method of a video analysis server according to an aspect includes the steps of receiving an analysis request signal including a link of a sports video of a ball game; performing a pre-processing of filtering out static pixels to leave dynamic pixels included in a plurality of frames of the sports video; tracking a ball of the sports video based on the pre-processed video; detecting a score-related scene of the sports video from the pre-processed video; in response to the detection of the score-related scene, determining a non-identified player related to the score-related scene using a tracking result of the ball; tracking the non-identified player until an adjacent frame in which the non-identified player can be identified, thereby identifying the non-identified player; and outputting a time interval of the sports video and identification information of the non-identified player corresponding to the score-related scene.

[0007] The step of tracking the ball can include the step of detecting a ball based on dynamic pixels of a corresponding frame corresponding to each of the frames.

[0008] The step of detecting the score-related scene can include the steps of detecting an edge based on dynamic pixels of a corresponding frame corresponding to each of the frames; and determining a frame adjacent to a frame in which the edge is detected as the score-related scene.

[0009] The step of determining non-identified players related to the score-related scene can include the steps of detecting dynamic pixels related to a player who attempts to score using tracking results of the ball in frames included in the score-related scene, and determining non-identified players related to the score-related scene by performing instance segmentation on the frames in which the dynamic pixels related to the player who attempts to score are detected.

[0010] The step of identifying the non-identified player can include the steps of extracting a feature from the determined non-identified player, comparing the extracted feature with a feature of a registered player, determining whether the non-identified player can be identified based on a result of the comparison, and tracking the non-identified player by performing instance segmentation on adjacent frames according to a determination that the non-identified player cannot be identified.

[0011] The step of performing the pre-processing can include the steps of filtering out static pixels based on a change in pixel values between adjacent frames of a predetermined range in the case where the sports video is a video captured with a fixed viewpoint, and filtering out static pixels based on a statistical value of optical flow of pixels within a frame in the case where the sports video is a video captured with a moving viewpoint.

[0012] An operation method of a server for providing a platform service based on a sports video according to an aspect includes the steps of transmitting a signal requesting analysis of a sports video to a video analysis module based on a link of the sports video, storing player-specific clusters received from the video analysis module in a database, providing information for extracting player-specific video clips from the sports video to a user terminal based on the database, receiving an input for identifying a non-identified player of at least one cluster from the user terminal, wherein the user terminal receives the video clips from a streaming server by the player, and updating identification information of at least one corresponding cluster in the database based on the input.

[0013] The operation method of the server for providing a platform service based on a sports video can further include the steps of providing statistics indicating a degree of contribution of players to the user terminal, receiving an input of selecting a detailed record included in the statistics from the user terminal, obtaining at least one sub-cluster related to the selected detailed record based on the database, and providing information for extracting a video clip from the sports video to the user terminal based on the at least one sub-cluster.

[0014] The operation method of the server providing the platform service based on the sports video can further include the steps of receiving a search query including a search object player and a search object scene from the user terminal, searching for a sub-cluster corresponding to the search query from the database, and providing information for extracting a video clip from the sports video to the user terminal based on the searched sub-cluster.

[0015] The operation method of the server providing the platform service based on the sports video can further include at least one of the steps of determining a billing level based on reliability of the cluster, and determining a compensation level based on feedback input modifying the cluster.

[0016] The operation method of the server providing the platform service based on the sports video can further include the steps of receiving feedback input indicating that a player included in at least one interval in the at least one cluster does not belong to a corresponding cluster from the user, and excluding the corresponding interval from the corresponding cluster based on the feedback input.

[0017] The operation method of the server providing the platform service based on the sports video can further include the steps of receiving feedback input indicating that a player included in at least one interval in the at least one cluster belongs to another cluster from the user, and excluding the corresponding interval from the corresponding cluster and including the interval in the other cluster based on the feedback input.

[0018] The operation method of the server providing the platform service based on the sports video can further include the steps of generating training data depending on the updated database, and learning a specialized model estimating at least one of detection information, recognition information, and motion type information of a player based on the training data. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 FIG. 1 is a diagram illustrating a system for providing a platform service based on sports video according to an embodiment.

[0020] Figure 2 FIG. 2 is an operational flowchart illustrating an operation method of a service server for providing a platform service based on sports video according to an embodiment.

[0021] Figure 3 FIG. 3 is a diagram illustrating cluster operation according to an embodiment.

[0022] Figure 4 FIG. 4 is a diagram illustrating a platform service based on sports video according to an embodiment.

[0023] Figure 5 FIG. 5 is a diagram illustrating a platform service based on sports video according to an embodiment.

[0024] Figure 6 is a diagram illustrating distribution of feature vectors based on motion types according to an embodiment.

[0025] Figure 7 is a diagram illustrating clustering operation based on motion types according to an embodiment.

[0026] Figure 8 is a diagram illustrating a general model and a specialized model according to an embodiment.

[0027] Figure 9 is a diagram illustrating video clips provided in association with statistics representing contribution of players in a basketball game according to an embodiment.

[0028] Figure 10 is a diagram illustrating a function reflecting feedback of a user on a video clip according to an embodiment.

[0029] Figure 11 is a diagram illustrating a search function according to an embodiment.

[0030] Figure 12 is a diagram illustrating operation of generating a tracking cluster according to an embodiment.

[0031] Figure 13 is a diagram illustrating operation of matching tracking clusters according to an embodiment.

[0032] Figure 14 is a diagram illustrating operation of sensing a score event according to an embodiment. DETAILED DESCRIPTION

[0033] The specific structural or functional explanations disclosed in the present specification are merely exemplified for the purpose of illustrating embodiments based on technical concepts, and the embodiments can be implemented in various other forms, and are not limited to the embodiments described in the present specification.

[0034] The terms of "first" or "second" or the like can be used to explain various constituent elements, but such terms should be understood as merely for the purpose of distinguishing one constituent element from other constituent elements. For example, a first constituent element can be named as a second constituent element, and similarly, a second constituent element can be named as a first constituent element.

[0035] It should be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements can also be present. In contrast, when an element is referred to as being "directly connected" or "directly coupled" to another element, there are no intervening elements present. Other expressions that describe the relationship between elements should be interpreted in the same manner unless otherwise stated.

[0036] The singular form "a," "an," and "the" include plural references unless the context clearly dictates otherwise. In this specification, the terms "comprises", "comprising", "includes", "including", "has", "having" and the like are to be understood to be open-ended terms that specify the presence of stated features, integers, steps, operations, elements, components, or the like, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or the like.

[0037] Unless otherwise defined, all terms used herein, including technical terms and scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this technology belongs. Terms that are defined in generally used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the relevant technology, and should not be interpreted in an idealized or overly formal sense unless expressly so defined in the specification.

[0038] Embodiments can be provided as services in various forms of products such as personal computers, laptop computers, tablet computers, smart phones, televisions, smart home appliances, smart cars, self-service terminals, wearable devices, etc.

[0039] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Like reference numerals denote like elements throughout the drawings.

[0040] Figure 1 is a diagram illustrating a system for providing a platform service based on a sports video according to an embodiment. Referring to Figure 1 , the system according to an embodiment includes a service server, and according to the design of the system, at least one of a streaming server, a storage server, a social network service (SNS) server, and an instant messenger service (IMS) server can also be included. The service server can include a front-end server that communicates with an application program provided in a user terminal and a video analysis server that analyzes a sports video. According to the design of the system, the front-end server and the video analysis server can be implemented in the form of modules within the same server, or can be implemented as independent servers different from each other and implemented in a form that communicates with each other through a network.

[0041] The service server receives a sports video. The sports video is an image that captures a sports game, including a plurality of frames. The sports game can be one of a real-time ball game such as basketball, soccer, volleyball, handball, hockey, ice hockey, tennis, etc., a round ball game such as American football, rugby, baseball, croquet, golf, etc., and a non-ball game such as diving, swimming, skiing, snowboarding, etc. Hereinafter, a basketball game will be described as an example, but embodiments can be applied to other sports games other than a basketball game in substantially the same manner.

[0042] The sports video can be a video that has been uploaded to a streaming server or a storage server. In this case, the service server can receive information (e.g., a uniform resource locator (URL), etc.) for accessing the sports video through the streaming server or the storage server. Alternatively, the service server can receive a request to upload the sports video. The service server can upload the sports video requested to be uploaded to the streaming server or the storage server. The streaming server or the storage server can be operated by the same subject as the service server, or can be operated by a different subject according to embodiments.

[0043] The sports video includes a plurality of non-identified players. The non-identified player is a player whose identity is not identified, for example, can include a player whose identification information is not set. The plurality of non-identified players included in the sports video perform various motions for a sports game, and the sports video is analyzed, so that the ability of a player on an individual level or the ability of a team on a team level participating in the corresponding sports game can be evaluated. However, in order to analyze the sports video, one or more experts who are skilled in the rules of the corresponding sports game need to perform a record for each player for the entire time of the corresponding sports game.

[0044] Embodiments described below provide a technology that significantly reduces human resources required to analyze a sports video using an artificial intelligence technology.

[0045] According to an embodiment, the video analysis server can receive an analysis request signal including a link of a sports video of a ball game from the front-end server. The video analysis server can perform preprocessing by filtering out static pixels by preserving dynamic pixels from a plurality of frames included in the sports video. For example, in the case of a sports video that is captured at a fixed angle (a video captured at a fixed camera angle), static pixels can be filtered out based on a change in pixel values between adjacent frames of a predetermined range. In the case where a change in pixel values at the same position between adjacent frames exceeds a predetermined threshold range, it can be classified as a dynamic pixel.

[0046] If the sports video is a video captured with a moving viewpoint (a video captured with a camera angle moving following a ball), static pixels can be filtered out based on statistical values of optical flow of pixels within a frame. Pixels within a frame can have a common component of optical flow due to the movement of the camera angle. If the optical flow including the common component of pixels is removed, a component of the optical flow of pixels corresponding to an actually moving object (a moving player, a moving ball, a vibrating rim, etc.) can be obtained. After the common component of the optical flow is removed, pixels having optical flow exceeding a predetermined threshold value size can be classified as dynamic pixels.

[0047] The video analysis server can track a ball of the sports video based on the preprocessed video. For example, the video analysis server can detect a ball based on dynamic pixels of each frame of the sports video corresponding to the respective frame, thereby tracking the ball of the sports video. Since the ball included in the sports video is captured to be relatively small compared to other objects (players or a goal, etc.) in the image, it is difficult to track the ball by an existing object tracking model. Embodiments can significantly improve the performance of ball tracking by utilizing an artificial neural network trained to track the ball using dynamic pixels obtained by preprocessing.

[0048] The video analysis server can detect a score-related scene of the sports video from the preprocessed video. For example, the video analysis server can detect a rim based on dynamic pixels of each frame corresponding to the respective frame, and determine a frame adjacent to the frame in which the rim is detected as a score-related scene, thereby being able to detect the score-related scene of the sports video.

[0049] The rim denotes a predetermined shaped structure used to determine whether a ball passes to score in a ball game sports match, and hereinafter, a net attached to the rim can also be understood to be included in the rim. The rim is classified as a static pixel as it does not move in general, but can be classified as a dynamic pixel in a scene related to scoring, such as a case where the actual rim moves as a ball hits the rim or a case where a moving ball passes through the rim. Accordingly, if the rim is detected in the preprocessed image, it can be determined as a scene related to scoring.

[0050] The video analysis server, in response to the detection of the score-related scene, can determine a non-identified player related to the score-related scene using a tracking result of the ball. For example, the video analysis server can detect dynamic pixels related to a player attempting to score from among frames included in the score-related scene using the tracking result of the ball, and perform instance segmentation on frames in which the dynamic pixels related to the player attempting to score are detected, thereby being able to determine the non-identified player related to the score-related scene.

[0051] Since the video analysis server tracks the ball in the pre-processed image, if an edge is detected in the pre-processed image, a moving track of the ball included in a neighboring frame before the frame in which the edge is detected can be obtained starting from the frame in which the edge is detected. The video analysis server can filter dynamic pixels that are within a predetermined reference distance from the ball from dynamic pixels of the players included in the neighboring frame while searching for the moving track of the ball. The filtered dynamic pixels can include a non-identified player who attempts to score.

[0052] The video analysis server, when filtering the dynamic pixels that are within the predetermined reference distance from the ball, performs instance segmentation on the corresponding frame, so that a mask of the non-identified player who attempts to score can be acquired. At this time, an image of an original frame that is not pre-processed can be input as an input of a neural network model that performs instance segmentation.

[0053] The video analysis server tracks the non-identified player until a neighboring frame in which the non-identified player can be identified, so that the non-identified player can be identified. For example, the video analysis server can extract a feature from the determined non-identified player. The video analysis server can acquire pixel values corresponding to the non-identified player using the mask of the non-identified player, and extract a feature based on the acquired pixel values.

[0054] The video analysis server can compare the extracted feature with features of registered players, so as to determine whether the non-identified player can be identified. The video analysis server can receive registration information of players participating in a sports game for which analysis is requested from a service server. The registration information can include a list of players and photos of the players. The video analysis server can extract a feature from the photos of the registered players, and compare the feature with the feature of the non-identified player. The video analysis server can match the non-identified player with one of the registered players according to the comparison result.

[0055] In a case where the comparison result determines that the non-identified player cannot be identified, the video analysis server can track the non-identified player by performing instance segmentation on a neighboring frame (a past or future neighboring frame). Due to various factors such as the non-identified player being blocked by other players, the frame in which the non-identified player attempts to shoot can not include sufficient image information for identifying the non-identified player. The video analysis server can track the non-identified player until a neighboring frame including image information sufficient to identify the non-identified player. The video analysis server, after tracking the non-identified player in the neighboring frame, can determine whether the non-identified player can be identified based on image information of the non-identified player tracked in the corresponding neighboring frame. The video analysis server can repeatedly perform the tracking operation in the neighboring frame and the identification operation until the non-identified player can be identified.

[0056] The video analysis server can determine whether a score is successful or unsuccessful in a score-related scene. For example, the video analysis server can utilize a neural network model trained to take as input a moving trajectory of a ball based on positions of edges included in a plurality of frames in the score-related scene to determine whether the score is successful.

[0057] The video analysis server can output a time interval of the sports video, identification information of the non-identified player, and / or whether the score is successful or unsuccessful, corresponding to each score-related scene, in response to the analysis request signal of the front-end server. The front-end server can establish a database based on the response signal.

[0058] According to an embodiment, the service server can distinguish the non-identified players included in the sports video from each other using a non-guided learning scheme. The service server can distinguish a first non-identified player and a second non-identified player from each other even when the non-identified players cannot be identified as who they are. For example, within the sports video, the appearance characteristics of the first non-identified player and the appearance characteristics of the second non-identified player can be distinguished from each other. The appearance characteristics are appearance characteristics of the non-identified players, and for example, can include not only appearance characteristics of the non-identified players themselves such as body shape, height, skin color, hairstyle, face, etc., but also appearance characteristics of sports clothes, back numbers, shoes, protective tapes, accessories, etc. Also, the movement characteristics of the first non-identified player and the movement characteristics of the second non-identified player can be distinguished from each other. The movement characteristics are movement characteristics of the non-identified players, and for example, in a basketball game, can include characteristics of postures or movements inherent to the non-identified players in various movements such as a jump shot movement, a stand shot movement, a layup movement, a dunk movement, a dribble movement, a pass movement, a pick movement, a rebound movement, a block movement, a defense movement, etc. The service server can distinguish the non-identified players from each other using the appearance characteristics and / or the movement characteristics.

[0059] The service server can generate video clips for the non-identified players distinguished from each other. For example, the service server can generate a first video clip that selectively includes only images of the first non-identified player in the entire game image, and generate a second video clip that selectively includes only images of the second non-identified player. The service server can provide the video clips for the non-identified players to the user, and thus can receive information identifying the non-identified players of the video clips from the user. The user can be one of the non-identified players in the corresponding sports video, and according to circumstances, the user can be a person who can identify the non-identified players in the corresponding sports video although not being the non-identified players in the sports video.

[0060] The user can input information identifying non-identified players corresponding to at least one of the plurality of video clips. For example, the user can input information confirming himself in his own video clip, or information identifying a corresponding non-identified player in a video clip in which the user can identify the non-identified player.

[0061] The service server can analyze the sports video based on the identification information input per video clip. Since most sports games are conducted based on predetermined rules, the types of motions allowed can be determined in advance according to the rules of the sports game. Accordingly, one type of motion and another type of motion can be distinguished from each other. For example, in a basketball game, a shooting motion and a dribbling motion can be distinguished from each other. Further, in the shooting motion, a jump shooting motion and a dunking motion can be distinguished from each other. Accordingly, the service server can extract a feature scene or frame from the sports video, and classify the type of motion of the extracted scene. It will be described in detail below that the service server can also extract a partial region instead of the entire scene or frame, and classify the type of motion of the extracted region.

[0062] Accordingly, the embodiments can provide a technology of analyzing various moments within a sports game only by receiving identification information per video clip from a user. Further, as image data of the same player is accumulated, a specialized model automatically identifying the corresponding player can be learned. The specialized model can also be trained to not only automatically identify the corresponding player but also more accurately classify the motions of the corresponding player.

[0063] Also, the service server can label the identification information and the motion type information to the time detected in the sports video, thereby establishing a database. The user can search for a desired image in the database through various queries. For example, the user can input i) a query specifying a player to ii) search for a desired scene. The service server can search from the database for which time point (frame) in which sports video the scene corresponding to the query corresponds to, and can generate a video clip based on the search result and provide it to the user. According to the embodiments, a desired scene can be searched for by specifying a plurality of players (for example, a scene in which player B scores an assist from player A, or a scene in which player C is blocked by player D, etc.).

[0064] As will be described in detail below, the service server can generate statistics representing the contribution of players in a sports game. For example, the service server can generate a box score that numerically represents the outcome of a basketball game. The service server can provide a service linked to a scene of a detail record for each player-specific scene in the box score. For example, when a "steal" detail record of a specific player is selected in the box score, a video clip including a scene of the corresponding player's "steal" can be provided. According to an embodiment, the front-end server searches for a steal-related cluster of the corresponding game in response to a selection input for "steal" in the box score of a specific game. The front-end server can provide the user terminal with information (e.g., an image URL and at least one time interval) enabling the relevant video clip to be received through streaming based on the searched cluster.

[0065] The service server can share the video clip provided to the user with a social network server or an instant messenger server. The service server can provide a service to the user through a web interface, or can provide a service to the user through an application interface.

[0066] Figure 2 FIG. 1 is an operational flowchart illustrating an operational method of a service server providing a platform service based on sports videos according to an embodiment. As described above with reference to FIG. 1, the service server can be implemented by a single server of a front-end module and a video analysis module, or can be implemented by independent servers of a front-end server and a video analysis server. Hereinafter, an embodiment in which the service server is implemented by a single server will be described for convenience of explanation. Figure 1

[0067] Referring to FIG. 2, Figure 2 , the front-end module of the service server transmits a signal requesting analysis of sports videos to the video analysis module based on a link of the sports videos (210). The front-end module of the service server stores player-specific clusters received from the video analysis module in a database (220). The player-specific clusters can include interval information about scenes that can be calculated as player-specific contribution within a game (e.g., a score-related scene, a defense-related scene, a mistake or foul-related scene, etc.).

[0068] The front-end module of the service server provides a user terminal with information for extracting player-specific video clips from sports videos based on the database (230). The information provided to the user terminal can include a link of the videos and a time interval for a main scene of the player-specific scene. The user terminal can request a streaming server to selectively stream only a time interval corresponding to the main scene based on the received information.

[0069] ​The front-end module of the service server receives input identifying non-identified players of at least one cluster from the user terminal (240). The front-end module of the service server updates identification information of at least one corresponding cluster in the database based on the user input (250).

[0070] The service server receives input identifying non-identified players of at least one cluster from the user. Since the user receives video clips by non-identified players, the input identifying non-identified players can be made by video clips. The video clips are extracted or generated in a cluster unit, and the received input corresponds to information identifying non-identified players corresponding to the corresponding cluster.

[0071] The service server sets identification information of at least one corresponding cluster based on the input of the user. According to an embodiment, the cluster can include fields of {access information of sports video, identification information, index of region}. In this case, the service server can set identification information of the cluster based on the input of the user. The identification information can correspond to account information of a platform service based on sports video. Alternatively, the identification information can also correspond to account information of a social network service linked with the platform service based on sports video or account information of an instant message service linked with the platform service based on sports video. The identification information can include information of a predetermined template for identifying non-identified players (e.g., family name, given name, nickname, belonging team, back number, gender, age, height, weight, position, etc.). According to an embodiment, the identification information can also be stored in a region. In this case, the cluster can include a field of {index of region}, and each region can include a field of {identification information}.

[0072] Alternatively, the service server can receive feedback input that a player included in at least one interval of at least one cluster does not belong to the corresponding cluster from the user terminal, and exclude the corresponding interval from the corresponding cluster.

[0073] Alternatively, the service server can receive feedback input that a player included in at least one interval of at least one cluster belongs to another cluster from the user terminal, and can exclude the corresponding interval from the corresponding cluster and include it in another cluster.

[0074] Also, the front-end module of the service server can provide a platform service based on sports video (260). For example, the service server can provide a platform service based on sports video based on a cluster (or region) in which identification information is set. The service server can provide various services such as automatically analyzing a sports game, or providing statistical information indexing contents of a sports game, or providing a video clip linked with the statistical information, or providing a detail search function of a sports video, or providing a video clip corresponding to a search result, etc. Also, the service server can further provide a function of sharing a video clip to a social network service or an instant messenger service.

[0075] The service server can provide a statistic indicating a degree of contribution of the player to the user terminal, and receive an input of selecting a detailed record included in the statistic from the user terminal. The service server can acquire at least one sub-cluster related to the selected detailed record based on the database, and provide information for extracting a video clip from the sports video to the user terminal based on the at least one sub-cluster.

[0076] Alternatively, the service server can receive a search query including a search target player and a search target scene from the user terminal, and search for a sub-cluster corresponding to the search query from the database. The service server can provide information for extracting a video clip from the sports video to the user terminal based on the searched sub-cluster.

[0077] Alternatively, the service server can determine a billing level based on reliability of the cluster, or determine a reward level based on a feedback input modifying the cluster.

[0078] Furthermore, the service server can generate training data depending on an updated database to learn a specialized model estimating at least one of detection information, recognition information, and motion type information of the player.

[0079] The service server can learn a specialized model for the recognized player based on a cluster (or region) provided with the recognition information. Alternatively, the service server can learn a specialized model for a team to which a plurality of recognized players belong. The specialized model can be learned through a detection module detecting a region of a player in a sports video, a classification module classifying the region, a recognition module recognizing the region, or a composite module for a variety of combinations of the above functions, etc.

[0080] The service server can provide a higher quality service for a player inputting recognition information by using the specialized model. As an example, as the specialized model is learned by accumulating data provided with the recognition information, recognition information of a corresponding player in a new sports video can be automatically set. Also, a game content of the corresponding player can be more accurately analyzed through the specialized model.

[0081] Although not illustrated, according to an embodiment, the service server detects regions corresponding to the unidentified players from the sports video in which the unidentified players are photographed. The "detection" can be an operation of determining a partial region corresponding to a detection object within an image. The sports video includes a plurality of frames, and the service server can detect a region occupied by an individual unidentified player in an individual frame. The service server can detect the region corresponding to the unidentified player using a detector for detecting players who play a corresponding sports game in the sports video. The region corresponding to the unidentified player can have a predetermined shape, for example, can have a window shape of a quadrangle. The service server can detect a window corresponding to the unidentified player in an individual frame of the sports video. As an example, the service server can acquire a frame index indicating the frame, (x, y)-coordinates as information indicating a position of the detected window, and (width, height) as information indicating a size of the detected window. In this case, an individual region can be defined as {frame_index, x-coordinate, y-coordinate, width, height}. The service server can directly drive a detection module, or can request detection to another server that drives the detection module in cooperation with the service server.

[0082] The service server clusters the detected regions to distinguish the unidentified players from each other, and generates clusters specific to the unidentified players. The service server can cluster the regions based on a non-supervised learning method. For example, the service server can cluster the regions using a cluster analysis technique represented by a K-means clustering technique. The generated clusters can include information indicating regions belonging to a corresponding cluster (e.g., indices of the regions, etc.).

[0083] The K parameter for clustering can be set by receiving the number of players participating in the game from the user. Alternatively, the number of players who can participate in the sports game at the same time (e.g., 5 per team, 10 in total in the case of basketball) can be set as an initial value of the K parameter, and clustering can be performed iteratively while adjusting the value of the K parameter, so that the number of players participating in the sports game can also be estimated in the corresponding sports video. The number of players participating in the sports game can be greater than the number of players who can participate at the same time, considering substitutes. According to an embodiment, the service server can use a hierarchical clustering technique as a cluster analysis technique, as will be described in detail below.

[0084] The service server can extract features of the regions for clustering. According to an embodiment, the service server can extract appearance features from individual regions. The appearance features can be defined in various ways. For example, the appearance features are multi-dimensional vectors, and can include information related to body shape, height, skin color, hairstyle, face, sportswear, back number, shoes, protective gear, and / or accessories, etc. in each dimension. Alternatively, the service server can extract motion features from sequences of regions. The motion features can be defined in various ways. As an example, the motion features can be multi-dimensional vectors, can include information extracted from the posture of the non-identified player or information extracted from the motion of the non-identified player, etc.

[0085] The service server can cluster the regions based on the appearance features or the motion features. For example, the service server can classify regions having similar appearance features to each other as the same cluster, and classify regions having different appearance features to each other as different clusters. Alternatively, the service server can classify regions having similar motion features to each other as the same cluster, and classify regions having different motion features to each other as different clusters. According to an embodiment, the service server can cluster the regions based on a combination of the appearance features and the motion features. For example, referring to Figure 3 , the service server can classify regions having similar combinations of the appearance features and the motion features to each other as the same cluster, and classify regions having different combinations of the appearance features and the motion features to each other as different clusters. For ease of explanation, Figure 3 only three clusters are shown in, but as described above, a number of clusters corresponding to the number of non-identified players included in the sports video can be generated. Also, in Figure 3 , the appearance features and the motion features are shown in single dimensions respectively, but as described above, the appearance features or the motion features can include multi-dimensional information.

[0086] According to an embodiment, the appearance features can be extracted from regions of a single frame, and the motion features can be extracted from sequences of regions of multiple frames. In this case, a synchronization operation between the regions and the sequences of regions can be required. As an example, the appearance features of a region detected in a certain frame can be statistically processed (averaged, etc.) together with the appearance features of regions detected in previous and subsequent frames. The range of the previous and subsequent frames can correspond to the range of frames of the sequence of regions used for extracting the motion features.

[0087] The service server extracts video clips specific to the non-identified players from the sports video based on the generated clusters (230). The service server can acquire the frame indexes of the regions included in the corresponding cluster, the positions and sizes of the regions within each frame, etc. corresponding to each cluster. The service server can generate a video clip for the corresponding cluster by extracting frames having the frame indexes corresponding to the individual cluster from the sports video.

[0088] According to an embodiment, the service server can impart a visual effect to the video clip. The service server can crop a frame extracted from the sports video based on the position and size of the region within the frame. Alternatively, the service server can impart a visual effect that emphasizes the corresponding region in the corresponding frame based on the position and size of the region within the frame. Alternatively, the service server can add information related to the cluster or information related to the region in the form of a caption or the like.

[0089] The service server provides the extracted video clip to the user by non-identified player. The service server can provide the video clip to the user through a web interface and / or an application interface. According to an embodiment, the service server can provide the video clip to the user through a social network service and / or an instant messenger service.

[0090] Figure 4 is a diagram illustrating a platform service based on a sports video according to an embodiment. Referring to Figure 4 , the sports video can include a plurality of frames..., k, k+1,..., l, l+1,..., m, m+1,.... The service server can detect regions 401 to 416 of a plurality of non-identified players from the plurality of frames.

[0091] The service server can cluster the regions 401 to 416 using an unsupervised learning technique. For example, the service server can classify the region 401, the region 403, the region 413, and the region 416 as a first cluster C1, classify the region 405, the region 408, the region 411, and the region 414 as a second cluster C2, classify the region 402, the region 404, the region 406, and the region 409 as a third cluster C3, and classify the region 407, the region 410, the region 412, and the region 415 as a fourth cluster C4.

[0092] The service server can extract a video clip for a non-identified player from the sports video based on the first cluster to the fourth cluster. For example, the service server can extract frames k, k+1, m, m+1 corresponding to the first cluster, and generate a video clip for a first non-identified player. Also, the service server can extract frames l, l+1, m, m+1 corresponding to the second cluster, and generate a video clip for a second non-identified player. The service server can extract frames k, k+1, l, l+1 corresponding to the third cluster, and generate a video clip for a third non-identified player. The service server can extract frames l, l+1, m, m+1 corresponding to the fourth cluster, and generate a video clip for a fourth non-identified player.

[0093] The service server provides the user with the video clip generated by the non-identified player, so that the identification information can be received by the video clip. For example, the service server can receive the identification information of the first player of team A corresponding to the first cluster, the identification information of the second player of team A corresponding to the second cluster, the identification information of the first player of team B corresponding to the third cluster, and the identification information of the second player of team B corresponding to the fourth cluster.

[0094] The service server can collect the identification information received from a plurality of users. For example, the server can receive an input of identifying the video clip in the first cluster from a first user, and receive an input of identifying the video clip of the second cluster from a second user. Alternatively, the service server can receive information of identifying the video clip of the same cluster from a plurality of users, and adopt the identification information having the highest reliability. For example, the service server can receive identification information of the video clip of the same cluster from a plurality of users, which are different from each other. In this case, the service server can adopt the identification information input by the most users. Alternatively, the service server can adopt the identification information input by the user having the highest reliability. Alternatively, the service server can assign a score to the identification information input by the user based on the reliability of the user, and select the identification information having the highest score. The reliability of the user can be determined based on a history of using an existing sports video-based platform service by the corresponding user or a self-authentication level of the corresponding user, etc.

[0095] Figure 5 is a diagram illustrating a sports video-based platform service according to an embodiment. The service server can perform clustering by a motion type. To this end, the service server can classify regions of non-identified players detected from a sports video by a predetermined motion type, and cluster the motion type-specific regions. Alternatively, the service server can hierarchically cluster the regions using a hierarchical clustering technique.

[0096] The motion features by a motion type occurring in a sports game can be collectively included in the non-identified players. For example, a dunk motion of a first non-identified player and a dunk motion of a second non-identified player can collectively include a motion feature for a dunk. Also, a jump shot motion of the first non-identified player and a jump shot motion of the second non-identified player can collectively include a motion feature of a jump shot motion. Referring to Figure 6 , a dunk motion of the first non-identified player can be expressed by a motion vector 610, a jump shot motion of the first non-identified player can be expressed by a motion vector 620, a dunk motion of the second non-identified player can be expressed by a motion vector 630, and a jump shot motion of the second non-identified player can be expressed by a motion vector 640.

[0097] In Figure 6In an embodiment, the distance between the motion vector 610 and the motion vector 620 can be greater than the distance between the motion vector 610 and the motion vector 630, and the distance between the motion vector 640 and the motion vector 630 can be greater than the distance between the motion vector 640 and the motion vector 620. In this case, clustering can not be performed by non-identified players, but can be performed by motion types.

[0098] According to an embodiment, the service server can classify the regions by motion types, and then perform clustering between regions corresponding to the same motion type. In this case, the regions can be classified into a first cluster 650 corresponding to a dunking motion and a second cluster 660 corresponding to a jump shooting motion. According to an embodiment, region detection and motion type classification can also be simultaneously performed. For example, a detection module that is learned to detect non-identified players from a sports video while classifying motion types of corresponding regions can be utilized. Of course, according to an embodiment, region detection and motion type classification can also be performed by separate modules (or neural networks). After the regions are classified by motion types, the service server can cluster the motion type-specific regions so as to distinguish non-identified players from each other by motion types.

[0099] According to an embodiment, the service server can utilize a hierarchical clustering technique to first perform primary clustering and then perform secondary clustering within each cluster. In this case, the first cluster 650 and the second cluster 660 are generated by the primary clustering, and the first non-identified player and the second non-identified player are distinguished within the first cluster 650 and the second cluster 660 by the secondary clustering. K1 for the primary clustering can correspond to the number of motion types allowed in the corresponding sports game, and K2 for the secondary clustering can correspond to the number of players participating in the corresponding sports game.

[0100] Referring to Figure 7 , an embodiment in which sub-clusters are generated by clustering regions by motion types, motion features, and appearance features is illustrated. Referring again to Figure 6 , it will be explained in detail below that the sub-cluster corresponding to the first non-identified player within the first cluster 650 and the sub-cluster corresponding to the first non-identified player within the second cluster 660 can be matched to each other based on tracking information, appearance information, or a combination thereof. Of course, the sub-cluster corresponding to the second non-identified player within the first cluster 650 and the sub-cluster corresponding to the second non-identified player within the second cluster 660 can also be matched to each other based on tracking information, appearance information, or a combination thereof.

[0101] Referring to Figure 5For example, the service server can classify the regions 501 to 516 by the type of motion. For example, the region 501 and the region 511 can be classified as a ball carrying motion, the region 503, the region 510, and the region 514 can be classified as a shooting motion, the region 504 can be classified as a blocking motion, the region 506 can be classified as a passing motion, and the region 513 and the region 516 can be classified as a screening motion.

[0102] The service server can cluster the regions classified by the type of motion so as to distinguish non-identified players from each other by the type of motion. For example, the service server can cluster the regions 501 and 511 classified as the ball carrying motion as different sub-clusters DR1, DR2 from each other. The service server can cluster the regions 503, 510, and 514 classified as the shooting motion as different sub-clusters SH1, SH2, SH3 from each other. The service server can cluster the regions 513 and 516 classified as the screening motion as the same sub-cluster SC1.

[0103] The service server can match the sub-clusters of different types of motion to each other using the tracking information of the regions. For example, the service server can match DR1 and SH1 to be classified as the same cluster C1 based on the tracking information of the regions 501 and 503 being regions continuous to each other. The service server can match DR2 and SH3 to be classified as the same cluster C2 based on the tracking information of the regions 511 and 514 being regions continuous to each other.

[0104] The service server can match the sub-clusters of different types of motion to each other using the shape information of the regions. For example, the service server can classify BR1 and PA1 as the same cluster C3 based on a determination that the shape feature of the region 514 is similar to the shape feature of the region 506. The service server can classify SC1 as the cluster C1 based on a determination that the shape feature of the region 513 is similar to the shape feature of the region 502 or 503.

[0105] In the embodiment of FIG. 10, Figure 5 In the embodiment of FIG. 10,

[0106] According to an embodiment, the service server can generate a video clip using sub-clusters according to predetermined motion types. For example, the service server can generate a video clip using only sub-clusters corresponding to motion types of offensive motions. Further, the service server can generate a video clip using only region sequences corresponding to scenes of scoring success in offensive motions.

[0107] Figure 8 FIG. 1 is a diagram illustrating a general model and a specialized model according to an embodiment. Referring to FIG. 1, Figure 8 The general model can include a detector that detects non-identified players from sports videos in general and a classifier that classifies motion types of detected regions in general. The general model can include a detector or a classifier that is learned independently of data of non-identified players.

[0108] The specialized model can be specialized to a specific player, a specific group, or a specific team using a database in which identification information is set based on a platform service of sports videos, can include a detector that detects players, a classifier that classifies motion types, an identifier that identifies players, or a hybrid module in which detection functions, classification functions, and / or identification functions are combined in various ways. The general model and the specialized model can be models based on artificial neural networks. In this case, the database can include training data that depends on data of players for which identification information is set.

[0109] The specialized model can be a model newly learned for a specific group, and can be a model that is additionally learned in a manner suitable for the specific group based on the general model according to an embodiment. For example, the specialized model can be generated by additionally learning the general model so that the general model is specialized to a specific group according to nationality, age bracket, gender, etc., and exhibits higher performance.

[0110] The service server can automatically set identification information of clusters by non-identified players from newly received sports videos using the specialized model. The sports videos can be live streaming videos, in which case the service server can generate statistics representing the contribution degrees of players automatically identified from the sports videos in real time based on the specialized model. The statistics representing the contribution degrees of players can include not only scores of two teams in a basketball game in progress, but also detailed records such as the number of scoring successes, the number of scoring attempts, the team score and loss in game time, the total number of scores, the number of offensive rebounds, the number of defensive rebounds, the number of assists, the number of steals, the number of blocks, the number of blocked shots, the number of fouls, and the number of mistakes of individual players.

[0111] Figure 9 FIG. 2 is a diagram illustrating a video clip provided in association with statistics representing the contribution degrees of players in a basketball game according to an embodiment. Referring to FIG. 2, Figure 9The service server can generate a box score statistically representing the degree of contribution to the game by the player according to the identification information of the region detected from the basketball video and the motion type of the region.

[0112] The service server can determine whether to additionally analyze the sports video according to the motion type of the region. For example, the service server can additionally analyze whether the shot is successful according to the determination that the motion type is a shot motion. The service server can confirm whether the ball passes the rim after the frame of the shot motion. The service server can detect a region corresponding to the rim in the frame after the shot motion, and can determine whether the ball passes in the detected region.

[0113] The service server, while providing the box score, can provide a video clip of the scene including the selected detailed record in response to the selection of the detailed record within the box score. For example, FG within the box score can be represented by "number of successful scoring times" - "number of scoring attempt times" as "08-14". In the case where the user selects 08 corresponding to the "number of successful scoring times", the service server can generate a video clip including the scene in which the corresponding player successfully scores in the corresponding game and provide it to the user.

[0114] More specifically, the detailed record of the box score can store the corresponding region (or sub-cluster) as a data structure such as a linked list. The service server can acquire the relevant region according to the linked list, and extract a video clip from the basketball video based on the information (frame index, window position, window size, etc.) stored in each region. As described above, the service server can impart a visual effect to the video clip.

[0115] Figure 10 is a diagram illustrating a function of reflecting the user's feedback on the video clip according to an embodiment. Referring to Figure 10 The service server can receive the user's feedback on the video clip. For example, the service server can receive a feedback input from the user that the non-identified player included in at least one region in the cluster of the video clip does not belong to the corresponding cluster. The service server can exclude or remove the corresponding region from the corresponding cluster, or further exclude or remove a sub-cluster including the corresponding region from the corresponding cluster. In this case, according to an embodiment, the service server can temporarily store the excluded region or sub-cluster in an unclassified pool. The service server can provide the user with a video clip of the sub-cluster temporarily stored in the unclassified pool, and ask which cluster the corresponding sub-cluster belongs to.

[0116] Alternatively, the service server can receive a feedback input from the user that at least one region included in the cluster of the video clip belongs to another cluster. The service server can exclude or remove the corresponding region or a sub-cluster including the corresponding region from the corresponding cluster. The service server can include the excluded region or sub-cluster in another cluster designated by the user.

[0117] Although not illustrated, the service server can differently determine a charge level according to the accuracy (or reliability) of the service. For example, the service server can discount the service charge per game (e.g., $1) according to the detection accuracy, the classification accuracy, or a combination thereof. The detection accuracy and / or the classification accuracy can be referred to as the accuracy of the cluster.

[0118] Also, the service server can determine a compensation level according to the feedback input of the user. For example, the service server can accumulate points that can be used in the next service to the corresponding user according to the degree to which the accuracy (or reliability) of the cluster is improved based on the feedback input of the user or according to the degree to which the performance of the dedicated model is improved based on the feedback input of the user.

[0119] Figure 11 FIG. 1 is a diagram illustrating a search function according to an embodiment. The service server can establish a database for searching for sports videos. The service server can receive a search query from a user terminal. The search query can include a search object player and a search object scene. The service server can search for a URL of a sports video corresponding to the search query and a time interval within the corresponding sports video from the database.

[0120] The service server provides information for extracting a video clip to the streaming server, thereby enabling the video clip to be directly transmitted to the user from the streaming server. The service server can provide the search result including the URL of the sports video and the time interval to the user terminal. The user terminal can request an image of the corresponding time interval of the sports video to the streaming server or the storage server based on the search result.

[0121] Although not illustrated, according to an embodiment, the service server can extract a video clip from the streaming server or the storage server based on {access information of the sports video} of the cluster including the searched sub-cluster (or interval sequence). The service server can provide the video clip to the user. The service server can cache the generated video clip. The service server can store whether the video clip is cached in the database. When it is determined that the cached video clip can be provided as a result of the query processing, the service server can omit the operation of extracting (or generating) the video clip and directly provide the cached video clip to the user.

[0122] According to an embodiment, the appearance feature or the motion feature can be extracted differently according to a viewpoint of a camera that captures the sports video. Here, the viewpoint of the camera can have values of six degrees of freedom (DOF: Degree of Freedom) as a 3D position of the camera and a 3D orientation of the camera.

[0123] The service server can utilize the appearance feature or the motion feature that is robust to a viewpoint change of the camera. For example, the service server can encode the appearance feature in a manner that makes information included in a multi-dimensional appearance vector independent of the viewpoint of the camera. The appearance feature independent of the viewpoint of the camera can include a style feature of the player himself such as a hairstyle, a skin style (or a skin type), and / or a tattoo style, a style feature of an object worn by the player such as a jersey style, a basketball shoe style, and / or other accessory style, and / or a physical feature of the player such as a height or a build of the player standardized based on a reference object.

[0124] The style feature of the player himself and the feature of the object worn by the player can be defined in a manner independent of the viewpoint of the camera. The reference object is an object included in an appearance standardized in a corresponding sports game, for example, can include a line drawn on a goal post or a playing field. The physical feature such as the height or the build of the player can be standardized with reference to a size of the reference object captured in the sports video. Since the size of the reference object is standardized in the corresponding sports game, the physical feature standardized with reference to the reference object can be independent without being dependent on the viewpoint of the camera.

[0125] Also, the service server can encode the motion feature in a manner that makes information included in a multi-dimensional motion vector independent without being dependent on the viewpoint of the camera. The motion feature independent of the viewpoint of the camera can include a motion (e.g., a direction, a size, a speed, etc.) or a posture standardized with reference to a reference object. In a case where a joint of the player is identifiable, the motion feature can include a motion or a posture standardized by a major joint. Here, the major joint for the motion feature can be defined by a motion type.

[0126] According to an embodiment, the sports video can be captured in a case where a part of a sports field is cropped. The service server can utilize a region sequence to estimate a motion of a player in a space that is not captured but cropped. For example, in a case where an event of a ball passing an edge or a ball hitting an edge is sensed in a state where a shooting motion is not sensed, the service server can estimate that the shooting motion has occurred in the space that is not captured but cropped. Further, the service server can find a player who moves to the space that is not captured but cropped from a previous frame adjacent to a corresponding frame, and can estimate that the shooting motion has occurred by the corresponding player.

[0127] According to an embodiment, the service server can analyze the sports video using the movement of the referee, the whistle of the referee, or the beep of the score system, etc. For example, the service server can distinguish whether it is a 2-point shot or a 3-point shot using the movement of the referee. Or, the service server can determine whether the game is suspended at the corresponding time point using the whistle or the beep, etc.

[0128] According to an embodiment, the service server can identify the association according to the type of movement. For example, the service server can identify the assist situation in association with the pass movement before the shot movement. Or, the service server can identify the block situation in association with the block movement after the shot movement. In the case of the block situation, the direction of travel of the ball after the shot movement can also be considered together to identify.

[0129] Figure 12 is a diagram illustrating an operation of generating a tracking cluster according to an embodiment. Referring to Figure 12 , the user can upload the sports video to the streaming server 1210. The user can provide the link of the streaming server to the service server. As described above, in addition to the streaming server 1210, the sports video can also be uploaded to the storage server, or can also be directly uploaded to the service server.

[0130] The service server can access the streaming server 1210 using the corresponding link to receive the sports video. According to an embodiment, the service server can process while streaming the sports video, or can also process after downloading the sports video.

[0131] The preprocessing module 1220 of the service server can pre-process the sports video. For example, the service server can extract the images of the game time except for the rest time from the sports video. In the case of a sports game consisting of four quarters, the service server can extract the game images of the first quarter to the fourth quarter. And, the service server can extract the game-in-progress images from the game images except for the game non-in-progress images. For example, even within a quarter, the game can be interrupted due to a foul or a time-out, etc. The service server can extract the game-in-progress images in the game-in-progress except for the game non-in-progress images of the game interruption. The game-in-progress images can be subdivided according to the rules of the sports game. For example, in the case of a basketball game, a free throw can be given to the player who is fouled, and in the case of a soccer game, a free kick can be given. The service server can extract the free throw image or the free kick image according to the rules of the corresponding sports game.

[0132] The service server can extract images by extracting frame intervals or time intervals within a sports video. The preprocessing of the sports video can be operated in an automatic mode of automatic processing, a semi-automatic mode of receiving user feedback after automatic processing, or a manual mode of receiving user input. For example, the service server can receive start times and end times of the first quarter to the fourth quarter from the user. Alternatively, the service server can automatically extract start scene candidates and end scene candidates of the first quarter to the fourth quarter through video analysis, and receive selections of actual start scenes and actual end scenes of the respective quarters from the user. Also, the service server can recognize a match interruption situation due to a foul by analyzing a referee's whistle or the referee's movement. Of course, the service server can recognize the match interruption situation through video analysis. For example, there can be a difference in the amount of movement of a player during a match and the amount of movement of the player in a match interruption situation. The service server acquires the amount of movement of the player through video analysis, and thus can recognize the match interruption situation.

[0133] The service server (e.g., the video analysis module 1230) can detect and track non-identified players in the preprocessed video. The service server can track regions corresponding to non-identified players from the sports video and generate tracking clusters i0, i1, j0, k0. The sports video includes a plurality of frames, and the tracking clusters can include regions of the same non-identified player included in at least a portion of consecutive frames among the plurality of frames.

[0134] The service server can assign inherent identifiers to the tracking clusters i0, i1, j0, k0. For example, the service server can assign an identifier to a tracking cluster through a combination of a start frame and an inherent number. If there is a tracking cluster starting from the i-th frame, i0 can be assigned as an identifier. If there is another tracking cluster starting from the i-th frame, i1 can be assigned as an identifier. Similarly, a tracking cluster starting from the j-th frame can be assigned j0 as an identifier, and a tracking cluster starting from the k-th frame can be assigned k0 as an identifier.

[0135] Also, the service server can add information indicating an end frame of a tracking cluster or a number of frames of a tracking cluster to the identifier. It will be described below that, when performing matching between tracking clusters, matching can be omitted if only a portion of frames overlap. The service server can easily determine whether the tracking clusters overlap each other using the end frame or the number of frames added to the identifier.

[0136] In the case of processing a sports video in a streaming manner, the service server can also add information indicating whether a tracking cluster is still valid in the current frame to the identifier.

[0137] In tracking non-identified players in a sports video, tracking of non-identified players can be interrupted due to various factors. For example, depending on the viewpoint of the camera, there can be a case where non-identified players overlap each other. Or, there can be a case where non-identified players are out of the viewpoint of the camera. Or, there can be a case where a part or all of the screen is blocked by an obstacle in the process of filming a sports video. Or, due to other technical reasons, the tracking module can lose tracking of non-identified players. In this case, a first tracking cluster until tracking is lost can be generated, and a second tracking cluster newly tracked after tracking is lost can be generated. Hereinafter, in Figure 13 an embodiment of merging the tracking clusters generated by matching corresponding to the same non-identified player is explained.

[0138] Figure 13 is a diagram illustrating an operation of matching tracking clusters according to an embodiment. Referring to Figure 13 , the service server can classify the type of motion by tracking cluster (1310) and extract features (1320). The service server can extract the shape features of non-identified players included in the tracking cluster. As described above, the shape features can include a multi-dimensional feature vector independent of the viewpoint of the camera.

[0139] The service server can classify the type of motion of at least a part of the intervals included in the tracking cluster based on a predetermined type of motion. For example, the service server can detect at least a part of the regions included in the tracking cluster that are continuous with one of the predetermined types of motion. As an example, referring to the tracking cluster i0, the service server can detect a ball possession interval and a pass interval. The service server can utilize a neural network model that sequentially receives continuous scenes included in the tracking cluster and outputs at least one of the predetermined types of motion. The service server can extract the motion features of non-identified players of the corresponding intervals included in the tracking cluster based on the type of motion. As described above, the motion features can include a multi-dimensional feature vector independent of the viewpoint of the camera.

[0140] In the case where a shot interval is detected, the service server can separately mark the position on the court where the corresponding non-identified player attempted to shoot. It will be explained below that the service server can sense a scoring event and additionally mark whether the shot attempt of the non-identified player was successful.

[0141] According to an embodiment, the service server extracts a region corresponding to a state of having a ball from among the plurality of regions included in the tracking cluster, and can detect at least a part of the regions that are continuous with one of the predetermined types of motion from among the extracted regions.

[0142] In Figure 13In the embodiment, the service server can extract the appearance feature af_i0 from the tracking cluster i0. The service server can extract the motion feature mf_i0_DR from the ball possession interval included in the tracking cluster i0, and extract the motion feature mf_i0_PA from the pass interval. In substantially the same manner, the service server can extract the appearance feature af_i1 from the tracking cluster i1, and extract the motion feature mf_i1_SH. Also, the service server can extract the appearance feature af_j0 and the motion feature mf_j0_SH from the tracking cluster j0, and extract the appearance feature af_k0 and the motion feature mf_k0_BL from the tracking cluster k0.

[0143] The service server can match the tracking clusters based on at least one of the appearance features and the motion features (1330). The service server can not perform matching for the tracking clusters having partially overlapped frame intervals. This is because even the tracking clusters different from each other with partially overlapped frame intervals can be regarded as different non-identified players from each other. Accordingly, the service server can omit matching between the tracking cluster i0 and the tracking cluster i1. The service server can determine a matching failure (Failure) after matching the appearance feature of the tracking cluster i0 and the appearance feature of the tracking cluster j0. The service server can determine a matching success (Success) after matching the appearance feature of the tracking cluster i0 and the appearance feature of the tracking cluster k0. In the case of the matching success, the service server can merge the tracking cluster i0 and the tracking cluster k0. Here, the merging can be understood as an operation of including the tracking cluster i0 and the tracking cluster k0 in the cluster per non-identified player for the same non-identified player.

[0144] Also, the service server can match the motion features of the same motion type between the tracking clusters having frame intervals not overlapping each other. For example, the service server can determine a matching failure (Failure) after matching mf_i1_SH extracted from the shooting interval of the tracking cluster i1 and mf_j0_SH extracted from the shooting interval of the tracking cluster j0.

[0145] The service server can generate the cluster per non-identified player through the aforementioned matching operation. In the above embodiment, the "cluster" refers to the cluster per non-identified player, and can be understood as a concept distinguished from the "tracking cluster". The service server can generate the video clip per non-identified player based on the cluster (1340). In the operation after generating the cluster per non-identified player, the matters described above can be directly applied, and thus a more detailed description is omitted. Figures 1 to 11 The matters described above can be directly applied, and thus a more detailed description is omitted.

[0146] Figure 14 is a diagram illustrating an operation of sensing a scoring event according to an embodiment. Referring to FIG. 14, the service server can extract the appearance feature af_i0 from the tracking cluster i0, and extract the motion feature mf_i0_DR from the ball possession interval included in the tracking cluster i0. Also, the service server can extract the appearance feature af_i1 from the tracking cluster i1, and extract the motion feature mf_i1_SH from the shooting interval of the tracking cluster i1. Also, the service server can extract the appearance feature af_j0 from the tracking cluster j0, and extract the motion feature mf_j0_SH from the shooting interval of the tracking cluster j0. Figure 14In the case of a basketball game, whether the ball passes through the rim of the basket can be sensed. The service server detects a basket region or a rim region of the basket in the sports video, and as the frames proceed, can determine whether a scoring event in which the ball passes through the detected region is sensed.

[0147] In the case in which a scoring event is sensed, the service server can identify a tracking cluster related to the corresponding event among the tracking clusters based on the corresponding frame. For example, the service server can identify a tracking cluster including a section of a motion type related to the corresponding event in the corresponding frame and a previous frame among the tracking clusters. In the case in which a scoring event is sensed, the service server can identify a tracking cluster including a shot section. The service server can identify a tracking cluster including a most recent shot section in a previous frame of the scoring event. The service server can mark information that a shot attempted in the corresponding shot section is successful.

[0148] According to an embodiment, in generating a video clip of a non-identified player, the service server can extract a video clip from the sports video based on a sub-cluster related to a scoring event after selectively acquiring the sub-cluster related to the scoring event.

[0149] The embodiments described above can be implemented by a processing device of a hardware constituent element, a software constituent element, and / or a combination of the hardware constituent element and the software constituent element. For example, the apparatus, the method, and the constituent element described in the embodiments can be implemented using one or more general-purpose computers or special-purpose computers (for example, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor (DSP), a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device that can execute an instruction and respond thereto). The processing device can execute an operating system (OS) and one or more software applications executed on the operating system. Also, the processing device can access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, although there is a case where one processing device is used, it is known to one of ordinary skill in the art that the processing device can include a plurality of processing elements and / or a plurality of types of processing elements. For example, the processing device can include a plurality of processors or one processor and one controller. Also, it can be other processing configurations such as a parallel processor.

[0150] The software can include a computer program, a code, an instruction, or a combination of one or more thereof, can constitute a processing device in a manner that can operate as desired, or can command the processing device independently or collectively. In order to be parsed by the processing device or to provide the processing device with instructions or data, the software and / or data can be embodied in a certain type of machine, a constituent element, a physical device, a virtual device, a computer storage medium or device, or a transmitted signal wave, permanently or temporarily. The software is distributed on computer systems connected through a network, so it can also be stored or executed in a distributed manner. The software and data can be stored in one or more computer-readable recording media.

[0151] The method according to the embodiments can be realized in an executable program command form by various computer units and recorded in a computer readable medium. The computer readable medium can include program commands, data files, data structures, etc. individually or in combination. The program commands recorded in the medium can be program commands specially designed and constituted for the embodiments, or can be program commands well known and available to those skilled in the computer software field. Examples of the computer readable recording medium include magnetic media such as a hard disk, a floppy disk, and a magnetic tape, optical media such as a CD-ROM, a DVD, a floptical disk, and a magneto-optical media, and hardware devices specially constituted to store and execute program commands such as a ROM, a RAM, and a flash memory. Examples of the program commands include not only machine language codes generated by a compiler but also high level language codes executable by a computer using an interpreter. The hardware devices can be constituted to operate as one or more software modules in order to execute the operations of the embodiments, and vice versa.

[0152] As described above, although the embodiments have been described through the limited drawings, those of ordinary skill in the art to which the present invention pertains can apply various technical modifications and variations based on the above-described content. For example, even if the described techniques are executed in a different order from the described method, and / or the constituent elements of the described system, structure, device, circuit, etc. are combined or combined in a different form from the described method, or are replaced or substituted by other constituent elements or equivalents, appropriate results can be achieved.

Claims

1.An operation method of a video analysis server, comprising the steps of: receiving information of a sports video of an item of ball game; performing a preprocessing of filtering out static pixels to leave dynamic pixels included in a plurality of frames of the sports video; tracking a ball of the sports video based on the preprocessed video; detecting a score-related scene of the sports video from the preprocessed video; in response to the detection of the score-related scene, determining a non-identified player related to the score-related scene using a tracking result of the ball; tracking the non-identified player until an adjacent frame in which the non-identified player can be identified, thereby identifying the non-identified player; and outputting a time interval of the sports video and identification information of the non-identified player corresponding to the score-related scene, wherein the step of detecting the score-related scene comprises the steps of: detecting an edge based on dynamic pixels of each of the frames; and determining a frame adjacent to a frame in which the edge is detected as the score-related scene. 2.The operation method of a video analysis server according to claim 1, wherein the step of tracking the ball comprises the step of: detecting a ball based on dynamic pixels of each of the frames. 3.The operation method of a video analysis server according to claim 1, wherein the step of determining the non-identified player related to the score-related scene comprises the steps of: detecting dynamic pixels related to a player who attempts to score in a frame included in the score-related scene using the tracking result of the ball; and determining the non-identified player related to the score-related scene by instance segmentation of the frame in which the dynamic pixels related to the player who attempts to score are detected. 4.The operation method of a video analysis server according to claim 1, wherein the step of identifying the non-identified player comprises the steps of: extracting a feature from the determined non-identified player; comparing the extracted feature with a feature of a registered player; determining whether the non-identified player can be identified based on a result of the comparison; and tracking the non-identified player by instance segmentation of an adjacent frame according to a determination that the non-identified player cannot be identified. 5.The operation method of a video analysis server according to claim 1, wherein the step of performing the preprocessing comprises the steps of: in a case where the sports video is a video captured with a fixed viewpoint, filtering out static pixels based on a change in pixel values between adjacent frames of a predetermined range; and in a case where the sports video is a video captured with a moving viewpoint, filtering out static pixels based on a statistical value of an optical flow of pixels within a frame. 6.A computer-readable recording medium recording a computer program which, when executed by a processor, causes the processor to perform the operation method of a video analysis server according to claim 1. 7.An operation method of a server providing a platform service based on a sports video, comprising the steps of: ​ ​ ​ transmitting a signal requesting to analyze the sports video to the video analysis module based on the information on the accessed sports video to make the video analysis module analyze the sports video according to the operation method of claim 1; storing the clusters by players received from the video analysis module in a database; providing information for extracting a video clip of a player from the sports video to a user terminal based on the database; receiving an input for identifying a non-identified player of at least one cluster from the user terminal, wherein the user terminal receives the video clip of the player from a streaming server; and updating identification information of at least one corresponding cluster in the database based on the input. 8.The operation method of a server for providing a platform service based on sports videos according to claim 7, further comprising the steps of: providing statistics representing a degree of contribution of a player to the user terminal; receiving an input of selecting a detailed record included in the statistics from the user terminal; acquiring at least one sub-cluster related to the selected detailed record based on the database; and providing information for extracting a video clip from the sports video to the user terminal based on the at least one sub-cluster. 9.The operation method of a server for providing a platform service based on sports videos according to claim 7, further comprising the steps of: receiving a search query including a search target player and a scene of the search target from the user terminal; searching for a sub-cluster corresponding to the search query from the database; and providing information for extracting a video clip from the sports video to the user terminal based on the searched sub-cluster. 10.The operation method of a server for providing a platform service based on sports videos according to claim 7, further comprising at least one of the steps of: determining a billing level based on reliability of the cluster; and determining a compensation level based on a feedback input modifying the cluster. 11.The operation method of a server for providing a platform service based on sports videos according to claim 7, further comprising the steps of: receiving a feedback input indicating that a player of at least one interval included in the at least one cluster does not belong to a corresponding cluster from the user terminal; and excluding the corresponding interval from the corresponding cluster based on the feedback input. 12.The operation method of a server for providing a platform service based on sports videos according to claim 7, further comprising the steps of: receiving a feedback input indicating that a player of at least one interval included in the at least one cluster belongs to another cluster from the user terminal; and excluding the corresponding interval from the corresponding cluster and including the interval in the other cluster based on the feedback input. 13.The operation method of a server for providing a platform service based on sports videos according to claim 7, further comprising the steps of: generating training data depending on the updated database; and learning a specialized model for estimating at least one of detection information, identification information, and motion type information of a player based on the training data. ​ 14.A computer-readable recording medium having recorded thereon a computer program that, when executed by a processor, causes the processor to execute an operation method of a server for providing a platform service based on a sports video according to claim 7.

Images